Iyo nguva yakatevedzana dhatabhesi (TSDB) muPrometheus 2 muenzaniso wakanaka kwazvo weinjiniya mhinduro inopa kuvandudzwa kukuru pamusoro peiyo v2 chengetedzo muPrometheus 1 maererano nekumhanyisa kuunganidza data, kuita mibvunzo, uye kushanda nesimba. Takanga tichiita Prometheus 2 muPercona Monitoring and Management (PMM) uye ndakava nemukana wekunzwisisa mashandiro ePrometheus 2 TSDB. Muchikamu chino ndichataura pamusoro pemigumisiro yekucherechedza uku.
Avhareji yePrometheus Workload
Kune avo vajaira kubata neyakajairwa chinangwa dhatabhesi, yakajairika Prometheus basa rekuita rinonakidza. Mwero wekuunganidzwa kwedata unowanzo kugadzikana: kazhinji masevhisi aunotarisa anotumira angangoita nhamba yakafanana yemametrics, uye zvivakwa zvinoshanduka zvishoma nezvishoma.
Zvikumbiro zveruzivo zvinogona kubva kwakasiyana-siyana. Dzimwe dzadzo, dzakadai seyambiro, dzinovavarirawo kukosha kwakagadzikana uye kunofungidzirwa. Zvimwe, sezvikumbiro zvevashandisi, zvinogona kukonzera kuputika, kunyangwe izvi zvisiri izvo kune akawanda mabasa.
Mutoro bvunzo
Panguva yekuedzwa, ndakatarisa pakukwanisa kuunganidza data. Ndakatumira Prometheus 2.3.2 yakabatanidzwa neGo 1.10.1 (sechikamu chePMM 1.14) paLinode sevhisi ndichishandisa chinyorwa ichi: . Kune iyo chaiyo inotakura mutoro chizvarwa, uchishandisa izvi Ini ndakatangisa akati wandei eMySQL node ine mutoro chaiwo (Sysbench TPC-C Test), imwe neimwe yaitevedzera gumi Linux/MySQL node.
Ese aya anotevera bvunzo akaitwa paLinode server ine masere chaiwo cores uye 32 GB yendangariro, ichimhanya makumi maviri ekuyedza mitoro inotarisisa mazana maviri eMySQL zviitiko. Kana, maererano nePrometheus, 20 targets, 800 scrapes pasekondi, 440 zviuru zvinyorwa pasekondi, uye 380 miriyoni inoshanda nguva yakatevedzana.
patani
Maitiro echinyakare dhatabhesi, kusanganisira iyo inoshandiswa naPrometheus 1.x, ndeye . Kana isina kukwana kubata mutoro, iwe uchasangana nepamusoro latencies uye zvimwe zvikumbiro zvinokundikana. Memory kushandiswa muPrometheus 2 inogadziriswa kuburikidza nekiyi storage.tsdb.min-block-duration, iyo inosarudza kuti kurekodha kuchachengetwa kwenguva yakareba sei mundangariro usati watsveta ku diski (default ndeye 2 maawa). Huwandu hwendangariro hunodiwa hunoenderana nenhamba yenguva yakatevedzana, mavara, uye zvikwapuro zvakawedzerwa kune mambure anouya. Panyaya yedhisiki nzvimbo, Prometheus inotarisira kushandisa 3 bytes pane rekodhi (sample). Kune rumwe rutivi, zvinodikanwa zvendangariro zvakakwirira zvikuru.
Kunyangwe zvichigoneka kugadzirisa saizi yebhuroka, hazvikurudzirwe kuigadzirisa nemaoko, saka unomanikidzwa kupa Prometheus ndangariro yakawandisa sezvainoda pamutoro wako webasa.
Kana pasina ndangariro yakakwana yekutsigira iri kuuya yemetrics, Prometheus inodonha mundangariro kana mhondi yeOOM ichasvika pairi.
Kuwedzera kuchinjanisa kunonoka kuparara kana Prometheus apera mundangariro hakunyatso kubatsira, nekuti kushandisa basa iri kunokonzeresa kuputika ndangariro. Ini ndinofunga chimwe chinhu chine chekuita neGo, muunganidzi wayo wemarara uye mabatiro aanoita nekuchinjanisa.
Imwe nzira inofadza ndeyekugadzirisa musoro wekuvhara kuti uende ku diski pane imwe nguva, panzvimbo yekuiverenga kubva pakutanga kwekuita.
Sezvauri kuona kubva pagirafu, flushes kune disk inoitika maawa maviri oga oga. Kana iwe ukashandura iyo min-block-duration parameter kusvika awa imwe, zvino izvi zvigadziriso zvichaitika awa yega yega, kutanga mushure mehafu yeawa.
Kana iwe uchida kushandisa ichi uye mamwe magirafu mune yako Prometheus yekumisikidza, unogona kushandisa izvi . Yakagadzirirwa PMM asi, ine zvidiki zvigadziriso, inokodzera chero kuisirwa Prometheus.
Tine chivharo chinoshanda chinonzi head block icho chinochengetwa mundangariro; mabhuroko ane data rekare anowanikwa kuburikidza mmap(). Izvi zvinobvisa kukosha kwekugadzirisa cache zvakasiyana, asi zvakare zvinoreva kuti iwe unofanirwa kusiya nzvimbo yakakwana yeanoshanda system cache kana iwe uchida kubvunza data rekare pane izvo musoro block inogona kugara.
Izvi zvinoreva zvakare kuti Prometheus chaiyo ndangariro yekushandisa inotaridzika yakakwira, icho chisiri chinhu chekunetsekana nacho.
Imwe inonakidza dhizaini poindi ndeye kushandiswa kweWAL (nyora pamberi pegi). Sezvauri kuona kubva muzvinyorwa zvekuchengetedza, Prometheus anoshandisa WAL kudzivirira kubondera. Maitiro chaiwo ekuvimbisa kupona kwedata, zvinosuwisa, haana kunyorwa zvakanaka. Prometheus shanduro 2.3.2 flushes WAL kuti dhisiki ose 10 masekonzi uye iyi sarudzo haisi mushandisi configurable.
compaction
Prometheus TSDB yakagadzirwa senge LSM (Log Structured Merge) chitoro: iyo musoro block inonyungudutswa nguva nenguva kudhisiki, nepo compaction michina inosanganisa akawanda mabhuraki pamwechete kudzivirira kutarisisa mabhuroki akawanda panguva yekubvunza. Pano iwe unogona kuona huwandu hwezvivharo zvandakaona pane test system mushure mezuva rekuremerwa.
Kana iwe uchida kudzidza zvakawanda pamusoro pechitoro, unogona kuongorora meta.json faira, ine ruzivo pamusoro pezvivharo zviripo uye kuti zvakavapo sei.
{
"ulid": "01CPZDPD1D9R019JS87TPV5MPE",
"minTime": 1536472800000,
"maxTime": 1536494400000,
"stats": {
"numSamples": 8292128378,
"numSeries": 1673622,
"numChunks": 69528220
},
"compaction": {
"level": 2,
"sources": [
"01CPYRY9MS465Y5ETM3SXFBV7X",
"01CPYZT0WRJ1JB1P0DP80VY5KJ",
"01CPZ6NR4Q3PDP3E57HEH760XS"
],
"parents": [
{
"ulid": "01CPYRY9MS465Y5ETM3SXFBV7X",
"minTime": 1536472800000,
"maxTime": 1536480000000
},
{
"ulid": "01CPYZT0WRJ1JB1P0DP80VY5KJ",
"minTime": 1536480000000,
"maxTime": 1536487200000
},
{
"ulid": "01CPZ6NR4Q3PDP3E57HEH760XS",
"minTime": 1536487200000,
"maxTime": 1536494400000
}
]
},
"version": 1
}MaCompactions muPrometheus akasungirirwa panguva iyo musoro block inonyungudutswa kune disk. Panguva ino, maitiro akawanda akadai anogona kuitwa.
Zvinotaridza kuti compaction haina kuganhurirwa chero nzira uye inogona kukonzera hombe disk I / O spikes panguva yekuurayiwa.
CPU mutoro spikes
Ehe, izvi zvine zvazvinokanganisa kukurumidza kwehurongwa, uye zvakare zvinounza dambudziko rakakura kune LSM kuchengetedza: maitiro ekuita compaction kutsigira yakakwirira yekukumbira mitengo pasina kukonzera yakawandisa kumusoro?
Iko kushandiswa kwendangariro mukugadzirisa maitiro zvakare kunotaridzika kunonakidza.
Isu tinogona kuona kuti, mushure mekubatana, mazhinji endangariro anoshandura mamiriro kubva kuCached kuenda kuMahara: izvi zvinoreva kuti ruzivo rwakakosha rwakabviswa kubva ipapo. Kuda kuziva kana ichishandiswa pano fadvice() kana imwe nzira yekudzikisa, kana kuti imhaka yekuti cache yakasunungurwa kubva kumabhuroko akaparadzwa panguva yekubatana?
Kupora mushure mekukundikana
Kupora kubva mukukundikana kunotora nguva, uye nechikonzero chakanaka. Nekuyerera kuri kuuya kwemiriyoni rekodhi pasekondi, ndaifanira kumirira maminetsi makumi maviri nemashanu apo kudzoreredza kwaiitwa ndichifunga nezve SSD drive.
level=info ts=2018-09-13T13:38:14.09650965Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.2, branch=v2.3.2, revision=71af5e29e815795e9dd14742ee7725682fa14b7b)"
level=info ts=2018-09-13T13:38:14.096599879Z caller=main.go:223 build_context="(go=go1.10.1, user=Jenkins, date=20180725-08:58:13OURCE)"
level=info ts=2018-09-13T13:38:14.096624109Z caller=main.go:224 host_details="(Linux 4.15.0-32-generic #35-Ubuntu SMP Fri Aug 10 17:58:07 UTC 2018 x86_64 1bee9e9b78cf (none))"
level=info ts=2018-09-13T13:38:14.096641396Z caller=main.go:225 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-09-13T13:38:14.097715256Z caller=web.go:415 component=web msg="Start listening for connections" address=:9090
level=info ts=2018-09-13T13:38:14.097400393Z caller=main.go:533 msg="Starting TSDB ..."
level=info ts=2018-09-13T13:38:14.098718401Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536530400000 maxt=1536537600000 ulid=01CQ0FW3ME8Q5W2AN5F9CB7R0R
level=info ts=2018-09-13T13:38:14.100315658Z caller=web.go:467 component=web msg="router prefix" prefix=/prometheus
level=info ts=2018-09-13T13:38:14.101793727Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536732000000 maxt=1536753600000 ulid=01CQ78486TNX5QZTBF049PQHSM
level=info ts=2018-09-13T13:38:14.102267346Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536537600000 maxt=1536732000000 ulid=01CQ78DE7HSQK0C0F5AZ46YGF0
level=info ts=2018-09-13T13:38:14.102660295Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536775200000 maxt=1536782400000 ulid=01CQ7SAT4RM21Y0PT5GNSS146Q
level=info ts=2018-09-13T13:38:14.103075885Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536753600000 maxt=1536775200000 ulid=01CQ7SV8WJ3C2W5S3RTAHC2GHB
level=error ts=2018-09-13T14:05:18.208469169Z caller=wal.go:275 component=tsdb msg="WAL corruption detected; truncating" err="unexpected CRC32 checksum d0465484, want 0" file=/opt/prometheus/data/.prom2-data/wal/007357 pos=15504363
level=info ts=2018-09-13T14:05:19.471459777Z caller=main.go:543 msg="TSDB started"
level=info ts=2018-09-13T14:05:19.471604598Z caller=main.go:603 msg="Loading configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499156711Z caller=main.go:629 msg="Completed loading of configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499228186Z caller=main.go:502 msg="Server is ready to receive web requests."Dambudziko guru rekugadzirisa maitiro ndiro kushandiswa kwekuyeuka kwakanyanya. Kunyangwe ichokwadi chekuti mune yakajairika mamiriro sevha inogona kushanda yakadzikama neyakafanana ndangariro, kana ikaparara inogona kusapora nekuda kweOOM. Mhinduro chete yandakawana yaive yekudzima kuunganidzwa kwedata, kuunza sevha, ita kuti idzore uye kutangazve nekuunganidza kwakagoneswa.
Kudziya
Imwe hunhu hwekuchengeta mupfungwa panguva yekudziya-up hukama pakati pekuita kwakaderera uye kushandiswa kwepamusoro zviwanikwa mushure mekutanga. Pakati pezvimwe, asi kwete zvese zvinotanga, ndakaona mutoro wakakomba paCPU uye ndangariro.
Makapu mukushandiswa kwendangariro anoratidza kuti Prometheus haigone kugadzirisa zvese zvakaunganidzwa kubva pakutanga, uye rumwe ruzivo rwakarasika.
Ini handisati ndafunga zvikonzero chaizvo zveiyo yakakwira CPU uye ndangariro mutoro. Ini ndinofungidzira kuti izvi zvakakonzerwa nekugadzirwa kweiyo nguva nyowani mumusoro block ine yakakwira frequency.
CPU mutoro unowedzera
Pamusoro pema compaction, ayo anogadzira yakakwira I / O mutoro, ndakaona zvakakomba spikes muCPU mutoro maminetsi maviri ega ega. Mabhomba acho akareba kana kuyerera kwekuisa kwakakwira uye kuchiita sekunge kuri kukonzerwa nemuunganidzi wemarara weGo, paine mamwe macores ari kuzara.
Kusvetuka uku hakusi kusakosha. Zvinotaridza kuti kana izvi zvikaitika, Prometheus yemukati yekupinda poindi uye metrics inove isingawanikwe, zvichikonzera gaka re data panguva dzakafanana dzenguva.
Iwe unogona zvakare kuona kuti iyo Prometheus mutengesi inovhara pasi kwesekondi imwe.
Tinogona kuona kuwirirana nekuunganidzwa kwemarara (GC).
mhedziso
TSDB muPrometheus 2 inokurumidza, inokwanisa kubata mamirioni enguva yakatevedzana uye panguva imwe chete zviuru zvemarekodhi pasekondi uchishandisa zvine mwero hardware. CPU uye dhisiki I / O kushandiswa zvakare kunonakidza. Muenzaniso wangu wakaratidza kusvika 200 metrics pasekondi yega yega yakashandiswa.
Kuronga kuwedzera, iwe unofanirwa kuyeuka nezvehuwandu hwakakwana hwendangariro, uye iyi inofanira kunge iri ndangariro chaiyo. Huwandu hwendangariro hwakashandiswa hwandakaona hwaive hunenge 5 GB pa100 marekodhi pasekondi yerukova runouya, iyo pamwe chete neyekushandisa system cache yakapa ingangoita 000 GB yekuyeuka.
Ehezve, pachine basa rakawanda rinofanira kuitwa kutame CPU uye disk I/O spikes, uye izvi hazvishamise tichifunga kuti mudiki TSDB Prometheus 2 inofananidzwa sei neInnoDB, TokuDB, RocksDB, WiredTiger, asi vese vaive nemhando yakafanana. dambudziko pakutanga kwehupenyu hwavo.
Source: www.habr.com