Sebaka sa polokelo ea nako (TSDB) ho Prometheus 2 ke mohlala o babatsehang oa tharollo ea boenjiniere e fanang ka lintlafatso tse kholo holim'a polokelo ea v2 ho Prometheus 1 mabapi le lebelo la ho bokella data, ho phethahatsa lipotso, le ho sebetsa hantle ha lisebelisoa. Re ne re kenya tšebetsong Prometheus 2 ho Percona Monitoring and Management (PMM) mme ke bile le monyetla oa ho utloisisa tšebetso ea Prometheus 2 TSDB. Sehloohong sena ke tla bua ka liphello tsa litebello tsena.
Karolelano ea mosebetsi oa Prometheus
Bakeng sa ba tloaetseng ho sebetsana le li-database tsa sepheo se akaretsang, mosebetsi o tloaelehileng oa Prometheus o khahla haholo. Sekhahla sa pokello ea lintlha se batla se tsitsitse: hangata lits'ebeletso tseo u li behang leihlo li romella palo e lekanang ea metrics, 'me lisebelisoa li fetoha butle butle.
Likopo tsa tlhahisoleseling li ka tsoa mehloling e fapaneng. Tse ling tsa tsona, joalo ka litlhokomeliso, le tsona li loanela ho fumana boleng bo tsitsitseng le bo ka lebelloang esale pele. Tse ling, joalo ka likopo tsa basebelisi, li ka baka ho phatloha, le hoja ho se joalo bakeng sa mesebetsi e mengata.
Teko ea mojaro
Nakong ea tlhahlobo, ke ile ka tsepamisa maikutlo ho bokhoni ba ho bokella data. Ke sebelisitse Prometheus 2.3.2 e hlophisitsoeng le Go 1.10.1 (e le karolo ea PMM 1.14) litšebeletsong tsa Linode ke sebelisa mongolo ona: . Bakeng sa moloko oa sebele oa mojaro, sebelisa sena Ke ile ka qala li-node tse 'maloa tsa MySQL tse nang le mojaro oa sebele (Sysbench TPC-C Test), e' ngoe le e 'ngoe ea tsona e neng e etsisa 10 Linux / MySQL nodes.
Liteko tsohle tse latelang li entsoe ho seva ea Linode e nang le li-cores tse robeli le 32 GB ea memori, e tsamaisang lipapiso tse 20 tse shebileng liketsahalo tse makholo a mabeli tsa MySQL. Kapa, ho ea ka Prometheus, sepheo sa 800, 440 scrapes ka motsotsoana, lirekoto tse likete tse 380 ka motsotsoana, le letoto la nako e sebetsang ea limilione tse 1,7.
moralo
Mokhoa o tloaelehileng oa li-database tsa setso, ho kenyelletsa le o sebelisoang ke Prometheus 1.x, ke ho . Haeba ho sa lekana ho sebetsana le mojaro, o tla ba le latency e phahameng mme likopo tse ling li tla hloleha. Tšebeliso ea memori ho Prometheus 2 e ka lokisoa ka senotlolo storage.tsdb.min-block-duration, e etsang qeto ea hore na lirekoto li tla bolokoa nako e kae mohopolong pele li fetisetsoa ho disk (ka ho feletseng ke lihora tse 2). Palo ea memori e hlokahalang e tla ipapisa le palo ea letoto la nako, lileibole le likhechana tse kentsoeng molapong o kenang oa marang-rang. Mabapi le sebaka sa disk, Prometheus e ikemiselitse ho sebelisa li-byte tsa 3 ka rekoto (sample). Ka lehlakoreng le leng, litlhoko tsa ho hopola li phahame haholo.
Le hoja ho ka khoneha ho lokisa boholo ba thibela, ha ho kgothaletswe ho e lokisa ka letsoho, kahoo o qobelloa ho fa Prometheus mohopolo o mongata kamoo o hlokang bakeng sa mosebetsi oa hau.
Haeba ho se na mohopolo o lekaneng oa ho ts'ehetsa mokhoa o kenang oa metrics, Prometheus o tla tsoa mohopolong kapa 'molai oa OOM o tla fihla ho eona.
Ho kenyelletsa swap ho liehisa ho oa ha Prometheus a felloa ke mohopolo ha ho thuse, hobane ho sebelisa ts'ebetso ena ho baka ts'ebeliso e matla ea mohopolo. Ke nahana hore e amana le Go, moqopi oa eona oa lithōle le tsela eo e sebetsanang le swap.
Tsela e 'ngoe e thahasellisang ke ho lokisa hlooho ea hlooho hore e phalle ho disk ka nako e itseng, ho e-na le ho e bala ho tloha qalong ea ts'ebetso.
Joalo ka ha u bona ho graph, flushes ho disk e etsahala lihora tse ling le tse ling tse peli. Haeba u fetola parameter ea min-block-duration ho hora e le 'ngoe, joale li-resets tsena li tla etsahala hora e' ngoe le e 'ngoe, ho qala ka mor'a halofo ea hora.
Haeba u batla ho sebelisa sena le li-graph tse ling ts'ebetsong ea hau ea Prometheus, u ka sebelisa sena . E ne e etselitsoe PMM empa, ka liphetoho tse nyane, e lumellana le kopo efe kapa efe ea Prometheus.
Re na le block e sebetsang e bitsoang "head block" e bolokiloeng mohopolong; li-blocks tse nang le data ea khale li fumaneha ka mmap(). Sena se felisa tlhoko ea ho hlophisa cache ka thoko, empa hape ho bolela hore o hloka ho siea sebaka se lekaneng bakeng sa cache ea sistimi ea ts'ebetso haeba o batla ho botsa data ea khale ho feta seo hlooho e ka se amohelang.
Sena se boetse se bolela hore tšebeliso ea memori ea Prometheus e tla shebahala e phahame haholo, eo e seng ntho ea ho tšoenyeha ka eona.
Ntlha e 'ngoe e thahasellisang ea moralo ke tšebeliso ea WAL (ngola pele log). Joalokaha u bona ho tsoa litokomaneng tsa polokelo, Prometheus e sebelisa WAL ho qoba likotsi. Mekhoa e ikhethileng ea ho netefatsa ho pholoha ha data, ka bomalimabe, ha e ea ngoloa hantle. Prometheus version 2.3.2 flushes WAL ho disk metsotsoana e meng le e meng e 10 'me khetho ena ha e khonehe ho sebelisoa.
Likopano
Prometheus TSDB e entsoe joalo ka lebenkele la LSM (Log Structured Merge): "block block" e hlatsuoa nako le nako ho disk, ha mochine oa compaction o kopanya li-blocks tse ngata hammoho ho qoba ho hlahloba li-blocks tse ngata nakong ea lipotso. Mona o ka bona palo ea li-blocks tseo ke li boneng ho sistimi ea liteko kamora letsatsi la mojaro.
Haeba u batla ho ithuta ho eketsehileng ka lebenkele, u ka hlahloba faele ea meta.json, e nang le boitsebiso bo mabapi le li-blocks tse teng le hore na li bile teng joang.
{
"ulid": "01CPZDPD1D9R019JS87TPV5MPE",
"minTime": 1536472800000,
"maxTime": 1536494400000,
"stats": {
"numSamples": 8292128378,
"numSeries": 1673622,
"numChunks": 69528220
},
"compaction": {
"level": 2,
"sources": [
"01CPYRY9MS465Y5ETM3SXFBV7X",
"01CPYZT0WRJ1JB1P0DP80VY5KJ",
"01CPZ6NR4Q3PDP3E57HEH760XS"
],
"parents": [
{
"ulid": "01CPYRY9MS465Y5ETM3SXFBV7X",
"minTime": 1536472800000,
"maxTime": 1536480000000
},
{
"ulid": "01CPYZT0WRJ1JB1P0DP80VY5KJ",
"minTime": 1536480000000,
"maxTime": 1536487200000
},
{
"ulid": "01CPZ6NR4Q3PDP3E57HEH760XS",
"minTime": 1536487200000,
"maxTime": 1536494400000
}
]
},
"version": 1
}Likopano tsa Prometheus li hokahane le nako eo hlooho ea hlooho e phahamisoang ho disk. Tabeng ena, ho ka 'na ha etsoa mesebetsi e mengata e joalo.
Ho bonahala eka li-compactions ha li na moeli ka tsela leha e le efe mme li ka baka li-spikes tse kholo tsa disk I / O nakong ea ts'ebetso.
Li-spikes tsa boima ba CPU
Ha e le hantle, sena se na le tšusumetso e mpe ho lebelo la tsamaiso, hape se baka phephetso e tebileng bakeng sa polokelo ea LSM: mokhoa oa ho etsa compaction ho tšehetsa litekanyetso tse phahameng tsa kopo ntle le ho baka ho hongata haholo?
Tšebeliso ea memori ts'ebetsong ea compaction e boetse e shebahala e thahasellisa haholo.
Re ka bona kamoo, ka mor'a compaction, boholo ba mohopolo o fetola boemo ho tloha Cached ho Free: sena se bolela hore boitsebiso bo ka 'nang ba e-ba bohlokoa bo tlositsoe moo. Kea kholoa haeba e sebelisoa mona fadvice() kapa mokhoa o mong oa ho nyenyefatsa, kapa na ke hobane cache e lokolotsoe ho li-blocks tse sentsoeng nakong ea compaction?
Ho hlaphoheloa ka mor'a ho hlōleha
Ho hlaphoheloa liphosong ho nka nako, 'me ke ka lebaka le utloahalang. Bakeng sa molaetsa o kenang oa lirekoto tse milione ka motsotsoana, ke ile ka tlameha ho ema metsotso e ka bang 25 ha ho hlaphoheloa ho ntse ho etsoa ho nahanela SSD drive.
level=info ts=2018-09-13T13:38:14.09650965Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.2, branch=v2.3.2, revision=71af5e29e815795e9dd14742ee7725682fa14b7b)"
level=info ts=2018-09-13T13:38:14.096599879Z caller=main.go:223 build_context="(go=go1.10.1, user=Jenkins, date=20180725-08:58:13OURCE)"
level=info ts=2018-09-13T13:38:14.096624109Z caller=main.go:224 host_details="(Linux 4.15.0-32-generic #35-Ubuntu SMP Fri Aug 10 17:58:07 UTC 2018 x86_64 1bee9e9b78cf (none))"
level=info ts=2018-09-13T13:38:14.096641396Z caller=main.go:225 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-09-13T13:38:14.097715256Z caller=web.go:415 component=web msg="Start listening for connections" address=:9090
level=info ts=2018-09-13T13:38:14.097400393Z caller=main.go:533 msg="Starting TSDB ..."
level=info ts=2018-09-13T13:38:14.098718401Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536530400000 maxt=1536537600000 ulid=01CQ0FW3ME8Q5W2AN5F9CB7R0R
level=info ts=2018-09-13T13:38:14.100315658Z caller=web.go:467 component=web msg="router prefix" prefix=/prometheus
level=info ts=2018-09-13T13:38:14.101793727Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536732000000 maxt=1536753600000 ulid=01CQ78486TNX5QZTBF049PQHSM
level=info ts=2018-09-13T13:38:14.102267346Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536537600000 maxt=1536732000000 ulid=01CQ78DE7HSQK0C0F5AZ46YGF0
level=info ts=2018-09-13T13:38:14.102660295Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536775200000 maxt=1536782400000 ulid=01CQ7SAT4RM21Y0PT5GNSS146Q
level=info ts=2018-09-13T13:38:14.103075885Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536753600000 maxt=1536775200000 ulid=01CQ7SV8WJ3C2W5S3RTAHC2GHB
level=error ts=2018-09-13T14:05:18.208469169Z caller=wal.go:275 component=tsdb msg="WAL corruption detected; truncating" err="unexpected CRC32 checksum d0465484, want 0" file=/opt/prometheus/data/.prom2-data/wal/007357 pos=15504363
level=info ts=2018-09-13T14:05:19.471459777Z caller=main.go:543 msg="TSDB started"
level=info ts=2018-09-13T14:05:19.471604598Z caller=main.go:603 msg="Loading configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499156711Z caller=main.go:629 msg="Completed loading of configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499228186Z caller=main.go:502 msg="Server is ready to receive web requests."Bothata bo ka sehloohong ba mokhoa oa ho hlaphoheloa ke tšebeliso e phahameng ea mohopolo. Ho sa tsotellehe taba ea hore boemong bo tloaelehileng seva se ka sebetsa ka mokhoa o tsitsitseng ka tekanyo e lekanang ea mohopolo, haeba e senyeha e ka 'na ea se ke ea hlaphoheloa ka lebaka la OOM. Tharollo feela eo ke e fumaneng e ne e le ho tima pokello ea data, ho hlahisa seva, ho e tlohella hore e hlaphoheloe ebe e qala bocha ka pokello e lumelletsoe.
Ho futhumala
Boitšoaro bo bong boo u lokelang ho bo hopola nakong ea ho futhumala ke kamano pakeng tsa ts'ebetso e tlaase le tšebeliso e phahameng ea lisebelisoa hang ka mor'a ho qala. Nakong e 'ngoe, empa e se tsohle tse qalang, ke bone mojaro o tebileng ho CPU le mohopolo.
Likheo tsa tšebeliso ea memori li bontša hore Prometheus ha e khone ho lokisa likoleke tsohle ho tloha qalong, 'me boitsebiso bo bong bo lahlehile.
Ha ke so fumane mabaka a tobileng a ho ba le CPU e phahameng le moroalo oa memori. Ke belaela hore sena se bakoa ke ho thehoa ha letoto la nako e ncha hloohong ea hlooho e nang le maqhubu a phahameng.
CPU ea eketseha
Ntle le li-compactions, tse etsang hore ho be le mojaro o phahameng oa I / O, ke ile ka bona li-spikes tse tebileng tsa CPU ka mor'a metsotso e meng le e meng e 'meli. Ho phatloha ho ba nako e telele ha phallo ea ho kenya e le holimo 'me e bonahala e bakoa ke 'mokelli oa lithōle oa Go, bonyane li-cores tse ling li laetsoe ka botlalo.
Ho qhoma hona ha se ntho e nyenyane hakaalo. Ho bonahala eka ha tsena li etsahala, ntlha ea ho kena ka hare ea Prometheus le metrics ha e fumanehe, e leng se bakang likheo tsa data nakong ena ea nako.
U ka boela ua hlokomela hore morekisi oa Prometheus o koala motsotsoana o le mong.
Re ka hlokomela likamano le pokello ea lithōle (GC).
fihlela qeto e
TSDB ho Prometheus 2 e potlakile, e khona ho sebetsana le letoto la nako le limilione mme ka nako e ts'oanang lirekoto tse likete motsotsoana li sebelisa lisebelisoa tse itekanetseng. Tšebeliso ea CPU le disk I/O le eona e ea khahla. Mohlala oaka o bonts'itse ho fihla ho 200 metrics motsotsoana ka konokono e sebelisitsoeng.
Ho rera katoloso, o hloka ho hopola ka bongata bo lekaneng ba mohopolo, mme sena e tlameha ho ba mohopolo oa 'nete. Palo ea memori e sebelisitsoeng eo ke e boneng e ne e ka ba 5 GB ka lirekoto tse 100 motsotsoana oa molapo o kenang, oo hammoho le cache ea sistimi e sebetsang e faneng ka 000 GB ea mohopolo o tšoaretsoeng.
Ehlile, ho sa ntsane ho e-na le mosebetsi o mongata o lokelang ho etsoa ho laola li-spikes tsa CPU le disk I/O, 'me sena ha se makatse ha ho nahanoa hore na TSDB Prometheus 2 e monyenyane e bapisoa joang le InnoDB, TokuDB, RocksDB, WiredTiger, empa kaofela ha tsona li ne li tšoana. mathata qalong ea potoloho ea bophelo ba bona.
Source: www.habr.com