O le taimi fa'asologa fa'amaumauga (TSDB) i le Prometheus 2 o se fa'ata'ita'iga sili lea o se fofo fa'ainisinia e ofoina atu fa'aleleia atili i luga o le v2 teuina i Prometheus 1 e tusa ai ma le saoasaoa fa'aputuina o fa'amaumauga, fa'atinoga o fesili, ma le fa'aogaina o punaoa. Sa matou faʻatinoina le Prometheus 2 i le Percona Monitoring and Management (PMM) ma sa ou maua le avanoa e malamalama ai i le faʻatinoga o le Prometheus 2 TSDB. I lenei tusiga o le a ou talanoa e uiga i taunuuga o nei matauga.
Aveavega o galuega a Prometheus
Mo i latou e masani ona faʻatautaia faʻamaumauga faʻamaumauga lautele, o le galuega masani a Prometheus e manaia tele. Ole fua ole fa'aputuina o fa'amaumauga e foliga mai e mautu: e masani lava o au'aunaga e te mata'ituina e tu'uina atu e tusa ma le numera tutusa o fua, ma e fa'agesegese le suiga o atina'e.
Talosaga mo faʻamatalaga e mafai ona sau mai punaoa eseese. O nisi oi latou, e pei o mataala, e taumafai foi mo se tau mautu ma taumatemate. O isi, e pei o talosaga a tagata, e ono mafua ai le pa, e ui e le o le tulaga lea mo le tele o galuega.
Su'ega uta
I le taimi o suʻega, sa ou taulai atu i le mafai ona faʻaputuina faʻamatalaga. Na ou faʻaaogaina le Prometheus 2.3.2 faʻatasi ma le Go 1.10.1 (o se vaega o le PMM 1.14) i luga o le auaunaga a Linode e faʻaaoga ai lenei tusitusiga: . Mo le gaosiga o uta sili ona moni, faʻaaoga lenei Na ou faʻalauiloaina nisi o MySQL faʻatasi ma se uta moni (Sysbench TPC-C Test), o ia mea taʻitasi e faʻataʻitaʻia 10 Linux/MySQL nodes.
O faʻataʻitaʻiga uma o loʻo mulimuli mai na faia i luga o le Linode server ma le valu faʻailoga mataʻutia ma le 32 GB o le manatua, faʻatautaia 20 uta faʻataʻitaʻiga e mataʻituina le lua selau MySQL faʻataʻitaʻiga. Po o, i le Prometheus faaupuga, 800 sini, 440 scrapes i le sekone, 380 afe faamaumauga i le sekone, ma le 1,7 miliona faʻasologa o taimi galue.
mamanu
O le masani masani o faʻamaumauga masani, e aofia ai le faʻaaogaina e Prometheus 1.x, o le . Afai e le lava le taulimaina o le uta, o le ae oʻo i tulaga maualuga ma o nisi talosaga o le a le manuia. O le fa'aogaina o le mafaufau ile Prometheus 2 e mafai ona fa'aogaina ile ki storage.tsdb.min-block-duration, lea e fuafua ai le umi o le a teu ai fa'amaumauga a'o le'i fufulu ile tisiki (e 2 itula le fa'aletonu). Ole aofa'i ole manatua ole a fa'alagolago ile numera o taimi fa'asologa, fa'ailoga, ma si'isi'i fa'aopoopo i le upega o lo'o sau. E tusa ai ma le avanoa o le disk, o le Prometheus e faʻamoemoe e faʻaaoga 3 bytes i faʻamaumauga (faʻataʻitaʻiga). I le isi itu, e sili atu le maualuga o manaʻoga manaʻomia.
E ui lava e mafai ona faʻapipiʻi le tele o poloka, e le fautuaina e faʻapipiʻi ma le lima, o lea e faʻamalosia ai oe e tuʻuina atu ia Prometheus le tele o manatua e manaʻomia mo lau galuega.
Afai e le lava le manatua e lagolago ai le tafe mai o metrics, Prometheus o le a le toe manatua pe o le a oʻo i ai le fasioti tagata OOM.
O le fa'aopoopoina o le swap e fa'atuai ai le fa'alavelave pe a uma le manatua o Prometheus e le fesoasoani tele, aua o le fa'aaogaina o lenei galuega e mafua ai le fa'aaogaina o mafaufauga. Ou te manatu o se mea e fai i le Go, lana ao lapisi ma le auala e feagai ai ma swap.
O le isi auala manaia o le faʻatulagaina lea o le poloka ulu e faʻafefe i le disk i se taimi, nai lo le faitauina mai le amataga o le faagasologa.
E pei ona e va'ai mai i le kalafi, e ta'i lua itula e fa'afefe ai le tisiki. Afai e te suia le min-block-duration parameter i le itula e tasi, ona toe faʻaleleia lea o le a tupu i itula uma, amata pe a maeʻa le afa itula.
Afai e te manaʻo e faʻaoga lenei mea ma isi kalafi i lau Prometheus faʻapipiʻi, e mafai ona e faʻaogaina lenei . Na mamanuina mo le PMM ae, faatasi ai ma ni suiga laiti, e fetaui i soʻo se faʻapipiʻi Prometheus.
O lo'o i ai la tatou poloka malosi e ta'ua o poloka ulu lea e teuina i le manatua; poloka ma fa'amaumauga tuai e maua e ala i mmap(). O lenei mea e faʻaumatia ai le manaʻoga e faʻapipiʻi ese le cache, ae o lona uiga foi e tatau ona e tuʻu se avanoa mo le faʻaogaina o le cache system pe afai e te manaʻo e suʻe faʻamatalaga e sili atu nai lo le mea e mafai e le poloka ulu ona faʻaogaina.
O lona uiga o le Prometheus virtual memory consumption o le a foliga maualuga, e le o se mea e popole ai.
O le isi manatu manaia o le mamanu o le faʻaaogaina o le WAL (tusi i luma ogalaau). E pei ona mafai ona e vaʻai mai i le teuina o faʻamaumauga, e faʻaaoga e Prometheus le WAL e aloese ai mai faʻalavelave. O auala fa'apitoa mo le fa'amautinoaina o fa'amaumauga e sao mai, o le mea e leaga ai, e le'o fa'amauina lelei. O le Prometheus version 2.3.2 e fufulu le WAL i le tisiki i 10 sekone ma o lenei filifiliga e le mafai ona faʻaogaina e tagata faʻaoga.
Fa'amau
Prometheus TSDB ua mamanuina e pei o se faleoloa LSM (Log Structured Merge): o le poloka ulu e faʻafefe i lea taimi ma lea taimi i le tisiki, aʻo se masini faʻapipiʻi e tuʻufaʻatasia le tele o poloka e aloese ai mai le suʻeina o le tele o poloka i taimi o fesili. O iinei e mafai ona e vaʻai i le numera o poloka na ou matauina i luga o le suʻega suʻega pe a uma le aso o le uta.
Afai e te manaʻo e aʻoaʻo atili e uiga i le faleoloa, e mafai ona e suʻesuʻeina le faila meta.json, o loʻo i ai faʻamatalaga e uiga i poloka o loʻo avanoa ma pe faʻapefea ona oʻo mai.
{
"ulid": "01CPZDPD1D9R019JS87TPV5MPE",
"minTime": 1536472800000,
"maxTime": 1536494400000,
"stats": {
"numSamples": 8292128378,
"numSeries": 1673622,
"numChunks": 69528220
},
"compaction": {
"level": 2,
"sources": [
"01CPYRY9MS465Y5ETM3SXFBV7X",
"01CPYZT0WRJ1JB1P0DP80VY5KJ",
"01CPZ6NR4Q3PDP3E57HEH760XS"
],
"parents": [
{
"ulid": "01CPYRY9MS465Y5ETM3SXFBV7X",
"minTime": 1536472800000,
"maxTime": 1536480000000
},
{
"ulid": "01CPYZT0WRJ1JB1P0DP80VY5KJ",
"minTime": 1536480000000,
"maxTime": 1536487200000
},
{
"ulid": "01CPZ6NR4Q3PDP3E57HEH760XS",
"minTime": 1536487200000,
"maxTime": 1536494400000
}
]
},
"version": 1
}Compactions i Prometheus o loʻo nonoa i le taimi e faʻafefe ai le poloka ulu ile tisiki. I le taimi nei, e mafai ona faia ni nai gaioiga.
E foliga mai e le fa'atapula'aina fa'atasi i so'o se auala ma e mafai ona fa'atupu ai le tele o tisiki I/O spikes i le taimi o le fa'atinoga.
Fa'atupu fa'aletonu o uta CPU
Ioe, e i ai se aafiaga le lelei i luga o le saoasaoa o le faiga, ma o se luʻitau ogaoga mo le teuina o le LSM: faʻafefea ona fai faʻapipiʻi e lagolago ai fua faatatau maualuga e aunoa ma le faʻateleina o le ulu?
O le faʻaogaina o le manatua i le faʻagasologa faʻapipiʻi e foliga mai e manaia tele.
E mafai ona tatou vaʻai pe faʻafefea, pe a maeʻa faʻapipiʻi, o le tele o le manatua e suia le setete mai le Cached i le Free: o lona uiga ua aveesea mai iina faʻamatalaga taua. E fia iloa pe fa'aoga iinei fadvice() po'o se isi faiga fa'aitiitiga, pe na mafua ona o le fa'aoga na fa'asa'olotoina mai poloka na fa'aumatia i le taimi o le fa'apipi'i?
Toe faʻaleleia pe a maeʻa se toilalo
O le toe faʻaleleia mai faʻaletonu e manaʻomia le taimi, ma mo mafuaaga lelei. Mo se vaitafe o loʻo oʻo mai o le miliona faʻamaumauga i le sekone, e tatau ona ou faʻatali pe a ma le 25 minute aʻo faia le toe faʻaleleia ma le amanaia o le SSD drive.
level=info ts=2018-09-13T13:38:14.09650965Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.2, branch=v2.3.2, revision=71af5e29e815795e9dd14742ee7725682fa14b7b)"
level=info ts=2018-09-13T13:38:14.096599879Z caller=main.go:223 build_context="(go=go1.10.1, user=Jenkins, date=20180725-08:58:13OURCE)"
level=info ts=2018-09-13T13:38:14.096624109Z caller=main.go:224 host_details="(Linux 4.15.0-32-generic #35-Ubuntu SMP Fri Aug 10 17:58:07 UTC 2018 x86_64 1bee9e9b78cf (none))"
level=info ts=2018-09-13T13:38:14.096641396Z caller=main.go:225 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-09-13T13:38:14.097715256Z caller=web.go:415 component=web msg="Start listening for connections" address=:9090
level=info ts=2018-09-13T13:38:14.097400393Z caller=main.go:533 msg="Starting TSDB ..."
level=info ts=2018-09-13T13:38:14.098718401Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536530400000 maxt=1536537600000 ulid=01CQ0FW3ME8Q5W2AN5F9CB7R0R
level=info ts=2018-09-13T13:38:14.100315658Z caller=web.go:467 component=web msg="router prefix" prefix=/prometheus
level=info ts=2018-09-13T13:38:14.101793727Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536732000000 maxt=1536753600000 ulid=01CQ78486TNX5QZTBF049PQHSM
level=info ts=2018-09-13T13:38:14.102267346Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536537600000 maxt=1536732000000 ulid=01CQ78DE7HSQK0C0F5AZ46YGF0
level=info ts=2018-09-13T13:38:14.102660295Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536775200000 maxt=1536782400000 ulid=01CQ7SAT4RM21Y0PT5GNSS146Q
level=info ts=2018-09-13T13:38:14.103075885Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1536753600000 maxt=1536775200000 ulid=01CQ7SV8WJ3C2W5S3RTAHC2GHB
level=error ts=2018-09-13T14:05:18.208469169Z caller=wal.go:275 component=tsdb msg="WAL corruption detected; truncating" err="unexpected CRC32 checksum d0465484, want 0" file=/opt/prometheus/data/.prom2-data/wal/007357 pos=15504363
level=info ts=2018-09-13T14:05:19.471459777Z caller=main.go:543 msg="TSDB started"
level=info ts=2018-09-13T14:05:19.471604598Z caller=main.go:603 msg="Loading configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499156711Z caller=main.go:629 msg="Completed loading of configuration file" filename=/etc/prometheus.yml
level=info ts=2018-09-13T14:05:19.499228186Z caller=main.go:502 msg="Server is ready to receive web requests."O le faʻafitauli autu o le toe faʻaleleia o le faʻaogaina o le mafaufau maualuga. E ui lava i le mea moni e faapea i se tulaga masani e mafai e le 'auʻaunaga ona galue faʻatasi ma le aofaʻi tutusa o le manatua, afai e paʻu atonu e le toe faʻaleleia ona o le OOM. Pau lava le fofo na ou maua o le faʻamalo le aoina o faʻamaumauga, aumai i luga le 'auʻaunaga, tuʻu ia toe faʻaleleia ma toe faʻafou ma faʻaogaina le aoina.
Faamafanafanaina
O le isi amioga e tatau ona manatua i le taimi o le mafanafana o le sootaga i le va o le maualalo o le faatinoga ma le maualuga o le faʻaaogaina o punaoa i le taimi lava e amata ai. I le taimi o nisi, ae le o mea uma e amata, na ou matauina se mamafa mamafa i luga o le PPU ma le manatua.
Gaps i le faʻaaogaina manatua e faʻaalia ai e le mafai e Prometheus ona faʻapipiʻi uma aoina mai le amataga, ma o nisi faʻamatalaga ua leiloa.
Ou te leʻi mafaufauina mafuaaga tonu mo le maualuga o le PPU ma le uta manatua. Ou te masalo o lenei mea e mafua ona o le fatuina o taimi fou faʻasologa i totonu o le poloka ulu ma le maualuga tele.
Faʻateleina uta CPU
I le faʻaopoopoga i faʻasalalauga, lea e fatuina ai se maualuga maualuga o le I / O, na ou matauina ni faʻalavelave matuia i le uta CPU i minute uma e lua. E umi atu le pa pe a maualuga le tafega o mea e fai ma e foliga mai e mafua mai i le aoina o lapisi a Go, ma e le itiiti ifo ma le tele o pusa o loʻo faʻatumu atoa.
O nei oso e le taua tele. E foliga mai pe a tupu ia mea, o le Prometheus's inside entry point ma metrics e le maua, ma mafua ai le va o faʻamatalaga i ia lava vaitaimi.
E mafai foi ona e maitauina o le Prometheus exporter ua tapunia mo le tasi le sekone.
E mafai ona tatou matauina fa'atasiga ma le aoina o lapisi (GC).
iʻuga
TSDB i Prometheus 2 e vave, e mafai ona faʻatautaia le faitau miliona o taimi faʻasologa ma i le taimi lava e tasi le afe o faʻamaumauga i le sekone e faʻaaoga ai meafaigaluega talafeagai. O le fa'aogaina o le PPU ma le disk I/O e mata'ina fo'i. O laʻu faʻataʻitaʻiga na faʻaalia i le 200 metrics i le sekone i le autu na faʻaaogaina.
Ina ia fuafua le faʻalauteleina, e tatau ona e manatua e uiga i le tele o le manatua, ma e tatau ona manatua moni. O le aofaʻi o manatuaga na faʻaaogaina na ou matauina e tusa ma le 5 GB i le 100 faʻamaumauga i le sekone o le vaitafe o loʻo oʻo mai, lea faʻatasi ai ma le faʻaogaina o le faʻaogaina o le cache na tuʻuina atu ai le 000 GB o le manatua.
O le mea moni, o loʻo i ai pea le tele o galuega e tatau ona faia e faʻafefe ai le PPU ma le disk I / O spikes, ma e le o se mea e ofo ai pe a faʻatusatusa le talavou TSDB Prometheus 2 i le InnoDB, TokuDB, RocksDB, WiredTiger, ae e tutusa uma lava. fa'afitauli i le amataga o latou olaga.
puna: www.habr.com