VictoriaMetrics yog DBMS ceev thiab scalable rau khaws cia thiab ua cov ntaub ntawv nyob rau hauv daim ntawv ntawm ib tug series ntawm lub sij hawm (cov ntaub ntawv muaj lub sij hawm thiab ib tug txheej ntawm cov nqi sib xws rau lub sij hawm no, piv txwv li, tau los ntawm lub sij hawm pov npav ntawm cov xwm txheej ntawm sensors los yog sau cov metrics).
Kuv lub npe yog Kolobaev Pavel. DevOps, SRE, LeroyMerlin, txhua yam zoo li cov cai - nws yog txhua yam hais txog peb: txog kuv thiab lwm tus neeg ua haujlwm LeroyMerlin.
Muaj huab raws li OpenStack. Muaj ib qhov txuas me me rau cov radar technical.
Nws yog tsim los ntawm Kubernetes kho vajtse, nrog rau tag nrho cov kev pabcuam cuam tshuam rau OpenStack thiab nkag.
Qhov no yog lub tswvyim uas peb muaj nyob rau hauv txoj kev loj hlob. Thaum peb tab tom tsim tag nrho cov no, peb muaj Prometheus tus neeg teb xov tooj uas khaws cov ntaub ntawv hauv K8s pawg nws tus kheej. Nws cia li pom dab tsi yuav tsum tau txhuam thiab muab tso rau hauv qab nws txhais taw, hais lus ntxhib.
Peb yuav tsum txav tag nrho cov ntaub ntawv tawm sab nraud Kubernetes pawg, vim tias yog tias muaj qee yam tshwm sim, peb yuav tsum nkag siab tias dab tsi thiab qhov twg.
Thawj qhov kev daws teeb meem yog tias peb siv lub koom haum thaum peb muaj ib tus neeg thib peb Prometheus, thaum peb mus rau Kubernetes pawg los ntawm lub koom haum mechanism.
Tab sis muaj qee qhov teeb meem me me ntawm no. Hauv peb qhov xwm txheej, cov teeb meem tau pib thaum peb muaj 250 qhov ntsuas, thiab thaum muaj 000 qhov ntsuas, peb pom tau tias peb tsis tuaj yeem ua haujlwm li ntawd. Peb nce scrape_timeout rau 400 vib nas this.
Vim li cas peb yuav tsum ua qhov no? Prometheus pib suav lub sijhawm los ntawm qhov pib ntawm lub laj kab. Nws tsis muaj teeb meem tias cov ntaub ntawv tseem ntws. Yog tias thaum lub sijhawm teev tseg no cov ntaub ntawv tsis sib koom ua ke thiab qhov kev sib kho tsis raug kaw ntawm http, ces qhov kev sib kho tau txiav txim siab ua tsis tiav thiab cov ntaub ntawv tsis nkag mus rau hauv Prometheus nws tus kheej.
Txhua tus paub txog cov duab uas peb tau txais thaum qee cov ntaub ntawv ploj lawm. Cov sij hawm raug torn thiab peb tsis txaus siab rau qhov no.
Qhov kev xaiv tom ntej yog sharding raws li ob qho sib txawv Prometheus los ntawm tib lub koom haum tswj hwm.
Piv txwv li, tsuas yog coj lawv thiab muab lawv los ntawm lub npe. Qhov no kuj tseem siv tau, tab sis peb txiav txim siab tsiv mus.
Tam sim no peb yuav tsum tau ua cov txheej txheem no qee yam. Koj tuaj yeem nqa promxy, uas mus rau thaj tsam shard thiab muab cov ntaub ntawv sib npaug. Nws ua haujlwm nrog ob lub shards ua ib qho chaw nkag. Qhov no tuaj yeem siv los ntawm promxy, tab sis nws tseem nyuaj dhau.
Thawj qhov kev xaiv yog tias peb xav tso tseg lub koom haum mechanism vim nws qeeb heev.
Cov neeg tsim tawm Prometheus tau hais meej meej tias, "Cov txiv neej, siv TimescaleDB sib txawv vim tias peb yuav tsis txhawb nqa cov kev ntsuas mus ntev." Qhov no tsis yog lawv txoj haujlwm.
Peb sau rau ntawm ib daim ntawv uas peb tseem yuav tsum tau tshem tawm sab nraud, yog li tsis txhob khaws txhua yam hauv ib qho chaw.
Qhov thib ob drawback yog nco noj. Yog lawm, kuv nkag siab tias ntau tus yuav hais tias xyoo 2020 ob peb gigabytes ntawm lub cim xeeb raug nqi ib lub nyiaj, tab sis tseem.
Tam sim no peb muaj ib puag ncig dev thiab khoom. Hauv dev nws yog li 9 gigabytes rau 350 metrics. Hauv cov khoom nws yog 000 gigabyte thiab me ntsis tshaj 14 metrics. Tib lub sijhawm, peb lub sijhawm khaws cia tsuas yog 780 feeb xwb. Qhov no phem. Thiab tam sim no kuv yuav piav qhia vim li cas.
Peb ua kev xam, uas yog, nrog ib thiab ib nrab lab metrics, thiab peb twb nyob ze rau lawv, nyob rau hauv lub tsim theem peb tau txais 35-37 gigabytes nco. Tab sis twb 4 lab metrics xav tau txog 90 gigabytes ntawm lub cim xeeb. Ntawd yog, nws tau suav nrog cov qauv muab los ntawm Prometheus cov neeg tsim khoom. Peb tau saib qhov kev sib raug zoo thiab pom tau tias peb tsis xav them ob peb lab rau ib tus neeg rau zaub mov tsuas yog saib xyuas.
Peb yuav tsis tsuas yog nce cov tshuab, peb kuj tseem saib xyuas cov tshuab virtual lawv tus kheej. Yog li ntawd, ntau lub tshuab virtual, ntau qhov kev ntsuas ntawm ntau yam, thiab lwm yam. Peb yuav muaj kev loj hlob tshwj xeeb ntawm peb pawg raws li kev ntsuas.
Nrog rau qhov chaw disk, tsis yog txhua yam phem heev ntawm no, tab sis kuv xav txhim kho nws. Peb tau txais tag nrho ntawm 15 gigabytes hauv 120 hnub, ntawm 100 yog cov ntaub ntawv compressed, 20 yog cov ntaub ntawv tsis txaus siab, tab sis peb ib txwm xav tau tsawg dua.
Raws li, peb sau ib qho ntxiv - qhov no yog kev siv ntau ntawm cov peev txheej, uas peb tseem xav txuag, vim tias peb tsis xav kom peb cov pawg saib xyuas kom noj ntau dua li peb pawg, uas tswj hwm OpenStack.
Muaj ib qho teeb meem ntxiv ntawm Prometheus, uas peb tau txheeb xyuas rau peb tus kheej, qhov no tsawg kawg yog qee yam kev nco txwv. Nrog Prometheus, txhua yam yog qhov phem dua ntawm no, vim tias nws tsis muaj qhov sib tw zoo li no. Siv qhov txwv hauv docker kuj tsis yog ib qho kev xaiv. Yog tias mam li nco dheev koj RAF poob thiab muaj 20-30 gigabytes, ces nws yuav siv sij hawm ntev heev.
Qhov no yog lwm qhov laj thawj vim li cas Prometheus tsis haum rau peb, piv txwv li peb tsis tuaj yeem txwv kev siv lub cim xeeb.
Nws yuav muaj peev xwm tuaj nrog cov tswv yim zoo li no. Peb xav tau cov tswv yim no txhawm rau txhawm rau txhim kho HA pawg. Peb xav kom peb cov metrics muaj nyob ib txwm thiab txhua qhov chaw, txawm tias lub server uas khaws cov ntsuas no poob. Thiab yog li peb yuav tau tsim kom muaj lub tswv yim zoo li no.
Cov txheej txheem no hais tias peb yuav muaj duplication ntawm shards, thiab, raws li, duplication ntawm tus nqi ntawm cov khoom siv. Nws tuaj yeem scaled yuav luag horizontally, tab sis txawm li cas los xij, kev siv cov peev txheej yuav yog ntuj raug txim.
Disadvantages nyob rau hauv kev txiav txim nyob rau hauv daim ntawv uas peb sau lawv rau peb tus kheej:
- Yuav tsum tau upload cov metrics sab nraud.
- Kev siv nyiaj ntau.
- Tsis muaj txoj hauv kev los txwv kev siv lub cim xeeb.
- Kev siv HA complex thiab muaj peev xwm siv tau.
Rau peb tus kheej, peb txiav txim siab tias peb tau tsiv tawm ntawm Prometheus ua qhov chaw khaws cia.
Peb tau txheeb xyuas cov kev xav tau ntxiv rau peb tus kheej uas peb xav tau. Qhov no:
- Qhov no yog kev txhawb nqa promql, vim hais tias ntau yam twb tau sau rau Prometheus: lus nug, ceeb toom.
- Thiab ces peb muaj Grafana, uas twb tau sau nyob rau hauv raws nraim tib txoj kev rau Prometheus raws li ib tug backend. Kuv tsis xav rov sau dua dashboards.
- Peb xav tsim kom muaj ib qho HA architecture.
- Peb xav txo qhov kev siv ntawm ib qho kev pab.
- Muaj lwm qhov me me nuance. Peb tsis tuaj yeem siv ntau hom huab metrics sau cov tshuab. Peb tsis paub dab tsi yuav poob rau hauv cov ntsuas no. Thiab txij li txhua yam tuaj yeem ya mus rau ntawd, peb yuav tsum txwv peb tus kheej mus rau qhov chaw nyob hauv zos.
Muaj kev xaiv me me. Peb sau txhua yam uas peb muaj kev paub nrog. Peb tau saib ntawm nplooj ntawv Prometheus hauv ntu kev koom ua ke, nyeem ib pawg ntawm cov ntawv, thiab pom dab tsi tshwm sim. Thiab rau peb tus kheej, peb xaiv VictoriaMetrics los hloov rau Prometheus.
Vim li cas? Vim:
- Paub promql.
- Muaj ib tug modular architecture.
- Tsis tas yuav hloov Grafana.
- Thiab qhov tseem ceeb tshaj plaws, peb yuav zaum muab cov metrics khaws cia hauv peb lub tuam txhab raws li kev pabcuam, yog li peb tab tom saib ua ntej ntawm kev txwv ntau yam kom cov neeg siv tuaj yeem siv tag nrho cov peev txheej ntawm pawg hauv qee txoj kev txwv, vim tias muaj sijhawm. tias nws yuav multitenancy.
Cia peb ua thawj qhov sib piv. Peb coj tib Prometheus hauv pawg, sab nraud Prometheus mus rau nws. Ntxiv ntawm remoteWrite VictoriaMetrics.
Kuv mam li ua qhov kev tshwj tseg tam sim ntawd ntawm no peb ntes tau nce me ntsis hauv CPU noj los ntawm VictoriaMetrics. VictoriaMetrics wiki qhia koj qhov kev ntsuas twg yog qhov zoo tshaj plaws. Peb tau kuaj xyuas lawv. Lawv tau txo CPU noj zoo heev.
Nyob rau hauv peb cov ntaub ntawv, lub cim xeeb noj ntawm Prometheus, uas yog nyob rau hauv lub Kubernetes pawg, tsis tau nce loj.
Peb sib piv ob qhov chaw ntawm cov ntaub ntawv tib yam. Hauv Prometheus peb pom tib cov ntaub ntawv uas ploj lawm. Txhua yam zoo ntawm VictoriaMetrics.
Cov txiaj ntsig ntawm qhov chaw disk. Peb ntawm Prometheus tau txais 120 gigabytes nyob rau hauv tag nrho. Ntawm VictoriaMetrics peb twb tau txais 4 gigabyte ib hnub. Muaj ib qho txawv me ntsis dua li qhov peb tau siv los pom hauv Prometheus. Uas yog, cov ntaub ntawv twb compressed zoo heev nyob rau hauv ib hnub, nyob rau hauv ib nrab ib teev. Lawv twb tau reaped zoo nyob rau hauv ib hnub, nyob rau hauv ib nrab ib teev, txawm lub fact tias cov ntaub ntawv tseem yuav ploj tom qab. Yog li ntawd, peb tau txais kev cawmdim ntawm qhov chaw disk.
Peb kuj txuag ntawm kev siv lub cim xeeb. Thaum lub sijhawm sim, peb tau siv Prometheus rau ntawm lub tshuab virtual - 8 cores, 24 gigabytes. Prometheus noj yuav luag txhua yam. Nws poob rau OOM Killer. Nyob rau tib lub sijhawm, tsuas yog 900 cov kev ntsuas nquag tau nchuav rau hauv nws. Qhov no yog hais txog 000-25 metrics ib ob.
Peb khiav VictoriaMetrics ntawm dual-core virtual tshuab nrog 8 gigabyte ntawm RAM. Peb tau tswj hwm kom tau txais VictoriaMetrics ua haujlwm zoo los ntawm kev sib tw nrog ob peb yam ntawm lub tshuab 8GB. Thaum kawg, peb khaws cia rau 7 gigabyte. Nyob rau tib lub sijhawm, qhov ceev ntawm kev xa cov ntsiab lus, piv txwv li kev ntsuas, tau siab dua li ntawm Prometheus.
CPU tau dhau los ua zoo dua piv rau Prometheus. Ntawm no Prometheus siv 2,5 cores, thiab VictoriaMetrics tsuas yog siv 0,25 cores. Thaum pib - 0,5 cores. Raws li nws sib koom ua ke, nws ncav cuag ib lub hauv paus, tab sis qhov no yog qhov tsis tshua muaj, tsis tshua muaj.
Hauv peb qhov xwm txheej, qhov kev xaiv poob rau VictoriaMetrics rau qhov laj thawj pom tseeb; peb xav txuag nyiaj thiab peb tau ua.
Cia peb hla tawm ob lub ntsiab lus tam sim ntawd - kev xa tawm ntawm kev ntsuas thiab kev siv nyiaj ntau. Thiab peb tsuas yog yuav tsum txiav txim siab ob lub ntsiab lus uas peb tseem tshuav rau peb tus kheej.
Ntawm no kuv yuav ua ib qho kev tshwj tseg tam sim ntawd, peb xav txog VictoriaMetrics raws li kev khaws cia ntawm kev ntsuas. Tab sis txij li thaum peb feem ntau yuav muab VictoriaMetrics cia rau tag nrho Leroy, peb yuav tsum txwv cov neeg uas yuav siv pawg no kom lawv tsis pub rau peb.
Nws muaj qhov zoo tshaj plaws parameter uas tso cai rau koj los txwv lub sijhawm, los ntawm ntim cov ntaub ntawv thiab lub sijhawm ua tiav.
Kuj tseem muaj qhov kev xaiv zoo tshaj plaws uas tso cai rau peb txwv kev siv lub cim xeeb, yog li peb tuaj yeem nrhiav qhov sib npaug heev uas yuav ua rau peb tau txais kev khiav hauj lwm ceev thiab txaus siv cov peev txheej.
Tshem tawm ib qho ntxiv, piv txwv li hla lub ntsiab lus - koj tsis tuaj yeem txwv kev siv lub cim xeeb.
Hauv thawj iterations, peb sim VictoriaMetrics Ib Node. Tom ntej no peb tsiv mus rau VictoriaMetrics Cluster Version.
Ntawm no peb muaj tes dawb los cais cov kev pabcuam sib txawv hauv VictoriaMetrics nyob ntawm seb lawv yuav khiav dab tsi thiab cov peev txheej twg lawv yuav siv. Qhov no yog ib qho yooj yim heev thiab yooj yim daws. Peb siv qhov no ntawm peb tus kheej.
Cov khoom tseem ceeb ntawm VictoriaMetrics Cluster Version yog vmstsorage. Muaj peev xwm muaj N tus lej ntawm lawv. Hauv peb qhov xwm txheej tam sim no muaj 2 ntawm lawv.
Thiab muaj vminsert. Qhov no yog lub npe neeg rau zaub mov uas tso cai rau peb: npaj sharding ntawm txhua qhov chaw khaws cia uas peb tau hais txog, thiab nws kuj tso cai rau ib qho kev hloov pauv, piv txwv li koj yuav muaj ob qho tib si sharding thiab replica.
Vminsert txhawb nqa OpenTSDB, Graphite, InfluxDB thiab remoteWrite raws tu qauv los ntawm Prometheus.
Tseem muaj vmselect. Nws lub luag haujlwm tseem ceeb yog mus rau vmstorage, txais cov ntaub ntawv los ntawm lawv, deduplicate cov ntaub ntawv no thiab muab rau tus neeg siv khoom.
Muaj ib qho zoo heev hu ua vmagent. Peb nyiam nws heev. Nws tso cai rau koj los teeb tsa raws nraim li Prometheus thiab tseem ua txhua yam raws nraim li Prometheus. Ntawd yog, nws sau cov kev ntsuas los ntawm cov chaw sib txawv thiab cov kev pabcuam thiab xa mus rau vminsert. Ces txhua yam nyob ntawm koj.
Lwm qhov kev pabcuam zoo yog vmalert, uas tso cai rau koj siv VictoriaMetrics ua tus thaub qab, tau txais cov ntaub ntawv tiav los ntawm vminsert thiab xa mus rau vmselect. Nws ua cov lus ceeb toom lawv tus kheej, nrog rau cov cai. Nyob rau hauv cov ntaub ntawv ntawm kev ceeb toom, peb tau txais kev ceeb toom los ntawm alertmanager.
Muaj ib tug wmauth tivthaiv. Tej zaum peb yuav los yog tsis tau (peb tsis tau txiav txim siab txog qhov no tsis tau) siv nws ua qhov kev tso cai rau ntau qhov kev sib tw ntawm pawg. Nws txhawb remoteWrite rau Prometheus thiab tuaj yeem tso cai raws li qhov url, lossis qhov thib ob ntawm nws, qhov twg koj tuaj yeem sau tsis tau.
Kuj tseem muaj vmbackup, vmrestore. Qhov no yog, nyob rau hauv qhov tseeb, lub restoration thiab backup ntawm tag nrho cov ntaub ntawv. Yuav ua tau S3, GCS, ntaub ntawv.
Thawj qhov rov ua dua ntawm peb pawg tau ua thaum lub sijhawm cais tawm. Lub sijhawm ntawd, tsis muaj qhov hloov pauv, yog li peb qhov kev rov ua dua muaj ob pawg sib txawv thiab ywj pheej rau hauv uas peb tau txais cov ntaub ntawv ntawm remoteWrite.
Ntawm no kuv yuav ua kom tau txais kev tshwj tseg tias thaum peb hloov los ntawm VictoriaMetrics Ib Node rau VictoriaMetrics Cluster Version, peb tseem nyob nrog tib cov khoom siv, piv txwv li qhov tseem ceeb yog lub cim xeeb. Qhov no yog kwv yees li cas peb cov ntaub ntawv, piv txwv li kev siv peev txheej, tau muab faib.
Ib tug replica twb tau muab tso rau ntawm no. Peb muab tag nrho cov no rau hauv ib pawg loj heev. Tag nrho peb cov ntaub ntawv yog ob qho tib si sharded thiab replicated.
Tag nrho pawg muaj N cov ntsiab lus nkag, lub ntsiab lus Prometheus tuaj yeem ntxiv cov ntaub ntawv ntawm HAPROXY. Ntawm no peb muaj qhov nkag nkag. Thiab los ntawm qhov nkag nkag no koj tuaj yeem nkag mus hauv Grafana.
Hauv peb qhov xwm txheej, HAPROXY yog tib qhov chaw nres nkoj uas cov neeg xaiv tsa xaiv, ntxig thiab lwm yam kev pabcuam hauv pawg no. Hauv peb qhov xwm txheej, nws tsis tuaj yeem ua ib qhov chaw nyob; peb yuav tsum ua ntau lub ntsiab lus nkag, vim tias cov tshuab virtual lawv tus kheej uas cov VictoriaMetrics pawg khiav nyob rau hauv ntau thaj tsam ntawm tib tus neeg muab huab, piv txwv li tsis nyob hauv peb huab, tab sis sab nraud. .
Peb muaj kev ceeb toom. Peb siv nws. Peb siv alertmanager los ntawm Prometheus. Peb siv Opsgenie thiab Telegram raws li kev ceeb toom xa xov xwm. Hauv Telegram lawv nchuav los ntawm dev, tej zaum ib yam dab tsi los ntawm cov khoom, tab sis feem ntau yog ib yam dab tsi txheeb cais, xav tau los ntawm engineers. Thiab Opsgenie tseem ceeb heev. Cov no yog hu, kev tswj qhov xwm txheej.
Lo lus nug nyob mus ib txhis: “Leej twg saib xyuas kev saib xyuas?” Hauv peb qhov xwm txheej, kev soj ntsuam xyuas saib xyuas nws tus kheej, vim peb siv vmagent ntawm txhua qhov. Thiab txij li thaum peb cov nodes tau faib thoob plaws cov chaw sib txawv ntawm cov chaw muab kev pabcuam tib yam, txhua lub chaw pabcuam muaj nws tus kheej channel, lawv muaj kev ywj pheej, thiab txawm tias lub hlwb tawg tuaj txog, peb tseem yuav tau txais kev ceeb toom. Yog, yuav muaj ntau ntawm lawv, tab sis nws yog qhov zoo dua kom tau txais kev ceeb toom ntau dua li tsis muaj.
Peb xaus peb cov npe nrog kev siv HA.
Thiab ntxiv kuv xav kom nco ntsoov qhov kev paub ntawm kev sib txuas lus nrog VictoriaMetrics zej zog. Nws muab tawm zoo heev. Cov txiv neej no teb. Lawv sim delve rau hauv txhua rooj plaub uas muaj.
Kuv pib teeb meem ntawm GitHub. Lawv tau daws sai heev. Muaj ob peb qhov teeb meem ntxiv uas tsis tau kaw tag nrho, tab sis kuv tuaj yeem pom los ntawm cov cai uas ua haujlwm hauv qhov kev taw qhia no tab tom pib.
Qhov mob tseem ceeb rau kuv thaum iterations yog tias yog tias kuv kaw qhov node, ces thawj 30 vib nas this vminsert tsis tuaj yeem nkag siab tias tsis muaj backend. Qhov no tau tam sim no tau txiav txim siab. Thiab lus nyob rau hauv ib tug thib ob los yog ob, cov ntaub ntawv yog muab los ntawm tag nrho cov seem nodes, thiab qhov kev thov nres tos rau qhov uas ploj lawm node.
Qee lub sij hawm peb xav kom VictoriaMetrics ua tus tswv VictoriaMetrics. Peb tos nws. Peb tam sim no nquag tsim lub moj khaum rau VictoriaMetrics tus neeg teb xov tooj kom coj tag nrho cov kev cai ua ntej xam, thiab lwm yam Prometheus, vim peb nquag siv cov kev cai uas tuaj nrog Prometheus tus neeg teb xov tooj.
Muaj cov tswv yim los txhim kho qhov kev siv ntawm pawg. Kuv piav lawv saum toj no.
Thiab kuv yeej xav downsample. Nyob rau hauv peb cov ntaub ntawv, downsampling yog xav tau tshwj xeeb rau kev saib cov tiam sis. Hais lus ntxhib, ib qho metric txaus rau kuv thaum nruab hnub. Cov qauv no xav tau rau ib xyoos, peb, tsib, kaum xyoo. Thiab ib qho metric tus nqi yog txaus txaus.
- Peb tau paub qhov mob, zoo li muaj qee tus ntawm peb cov npoj yaig, thaum siv Prometheus.
- Peb xaiv VictoriaMetrics rau peb tus kheej.
- Nws scales zoo heev ob leeg vertically thiab horizontally.
- Peb tuaj yeem faib cov khoom sib txawv rau ntau tus lej ntawm cov nodes hauv pawg, txwv lawv los ntawm kev nco, ntxiv kev nco, thiab lwm yam.
Peb yuav siv VictoriaMetrics hauv tsev vim peb nyiam heev. Qhov no yog dab tsi thiab dab tsi tau ua.
Ob peb ntawm QR cov lej rau VictoriaMetrics tham, kuv cov neeg sib cuag, LeroyMerlin technical radar.
Tau qhov twg los: www.hab.com