He DBMS tere, ka taea te whakawhanui a VictoriaMetrics mō te penapena me te tukatuka i ngā raraunga raupapa-wā (he rekoata e tohu ana i te wā me te huinga uara e rite ana ki taua wā, hei tauira, i whiwhihia mā te rangahau ā-wā o te mana o te pūoko, te kohinga ine rānei).
Ko Pavel Kolobaev tōku ingoa. DevOps, SRE, LeroyMerlin, ngā mea katoa pērā i te waehere – koinā katoa mō mātou: ko au me ētahi atu kaimahi a LeroyMerlin.
He kapua kei runga i te OpenStack. He hononga iti ki TechRadar.
He mea hanga i runga i ngā taputapu Kubernetes, tae atu ki ngā ratonga OpenStack me ngā ratonga takiuru katoa e haere tahi ana.
Koinei te whakatakotoranga i whakamahia e mātou i te wā whakawhanake. I a mātou e whakawhanake ana i ēnei mea katoa, he kaiwhakahaere Prometheus tā mātou i penapena raraunga i roto i te rōpū K8s. Ka kitea aunoatia e ia ngā mea e hiahiatia ana kia waruhia, ā, ka rongoatia, i te nuinga o te wā.
Me nuku e tātou ngā raraunga katoa ki waho i te tautau Kubernetes, nā te mea ki te puta he mea, me mārama tātou ki te aha me te wāhi.
Ko te otinga tuatahi ko te whakamahi i te federation ina he Prometheus tuatoru tā mātou, ina haere mātou ki te rōpū Kubernetes mā te tikanga federation.
Engari he iti noa ngā take i konei. I roto i tā mātou take, i tīmata ngā raruraru i te wā i 250,000 ā mātou inenga, ā, i te taenga ki te 400,000 inenga, ka kite mātou kāore e taea e mātou te mahi pērā. I whakanuia e mātou te scrape_timeout ki te 25 hēkona.
He aha mātou i mahi ai i tēnei? Ka tīmata a Prometheus ki te tatau i te wā mutu mai i te wā i kohia ai ngā raraunga. Ahakoa kei te rere tonu ngā raraunga, kāore he aha. Ki te kore ngā raraunga e kohia i roto i tēnei wā kua tohua, ā, ki te kore te wātū e kati mā HTTP, ka kiia te wātū kua rahua, ā, kāore ngā raraunga e tukuna ki a Prometheus tonu.
E mōhio ana tātou katoa ki ngā kauwhata e puta mai ana ina ngaro ētahi raraunga. He kopikopiko ngā kauwhata, ā, kāore tātou e harikoa ki tēnā.
Ko te kōwhiringa e whai ake nei ko te wehewehe i runga i ngā Prometheus rerekē e rua mā te tikanga whakakotahitanga kotahi.
Hei tauira, ka taea e tātou te wehewehe noa i a rātou mā te ingoa. Ka taea hoki tēnei te whakamahi, engari i whakatau tātou kia neke atu.
Inaianei me whakahaere e tātou ēnei maramara i tētahi ara. Ka taea e tātou te whakamahi i te promxy, e uru atu ana ki te rohe maramara, ā, ka whakareatia ngā raraunga. Ka mahi tahi me ngā maramara e rua hei pūwāhi whakauru kotahi. Ka taea te whakatinana i tēnei mā te whakamahi i te promxy, engari he uaua rawa i nāianei.
Ko te kōwhiringa tuatahi ko te whakarērea o te tikanga whakakotahitanga nā te mea he tino puhoi.
E mārama ana te kī a ngā kaiwhakawhanake o Prometheus, "E tama mā, whakamahia ētahi atu TimescaleDB nā te mea kāore mātou e tautoko i te rokiroki inenga mō te wā roa." Ehara tēnā i tā rātou mahi. 
Ka tuhia e mātou ki runga i tētahi pepa me tango tonu e mātou ki waho, kia kore ai e rongoatia ngā mea katoa ki te wāhi kotahi.
Ko te ngoikoretanga tuarua ko te whakapaunga mahara. Ae, e mārama ana ahau he tokomaha ka kī i te tau 2020, he iti noa te utu o ētahi kikapaita mahara, engari ahakoa.
He taiao whakawhanake me te taiao whakaputa kei a mātou i tēnei wā. I roto i te whakawhanake, e 9 kikapaita te rahi me te 350,000 inenga. I roto i te whakaputa, he iti iho i te 14 kikapaita te rahi me te 780,000 inenga. I taua wā, e 30 meneti noa iho te roa o tō mātou pupuri. He kino tēnā. Māku e whakamārama te take.
Kei te mahi mātou i ngā tātaitanga, arā, me te 1.5 miriona inenga—ā, kua tata kē mātou ki tēnā—i te wāhanga hoahoa, ka whiwhi mātou i te 35-37 kikapaita o te mahara. Engari mō te 4 miriona inenga, kua tata ki te 90 kikapaita te hiahiatanga o te mahara. Arā, i tātaihia tēnei mā te whakamahi i te tātai i whakaratohia e ngā kaiwhakawhanake Prometheus. I tirohia e mātou te hononga, ā, i kite mātou kāore mātou i hiahia ki te utu i te rua miriona mō te tūmau hei aroturuki noa iho.
Ehara i te mea ko te whakanui ake i te maha o ngā mīhini anake; engari ka aroturukihia anō hoki e mātou ngā mīhini mariko. Nō reira, ki te nui ake ā mātou mīhini mariko, ka nui ake ā mātou inenga, me ētahi atu. Ka kite mātou i te tipu motuhake o tā mātou rōpū inenga.
Kāore i te kino rawa te wāhi kōpae, engari me whakapai ake. I tae mai te katoa o te 120 kikapaita i roto i ngā rā 15, 100 o ēnei he raraunga kōpeke, 20 kāore i kōpeketia, engari he iti iho te hiahia.
Nō reira, ka tuhia e mātou tētahi atu kōrero - he nui te whakapaunga rauemi tēnei, e hiahia tonu ana mātou ki te penapena, nā te mea kāore mātou e hiahia kia nui ake ngā rauemi e pau ana i tā mātou rōpū aroturuki i tā mātou rōpū e whakahaere ana i a OpenStack.
Tērā tētahi atu ngoikoretanga o Prometheus kua kitea e mātou: tētahi momo rohe mahara. Ki a Prometheus, he kino rawa atu ngā āhuatanga nā te mea kāore he mana whakahaere pēnei. Kāore hoki he kōwhiringa ki te whakamahi i tētahi rohe i roto i te Docker. Ki te hinga tō RAF, ā, ka eke ki te 20-30 kigabytes, ka roa te wā e ora ai.
Ko tētahi atu take tēnei kāore a Prometheus i te pai mō tātou, arā, kāore e taea te whakawhāiti i te whakamahinga o te mahara.
Ka taea e tātou te whakaaro i tētahi hoahoa pēnei. Me whakamahi tēnei hoahoa hei whakatū i tētahi rōpū HA. E hiahia ana tātou kia wātea ā tātou inenga i ngā wāhi katoa, i ngā wā katoa, ahakoa ka pakaru te tūmau e penapena ana i aua inenga. Nō reira, me hanga e tātou tētahi hoahoa pēnei.
Ko te tikanga o tēnei kaupapa ka tāruatia ngā maramara, ā, ka tāruatia te whakamahinga rauemi. Tata tonu ka taea te whakawhanui whakapae, engari ka nui rawa tonu te whakamahinga rauemi.
Ngā ngoikoretanga i te raupapa kua tuhia e mātou mā mātou anō:
- He mea tika kia kaweakehia ngā inenga ki waho.
- Te nui o te whakamahinga o ngā rauemi.
- Kāore e taea te whakawhāiti i te whakamahinga mahara.
- He whakatinanatanga uaua, he nui hoki ngā rauemi e hiahiatia ana mō te HA.
I whakatau mātou mā mātou anō ka neke atu mātou i a Prometheus hei otinga rokiroki.
Kua tautuhia e mātou ētahi atu whakaritenga mā mātou anō. Ko ēnei:
- He tautoko promql tēnei, nā te mea he maha ngā mea kua tuhia mō Prometheus: ngā patai, ngā matohi.
- Ā, kei a tātou a Grafana, i tuhia hoki mō Prometheus hei papamuri. Kāore au e hiahia ki te tuhi anō i ngā papaaho whakahaere.
- E hiahia ana mātou ki te hanga i tētahi hoahoa HA noa.
- E hiahia ana mātou ki te whakaiti i te whakamahinga o ngā rauemi katoa.
- Tētahi atu kōrero iti anō. Kāore e taea e mātou te whakamahi i tētahi momo pūnaha kohikohi inenga kapua. Kāore anō mātou kia mōhio he aha te mutunga ki roto i ēnei inenga. Ā, nā te mea ka mutu pea tetahi mea ki reira, me whakawhāiti noa mātou ki te manaaki ā-rohe.
He iti noa ngā kōwhiringa. I kohia e mātou ngā mea katoa i mōhiotia e mātou. I tirohia e mātou te whārangi Prometheus i te wāhanga whakauru, i pānuihia he maha ngā tuhinga, ā, i tirohia hoki ngā mea e wātea ana. Ā, i whiriwhiria e mātou a VictoriaMetrics hei whakakapi mō mātou mō Prometheus.
He aha? Nā te mea:
- Ka taea te whakamahi i te promql.
- He hoahoa modular kei reira.
- Kāore he huringa e hiahiatia ana i roto i a Grafana.
- Ā, ko te mea nui, kei te whakaratohia pea e mātou he ratonga rokiroki inenga i roto i tā mātou kamupene, nō reira kei te titiro whakamua mātou ki ngā momo here kia taea ai e ngā kaiwhakamahi te whakamahi i ngā rauemi katoa o te taupū i roto i tētahi huarahi iti, i te mea he tūponotanga ka maha-kairēhita.
Me whakarite tā tātou whakatairite tuatahi. Ka whakamahia e tātou te Prometheus kotahi i roto i te rōpū, ka taea te uru atu mā te Prometheus o waho. Ka tāpirihia e tātou a VictoriaMetrics mā te remoteWrite.
Me whakamārama tonu atu ahau i kite mātou i tētahi pikinga paku o te whakapaunga CPU mai i a VictoriaMetrics i konei. E whakamārama ana te wiki VictoriaMetrics ko ēhea ngā tautuhinga pai rawa atu. I whakamatauria e mātou, ā, i tino whakaitihia te whakapaunga CPU.
I tō mātou take, kāore i tino piki ake te whakapaunga mahara o Prometheus, kei roto i te tautau Kubernetes.
Kei te whakataurite mātou i ngā pūtake raraunga e rua mō ngā raraunga ōrite. I roto i a Prometheus, ka kitea e mātou ngā raraunga ngaro ōrite. I roto i a VictoriaMetrics, he pai ngā mea katoa.
Ngā hua whakamātautau mokowā kōpae. I whiwhi mātou i te 120 kikapaita katoa i roto i te Prometheus. I roto i te VictoriaMetrics, kua whiwhi kē mātou i te 4 kikapaita ia rā. He rerekē te tikanga o tā rātou i tā mātou i whakamahi ai i roto i te Prometheus. Arā, kua tino pai te kōpeketanga o ngā raraunga i roto i te rā kotahi, i roto i te hawhe haora. Kua tino pai te kōpeketanga i roto i te rā kotahi, i roto i te hawhe haora, ahakoa ka whakakotahihia e mātou ngā raraunga ā muri ake nei. Nō reira, i penapenahia e mātou te mokowā kōpae.
Ka penapena hoki mātou i te whakapaunga mahara. I te wā o te whakamātautau, i whakatūria a Prometheus ki runga i tētahi mīhini mariko—8 ngā matua, 24 kikapaita. Tata ki ngā mea katoa ka pau i a Prometheus. I hinga nā te OOM Killer. Heoi, e 900,000 noa iho ngā inenga hohe i whiwhihia. Tata ki te 25,000-27,000 inenga ia hekona.
I whakahaerehia e mātou a VictoriaMetrics i runga i tētahi mīhini mariko matua-rua me te 8 GB o te RAM. I taea e mātou te whakahaere pai i a VictoriaMetrics mā te whakarerekē i ētahi mea i runga i te mīhini 8 GB. I te mutunga, i taea e mātou te pupuri i te pūnaha i raro i te 7 GB. I taua wā anō, i tutuki i a mātou ngā tere tuku ihirangi, arā, ngā inenga, he teitei ake i ērā o Prometheus.
Kua tino pai ake te mahi a te PTM ki te whakataurite ki a Prometheus. I konei, e 2,5 ngā matua e pau ana i a Prometheus, ko VictoriaMetrics ia e 0,25 noa iho ngā matua e pau ana. I te whakarewatanga, he 0,5 ngā matua. I te haere tonu o te whakakotahitanga, ka eke ki te kotahi te matua, engari he tino onge tēnei.
Mō tā mātou take, i whiriwhiria e mātou a VictoriaMetrics mō ngā take mārama: i hiahia mātou ki te penapena moni, ā, i pērā mātou.
Ka whakakorea e mātou ngā take e rua i tēnei wā tonu: te tango i ngā inenga me te nui o te whakamahinga rauemi. Ā, e rua atu anō ngā take hei whakatika.
Kia whakamārama atu ahau i tēnei wā tonu: e titiro ana mātou ki a VictoriaMetrics hei putunga raraunga. Engari, nā te mea ka whakaratohia pea e mātou a VictoriaMetrics hei putunga raraunga mō Leroy katoa, me whakawhāiti e mātou te hunga e whakamahi ana i tēnei rōpū kia kore ai rātou e muku.
He tawhā nui tēnei e āhei ai koe ki te whakawhāiti mā te wā, mā te rahinga raraunga me te wā whakatinanatanga.
He kōwhiringa pai anō hoki tēnei e āhei ai koe ki te whakawhāiti i te whakamahinga mahara, kia kitea ai e tātou te taurite tino tika hei whakatutuki i te tere whakahaere noa me te whakamahinga rauemi e tika ana.
Tangohia tētahi atu tohu, arā, ka whitihia e mātou te tohu - kāore e taea te whakawhāiti i te whakamahinga mahara.
I ā mātou whakamātautau tuatahi, i whakamatauria e mātou te Pūnga Kotahi o VictoriaMetrics. Muri iho, ka neke atu mātou ki te Putanga Taupū o VictoriaMetrics.
Mā tēnei ka taea e mātou te wehewehe i ngā ratonga rerekē i roto i a VictoriaMetrics i runga i te mea ka whakahaerehia e rātou me ngā rauemi ka pau i a rātou. He otinga tino ngāwari, he otinga watea hoki. Kua whakamahia e mātou anō.
Ko ngā wāhanga matua o te Putanga Taupū o VictoriaMetrics ko te vmstsorage. E N pea ēnei. Mō tā mātou take, e rua i tēnei wā.
Ā, kei reira a vminsert. He tūmau takawaenga e āhei ai tātou ki te whakarite i te wehewehenga i waenga i ngā rokiroki katoa e kōrerotia ana e tātou, ā, e āhei ai hoki te tāruarua, arā, ka whai wehewehenga me te tāruarua koe.
Ka tautoko a Vminsert i ngā kawa OpenTSDB, Graphite, InfluxDB, me Prometheus's remoteWrite.
Kei reira anō hoki a vmselect. Ko tana mahi matua he uru atu ki a vmstorage, te tiki raraunga mai i reira, te tango i te tāruarua, me te whakahoki atu ki te kiritaki.
He mea whakamiharo tēnei e kiia nei ko vmagent. He tino pai ki a mātou. Mā tēnei ka taea e koe te whirihora kia rite ki a Prometheus, ā, ka mahi tonu i ngā mea katoa kia rite ki a Prometheus. Arā, ka kohia e ia ngā inenga mai i ngā momo hinonga me ngā ratonga ka tukuna atu ki vminsert. Mai i reira, kei a koe te tikanga.
Ko tētahi atu ratonga pai ko vmalert, e whakamahi ana i a VictoriaMetrics hei papamuri, e whiwhi ana i ngā raraunga mai i vminsert, ā, e tuku ana i ngā raraunga kua tukatukahia ki vmselect. Ka tukatukahia e ia ngā matohi me ngā ture. Mō ngā matohi, ka whiwhi mātou i aua matohi mā alertmanager.
He wāhanga wmauth tēnei. Tērā pea ka whakamahia, kāore rānei (kāore anō kia whakatauhia) hei pūnaha whakamana mō ngā tautau maha-rōpū. E tautoko ana i te remoteWrite mō Prometheus, ā, ka taea te whakamana i runga i te URL, i te wāhanga tuarua rānei, e tohu ana i te wāhi ka taea, kāore rānei e taea e koe te tuhi.
Kei reira anō hoki a vmbackup me vmrestore. Ko te tikanga, ka whakaora, ka tārua hoki ēnei i ō raraunga katoa. Ka tautokohia a S3, GCS, me ngā kōnae.
I hangaia te putanga tuatahi o tā mātou rōpū i te wā o te taratahi. I taua wā, kāore he tārua, nō reira ko tā mātou putanga he rua ngā rōpū motuhake, motuhake hoki, e whiwhi raraunga ana mā te remoteWrite.
Me whakamārama atu ahau i konei, i te wā i whakawhiti ai mātou mai i te VictoriaMetrics Single Node ki te VictoriaMetrics Cluster Version, i noho tonu mātou i roto i te awhe whakamahinga rauemi kotahi, ko te tikanga ko te mahara te rauemi matua. Koinei te āhua o te tohatoha o tā mātou tohatoha raraunga, arā, ko te whakamahinga rauemi.
Kua tāpirihia kētia he tārua ki konei. Kua whakakotahihia e mātou ngā mea katoa ki roto i tētahi rōpū nui. Ka wehea, ka tāruatia hoki ā mātou raraunga katoa.
E N ngā pūwāhi whakauru o te rōpū katoa, ko te tikanga ka taea e Prometheus te tāpiri raraunga mā HAPROXY. Koinei tā mātou pūwāhi whakauru. Ā, ka taea e Grafana te uru atu mā tēnei pūwāhi whakauru.
I roto i tā mātou take, ko HAPROXY te tauranga kotahi e tīpakohia ai, e whakaurua ai, me ētahi atu ratonga e ngā takawaenga ki roto i tēnei rōpū. I roto i tā mātou take, kāore i taea te waihanga i tētahi wāhitau kotahi; me waihanga e mātou he maha ngā pūwāhi whakauru nā te mea kei ngā rohe rerekē o te kaiwhakarato kapua kotahi ngā mīhini mariko e whakahaere ana i te rōpū VictoriaMetrics, arā, kāore rātou i roto i tā mātou kapua, engari kei waho atu.
Kei a mātou te whakatūpato. Ka whakamahia e mātou. Ka whakamahia e mātou te alertmanager mai i Prometheus. Ka whakamahia e mātou a Opsgenie me Telegram hei hongere tuku mō ngā whakatūpato. Ka whiwhi a Telegram i ngā whakatūpato mai i te kaiwhakawhanake, ētahi pea mai i te prod, engari ko te nuinga o ngā mea tatauranga e hiahiatia ana e ngā miihini. He mea nui ake a Opsgenie. Ka whakahaerehia ngā waeatanga me te whakahaere i ngā aituā.
Ko te pātai tawhito: "Ko wai e aroturuki ana i te aroturuki?" I roto i tā mātou take, ko te aroturuki tonu e aroturuki ana, nā te mea e whakamahi ana mātou i te vMagent i runga i ia pūnga. Ā, nā te mea kua tohatohahia ā mātou pūnga puta noa i ngā pokapū raraunga rerekē o te kaiwhakarato kotahi, he hongere motuhake tō ia pokapū raraunga, he motuhake, ā, ahakoa ka puta he roro wehewehe, ka whiwhi tonu mātou i ngā whakatūpato. Ae, ka nui ake, engari he pai ake te nui ake o ngā whakatūpato i te kore rawa.
Ka mutu tā mātou rārangi me te whakatinanatanga o te HA.
E hiahia ana hoki ahau ki te whakanui i taku wheako ki te hapori o VictoriaMetrics. He tino pai. He tino aro nui rātou, ā, e ngana ana ki te mārama ki ia take e tukuna ana.
I whakatuwherahia e au ngā take i runga i a GitHub. I tere tonu te whakatau. He iti tonu ngā take kāore anō kia tino whakatauhia, engari kua kite kē ahau i roto i te waehere kei te haere tonu te mahi.
Ko te tino raruraru mōku i ngā wā whakahounga ko te mea mēnā ka kati au i tētahi pūnga, kāore a vminsert e mōhio mō ngā hēkona 30 tuatahi kua kore te pūnaha o muri e mahi. Kua whakatauhia tēnei inaianei. I roto i te hēkona, e rua rānei, ka tangohia ngā raraunga mai i ngā pūnga katoa e toe ana, ā, ka mutu te tono i te tatari mō te pūnga ngaro.
I tētahi wā, i hiahia mātou kia whakatinanahia e VictoriaMetrics te kaiwhakahaere VictoriaMetrics. I tutuki i a mātou. Kei te hanga kaha mātou i tētahi takai huri noa i te kaiwhakahaere VictoriaMetrics hei whakauru i ngā ture tatau-mua katoa, me ērā atu mea, ki roto i a Prometheus, nā te mea kei te kaha mātou ki te whakamahi i ngā ture e haere tahi mai ana me te kaiwhakahaere Prometheus.
He whakaaro anō hei whakapai ake i te whakatinanatanga o te rōpū. Kua whakarārangihia e au i runga ake nei.
E hiahia ana hoki ahau ki te whakaheke tauira. I roto i tā mātou take, ko te whakaheke tauira anake te mea e hiahiatia ana hei tirotiro i ngā au. I te nuinga o te wā, kotahi te inenga e ranea ana mōku mō te rā. E hiahiatia ana ēnei au mō te tau, toru, rima, tekau tau. Ā, kotahi te uara inenga e ranea ana.
- Kua pāngia mātou e te mamae, pērā i ētahi o ō mātou hoa mahi, i te wā e whakamahi ana i a Prometheus.
- I whiriwhiria e mātou a VictoriaMetrics mā mātou anō.
- He tino pai te tauine i ngā taha poutū me ngā taha whakapae.
- Ka taea e tātou te tohatoha i ngā wāhanga rerekē puta noa i te maha o ngā pona i roto i te tautau, te whakawhāiti i te kaha o tō rātou mahara, te whakanui ake i te tohatoha mahara, me ētahi atu.
Ka whakamahia e mātou a VictoriaMetrics nā te mea he tino pai ki a mātou. Anei tōna āhua o mua, me tōna āhua o nāianei.
E rua ngā waehere QR mō te kōrerorero a VictoriaMetrics, aku hoapā, me te radar hangarau a LeroyMerlin.
Source: will.com
