ʻO VictoriaMetrics kahi DBMS wikiwiki a hiki ke hoʻopaʻa ʻia no ka mālama ʻana a me ka hoʻoili ʻana i ka ʻikepili ma ke ʻano o ka manawa manawa (he moʻolelo ka manawa a me kahi hoʻonohonoho o nā waiwai e pili ana i kēia manawa, no ka laʻana, loaʻa ma o ke koho balota ʻana i ke kūlana o nā sensor a i ʻole. hōʻiliʻili o nā ana).
ʻO Kolobaev Pavel koʻu inoa. DevOps, SRE, LeroyMerlin, ua like nā mea āpau me ke code - pili iā mākou a pau: e pili ana iaʻu a me nā limahana ʻē aʻe o LeroyMerlin.
Aia kahi ao ma luna o OpenStack. Aia kahi loulou liʻiliʻi i ka radar ʻenehana.
Kūkulu ʻia ia ma ka lako Kubernetes, a me nā lawelawe pili āpau no OpenStack a me ka logging.
ʻO kēia ka papahana a mākou i kūkulu ai. Ke hoʻomohala nei mākou i kēia mau mea a pau, loaʻa iā mākou kahi mea hoʻohana Prometheus nāna i mālama i ka ʻikepili i loko o ka hui K8s ponoʻī. ʻIke ʻo ia i nā mea e pono ai ke kāhili ʻia a waiho ʻia ma lalo o kona mau wāwae.
Pono mākou e hoʻoneʻe i nā ʻikepili āpau ma waho o ka pūʻulu Kubernetes, no ka mea, inā loaʻa kekahi mea, pono mākou e hoʻomaopopo i ka mea a me hea.
ʻO ka hopena mua, ʻo ia ka hoʻohana ʻana i ka federation ke loaʻa iā mākou kahi Prometheus ʻaoʻao ʻekolu, ke hele mākou i ka hui Kubernetes ma o ka mīkini hui.
Akā aia kekahi mau pilikia liʻiliʻi ma ʻaneʻi. I ko mākou hihia, hoʻomaka nā pilikia i ka wā i loaʻa iā mākou 250 metrics, a i ka wā i loaʻa ai nā metric 000, ʻike mākou ʻaʻole hiki iā mākou ke hana e like me ia. Ua hoʻonui mākou i ka scrape_timeout i 400 kekona.
No ke aha mākou i hana ai i kēia? Hoʻomaka ʻo Prometheus e helu i ka manawa pau mai ka hoʻomaka ʻana o ka pā. ʻAʻole ia he mea nui ke kahe mau nei ka ʻikepili. Inā i loko o kēia manawa i ʻōlelo ʻia ʻaʻole i hui ʻia ka ʻikepili a ʻaʻole i pani ʻia ke kau ma o http, a laila manaʻo ʻia ua hāʻule ka hālāwai a ʻaʻole i komo ka ʻikepili i Prometheus ponoʻī.
Ua kamaʻāina nā kānaka a pau i nā pakuhi i loaʻa iā mākou ke nalowale kekahi o nā ʻikepili. Ua haehae ʻia nā papa hana a ʻaʻole mākou hauʻoli i kēia.
ʻO ka koho aʻe ʻo sharding e pili ana i ʻelua Prometheus like ʻole ma o ka mīkini hui like.
No ka laʻana, lawe wale iā lākou a hoʻokaʻawale iā lākou ma ka inoa. Hiki ke hoʻohana ʻia kēia, akā ua hoʻoholo mākou e neʻe.
Pono mākou e hana i kēia mau ʻāpana i kekahi ʻano. Hiki iā ʻoe ke lawe i ka promxy, e hele ana i ka ʻāpana shard a hoʻonui i ka ʻikepili. Hana ia me ʻelua shards ma ke ʻano he helu komo hoʻokahi. Hiki ke hoʻokō ʻia kēia ma o ka promxy, akā paʻakikī loa.
ʻO ka koho mua, makemake mākou e haʻalele i ka mīkini federation no ka mea lohi loa.
Ke'ōlelo maopopo nei nā mea hoʻomohala Prometheus, "E nā kāne, e hoʻohana i kahi TimescaleDB ʻē aʻe no ka mea ʻaʻole mākou e kākoʻo i ka mālama lōʻihi o nā metric." ʻAʻole kēia kā lākou hana.
Kākau mākou ma kahi ʻāpana pepa e pono ai mākou e wehe i waho, i ʻole e mālama i nā mea āpau ma kahi hoʻokahi.
ʻO ka lua o ka drawback ka hoʻohana ʻana i ka hoʻomanaʻo. ʻAe, maopopo iaʻu he nui ka poʻe e ʻōlelo i ka makahiki 2020 ʻelua mau gigabytes o ka hoʻomanaʻo e kūʻai i kahi peni, akā naʻe.
I kēia manawa, loaʻa iā mākou kahi dev a prod environment. Ma dev aia ma kahi o 9 gigabytes no 350 metrics. I ka prod he 000 gigabytes a ma kahi o 14 metrics. I ka manawa like, ʻo ko mākou manawa paʻa he 780 mau minuke. He ino keia. A i kēia manawa e wehewehe wau i ke kumu.
Hana mākou i ka helu ʻana, ʻo ia hoʻi, me hoʻokahi a me ka hapa miliona metric, a ua kokoke mākou iā lākou, ma ka pae hoʻolālā e loaʻa iā mākou he 35-37 gigabytes o ka hoʻomanaʻo. Akā ʻo 4 miliona mau metric e koi ana ma kahi o 90 gigabytes o ka hoʻomanaʻo. ʻO ia hoʻi, ua helu ʻia me ka hoʻohana ʻana i ke ʻano i hāʻawi ʻia e nā mea hoʻomohala Prometheus. Ua nānā mākou i ka pilina a ʻike mākou ʻaʻole makemake mākou e uku i ʻelua miliona no kahi kikowaena no ka nānā wale ʻana.
ʻAʻole mākou e hoʻonui wale i ka helu o nā mīkini, ke nānā pū nei mākou i nā mīkini virtual iā lākou iho. No laila, ʻoi aku ka nui o nā mīkini virtual, ʻoi aku ka nui o nā metric o nā ʻano like ʻole, a pēlā aku.
Me kahi diski, ʻaʻole maikaʻi loa nā mea āpau ma aneʻi, akā makemake wau e hoʻomaikaʻi. Ua loaʻa iā mākou he 15 gigabytes i nā lā 120, ʻo 100 ka ʻikepili i hoʻopaʻa ʻia, 20 nā ʻikepili i hoʻopaʻa ʻole ʻia, akā makemake mau mākou i ka liʻiliʻi.
No laila, kākau mākou i hoʻokahi kikoʻī - he ʻai nui kēia o nā kumuwaiwai, a mākou e makemake ai e mālama, no ka mea, ʻaʻole mākou makemake i kā mākou pūʻulu kiaʻi e hoʻopau i nā kumu waiwai ma mua o kā mākou hui, nāna e mālama iā OpenStack.
Aia kekahi drawback o Prometheus, a mākou i ʻike ai no mākou iho, ʻo ia ka liʻiliʻi o kekahi ʻano palena hoʻomanaʻo. Me Prometheus, ʻoi aku ka maikaʻi o nā mea āpau ma ʻaneʻi, no ka mea, ʻaʻohe ona ʻano wili. ʻO ka hoʻohana ʻana i kahi palena ma docker ʻaʻole ia he koho. Inā hāʻule koke kāu RAF a he 20-30 gigabytes, a laila lōʻihi loa ka manawa e ala ai.
ʻO kēia kekahi kumu ʻaʻole kūpono ʻo Prometheus iā mākou, ʻo ia hoʻi, ʻaʻole hiki iā mākou ke kaupalena i ka hoʻohana ʻana i ka hoʻomanaʻo.
He mea hiki ke hana i ia papahana. Pono mākou i kēia papahana no ka hoʻonohonoho ʻana i kahi pūʻulu HA. Makemake mākou e loaʻa i kā mākou metric i nā manawa a pau a me nā wahi āpau, ʻoiai inā hāʻule ka server nāna e mālama i kēia mau ana. A pēlā mākou e kūkulu ai i kahi papahana.
Ke ʻōlelo nei kēia kumumanaʻo e loaʻa iā mākou ke kope kope ʻana o nā shards, a no laila, ʻo ke kope ʻana i nā kumukūʻai o nā waiwai i pau. Hiki ke hoʻonui ʻia ma kahi ākea, akā naʻe, ʻoi aku ka maikaʻi o ka hoʻohana waiwai.
ʻO nā hemahema ma ke ʻano i kākau ai mākou iā lākou iho no mākou iho:
- Pono e hoʻouka i nā ana ma waho.
- Hoʻohana waiwai kiʻekiʻe.
- ʻAʻohe ala e kaupalena ʻia ai ka hoʻohana ʻana i ka hoʻomanaʻo.
- Paʻakikī a me ka hoʻokō waiwai nui o HA.
No mākou iho, ua hoʻoholo mākou e neʻe ana mākou mai Prometheus ma ke ʻano he hale mālama.
Ua ʻike mākou i nā koi hou aʻe no mākou iho e pono ai. ʻO kēia:
- ʻO kēia ke kākoʻo promql, no ka mea, ua kākau mua ʻia nā mea he nui no Prometheus: nā nīnau, nā makaʻala.
- A laila loaʻa iā mākou ʻo Grafana, i kākau ʻia ma ke ʻano like no Prometheus ma ke ʻano he hope. ʻAʻole au makemake e kākau hou i nā dashboards.
- Makemake mākou e kūkulu i kahi hoʻolālā HA maʻamau.
- Makemake mākou e hōʻemi i ka hoʻohana ʻana i nā kumuwaiwai.
- Aia kekahi nuance liʻiliʻi. ʻAʻole hiki iā mākou ke hoʻohana i nā ʻano ʻōnaehana hōʻiliʻili kapua. ʻAʻole mākou ʻike i ka mea e hāʻule i kēia mau metric. A no ka mea hiki i kekahi mea ke lele i laila, pono mākou e kaupalena iā mākou iho i kahi hoʻonohonoho kūloko.
He liʻiliʻi ke koho. Ua hōʻiliʻili mākou i nā mea a pau i loaʻa iā mākou. Ua nānā mākou i ka ʻaoʻao Prometheus ma ka ʻāpana hoʻohui, heluhelu i kahi hui o nā ʻatikala, a ʻike i nā mea i waho. A no mākou iho, ua koho mākou iā VictoriaMetrics i mea pani no Prometheus.
No ke aha mai? No ka mea:
- ʻIke promql.
- Aia kahi hoʻolālā modular.
- ʻAʻole pono e hoʻololi iā Grafana.
- A ʻo ka mea nui loa, e hāʻawi paha mākou i ka waihona metric i loko o kā mākou ʻoihana ma ke ʻano he lawelawe, no laila ke nānā mua nei mākou i nā palena o nā ʻano like ʻole i hiki i nā mea hoʻohana ke hoʻohana i nā kumuwaiwai āpau o ka puʻupuʻu ma kekahi ala palena, no ka mea aia kahi manawa. ia e multitenancy.
E hoʻohālikelike mua kāua. Lawe mākou i ka Prometheus like i loko o ka hui, hele ʻo Prometheus waho iā ia. Hoʻohui ma o remoteWrite VictoriaMetrics.
E hoʻopaʻa koke wau ma aneʻi ua loaʻa iā mākou kahi piʻi iki o ka hoʻohana CPU mai VictoriaMetrics. Hōʻike ka wiki VictoriaMetrics iā ʻoe i nā ʻāpana ʻoi aku ka maikaʻi. Ua nānā mākou iā lākou. Ua hōʻemi maikaʻi lākou i ka hoʻohana CPU.
I kā mākou hihia, ʻaʻole i piʻi nui ka hoʻomanaʻo ʻana o Prometheus, aia ma ka hui Kubernetes.
Hoʻohālikelike mākou i ʻelua kumu ʻikepili o ka ʻikepili like. Ma Prometheus ʻike mākou i nā ʻikepili like ʻole. Maikaʻi nā mea a pau ma VictoriaMetrics.
Nā hualoaʻa hoʻāʻo wahi disk. Ua loaʻa iā mākou ma Prometheus he 120 gigabytes i ka huina. Ma VictoriaMetrics ua loaʻa iā mākou he 4 gigabytes i kēlā me kēia lā. Aia kekahi ʻano hana ʻokoʻa ma mua o ka mea a mākou e ʻike ai ma Prometheus. ʻO ia hoʻi, ua hoʻopili maikaʻi ʻia ka ʻikepili i ka lā, i ka hapalua hola. Ua hōʻiliʻili maikaʻi ʻia lākou i ka lā, i ka hapalua hola, ʻoiai ʻo ka mea e nalowale ana ka ʻikepili ma hope. ʻO ka hopena, ua mālama mākou ma kahi diski.
Mālama pū mākou i ka hoʻohana ʻana i nā kumuwaiwai hoʻomanaʻo. I ka manawa o ka hoʻāʻo ʻana, ua kau ʻia mākou Prometheus ma kahi mīkini virtual - 8 cores, 24 gigabytes. ʻAi ʻo Prometheus i nā mea āpau. Ua hāʻule ʻo ia ma luna o OOM Killer. I ka manawa like, ʻo 900 wale nō nā metric ikaika i ninini ʻia i loko. ʻO kēia ma kahi o 000-25 metrics i kēlā me kēia kekona.
Ua holo mākou iā VictoriaMetrics ma kahi mīkini uila lua-core me 8 gigabytes o RAM. Ua hoʻokō mākou i ka hana maikaʻi ʻana o VictoriaMetrics ma ka hoʻopili ʻana me kekahi mau mea ma kahi mīkini 8GB. I ka hopena, ua mālama mākou iā 7 gigabytes. I ka manawa like, ʻoi aku ka kiʻekiʻe o ka wikiwiki o ka hāʻawi ʻana i nā ʻike, ʻo ia hoʻi nā metric ma mua o ka Prometheus.
Ua ʻoi aku ka maikaʻi o ka CPU ma mua o Prometheus. Ma ʻaneʻi, hoʻopau ʻo Prometheus i 2,5 cores, a ʻo VictoriaMetrics e hoʻopau wale i 0,25 cores. I ka hoʻomaka - 0,5 cores. I ka hui ʻana, hiki i hoʻokahi kumu, akā he mea nui loa kēia.
I kā mākou hihia, hāʻule ka koho ma VictoriaMetrics no nā kumu maopopo; makemake mākou e mālama kālā a ua hana mākou.
E hoʻokaʻawale i ʻelua mau kiko i kēia manawa - ka hoʻouka ʻana i nā metric a me ka hoʻohana nui ʻana i nā kumuwaiwai. A pono mākou e hoʻoholo i ʻelua mau helu i waiho ʻia no mākou iho.
Ma ʻaneʻi e hoʻopaʻa koke wau, e noʻonoʻo mākou iā VictoriaMetrics ma ke ʻano he mālama ʻana i nā metric. Akā, no ka mea e hāʻawi mākou iā VictoriaMetrics i mea e mālama ai no Leroy āpau, pono mākou e kaupalena i ka poʻe e hoʻohana i kēia pūʻulu i ʻole lākou e hāʻawi mai iā mākou.
Aia kahi ʻāpana nani e hiki ai iā ʻoe ke kaupalena i ka manawa, ma ka nui o ka ʻikepili a me ka manawa hoʻokō.
Aia kekahi koho maikaʻi loa e hiki ai iā mākou ke kaupalena i ka hoʻohana ʻana i ka hoʻomanaʻo, a laila hiki iā mākou ke ʻike i ke koena e hiki ai iā mākou ke loaʻa ka wikiwiki hana maʻamau a me ka hoʻohana pono ʻana i nā kumuwaiwai.
Minus hoʻokahi kikoʻī hou aku, ʻo ia hoʻi, e ʻoki i ke kiko - ʻaʻole hiki iā ʻoe ke kaupalena i ka hoʻohana ʻana i ka hoʻomanaʻo.
I nā hoʻomaka mua, ua hoʻāʻo mākou iā VictoriaMetrics Single Node. A laila neʻe mākou i ka VictoriaMetrics Cluster Version.
Eia kā mākou lima manuahi e hoʻokaʻawale i nā lawelawe like ʻole ma VictoriaMetrics ma muli o ka mea e holo ai lākou a me nā kumuwaiwai a lākou e ʻai ai. He hopena maʻalahi a maʻalahi kēia. Ua hoʻohana mākou iā mākou iho.
ʻO nā mea nui o VictoriaMetrics Cluster Version he vmstsorage. Hiki ke loaʻa ka helu N o lākou. I kā mākou hihia aia 2 o lākou i kēia manawa.
A aia kekahi vminsert. He kikowaena proxy kēia e hiki ai iā mākou ke: hoʻonohonoho i ka sharding ma waena o nā waihona āpau a mākou i haʻi ai iā ia, a hiki nō hoʻi i kahi kope, ʻo ia hoʻi, loaʻa iā ʻoe ka sharding a me kahi kope.
Kākoʻo ʻo Vminsert i nā protocol OpenTSDB, Graphite, InfluxDB a me remoteWrite mai Prometheus.
Aia kekahi vmselect. ʻO kāna hana nui e hele i ka vmstorage, loaʻa ka ʻikepili mai lākou, hoʻokaʻawale i kēia ʻikepili a hāʻawi i ka mea kūʻai aku.
Aia kekahi mea kupanaha i kapa ʻia ʻo vmagent. Makemake nui mākou iā ia. Hiki iā ʻoe ke hoʻonohonoho like me Prometheus a hana mau i nā mea āpau e like me Prometheus. ʻO ia hoʻi, hōʻiliʻili ia i nā metric mai nā hui like ʻole a me nā lawelawe a hoʻouna iā lākou i vminsert. A laila pili nā mea a pau iā ʻoe.
ʻO kahi lawelawe maikaʻi ʻē aʻe ʻo vmalert, kahi e hiki ai iā ʻoe ke hoʻohana iā VictoriaMetrics ma ke ʻano he backend, e loaʻa i ka ʻikepili i hoʻoponopono ʻia mai vminsert a hoʻouna iā ia i vmselect. Hoʻoponopono ia i nā makaʻala iā lākou iho, a me nā lula. I ka hihia o nā makaʻala, loaʻa iā mākou ka makaʻala ma o ka alertmanager.
Aia kekahi ʻāpana wmauth. Hiki iā mākou a ʻaʻole paha (ʻaʻole mākou i hoʻoholo i kēia) e hoʻohana iā ia ma ke ʻano he ʻōnaehana mana no ka mana multitenancy o nā pūʻulu. Kākoʻo ia i kahi mamaoWrite no Prometheus a hiki ke ʻae ʻia ma muli o ka url, a i ʻole ka ʻāpana ʻelua o ia mea, kahi āu e hiki ai a hiki ʻole ke kākau.
Aia kekahi vmbackup, vmrestore. ʻO kēia, ma ke ʻano, ka hoʻihoʻi a me ka hoʻihoʻi ʻana i nā ʻikepili āpau. Hiki iā S3, GCS, faila.
Ua hana ʻia ka hana mua o kā mākou hui i ka wā quarantine. I kēlā manawa, ʻaʻohe kope, no laila, ʻo kā mākou ʻike ʻana he ʻelua mau pūʻulu ʻokoʻa a kūʻokoʻa i loaʻa iā mākou ka ʻikepili ma o remoteWrite.
Ma ʻaneʻi, e hoʻopaʻa wau i ka wā i hoʻololi ai mākou mai VictoriaMetrics Single Node i VictoriaMetrics Cluster Version, ua noho mau mākou me nā kumuwaiwai like ʻole, ʻo ia ka mea nui ka hoʻomanaʻo. ʻO kēia ke ʻano o kā mākou ʻikepili, ʻo ia hoʻi ka hoʻohana waiwai.
Ua hoʻohui ʻia kahi kope ma ʻaneʻi. Hoʻohui mākou i kēia mau mea a pau i hoʻokahi hui nui. Hoʻopili ʻia kā mākou ʻikepili a pau.
Loaʻa i ka hui holoʻokoʻa nā helu komo N, ʻo ia hoʻi hiki iā Prometheus ke hoʻohui i ka ʻikepili ma o HAPROXY. Eia mākou i kēia wahi komo. A ma kēia wahi komo hiki iā ʻoe ke komo mai Grafana.
I kā mākou hihia, ʻo HAPROXY wale nō ke awa e koho ai, hoʻokomo a me nā lawelawe ʻē aʻe i loko o kēia pūʻulu. I kā mākou hihia, ʻaʻole hiki ke hana i hoʻokahi ʻōlelo; pono mākou e hana i nā helu komo, no ka mea, aia nā mīkini virtual kahi e holo ai ka hui ʻo VictoriaMetrics i nā ʻāpana like ʻole o ka mea hāʻawi kapuaʻi like, ʻo ia hoʻi ʻaʻole i loko o kā mākou ao, akā ma waho. .
He makaʻala mākou. Hoʻohana mākou. Hoʻohana mākou i ka alertmanager mai Prometheus. Hoʻohana mākou iā Opsgenie a me Telegram ma ke ʻano he ala hoʻolaha makaʻala. Ma Telegram e ninini lākou mai dev, malia paha kekahi mea mai ka prod, akā ʻo ka hapa nui o kahi mea helu helu, pono e nā ʻenekinia. A he koʻikoʻi ʻo Opsgenie. He mau kelepona kēia, hoʻoponopono hanana.
ʻO ka nīnau mau loa: "ʻO wai ke nānā i ka nānā ʻana?" I kā mākou hihia, nānā ka nānā ʻana i ka nānā ʻana iā ia iho, no ka mea, hoʻohana mākou i ka vmagent ma kēlā me kēia node. A ma muli o ka māhele ʻia ʻana o kā mākou mau node ma nā kikowaena ʻikepili like ʻole o ka mea hoʻolako hoʻokahi, aia kēlā me kēia kikowaena data i kona ala ponoʻī, kūʻokoʻa lākou, a inā paha e hiki mai ka lolo maʻemaʻe, e loaʻa mau iā mākou nā mākaʻikaʻi. ʻAe, ʻoi aku ka nui o lākou, akā ʻoi aku ka maikaʻi o ka loaʻa ʻana o nā makaʻala ma mua o ka ʻole.
Hoʻopau mākou i kā mākou papa inoa me kahi hoʻokō HA.
A ʻoi aku kaʻu makemake e nānā i ka ʻike o ke kamaʻilio ʻana me ke kaiāulu VictoriaMetrics. Ua lilo i mea maikaʻi loa. Ua pane nā kāne. Hoʻāʻo lākou e ʻimi i kēlā me kēia hihia i hāʻawi ʻia.
Ua hoʻomaka wau i nā pilikia ma GitHub. Ua hoʻoholo koke ʻia lākou. Aia kekahi mau pilikia ʻē aʻe ʻaʻole i pani paʻa ʻia, akā hiki iaʻu ke ʻike mai ke code e hana nei ma kēia kuhikuhi.
ʻO ka ʻeha nui iaʻu i ka wā o ka hoʻomaʻamaʻa ʻana, inā wau e pani i kahi node, a laila no ka 30 kekona mua ʻaʻole hiki iā vminsert ke hoʻomaopopo ʻaʻole he backend. Ua hoʻoholo ʻia kēia i kēia manawa. A ʻoiaʻiʻo i loko o kahi kekona a ʻelua paha, lawe ʻia ka ʻikepili mai nā node i koe a pau, a pau ka noi i ke kali ʻana i kēlā node i nalowale.
I kekahi manawa makemake mākou iā VictoriaMetrics e lilo i mea hoʻohana VictoriaMetrics. Ua kali mākou iā ia. Ke kūkulu ikaika nei mākou i kēia manawa no ka mea hoʻohana VictoriaMetrics e lawe i nā lula helu mua, etc. Prometheus, no ka mea ke hoʻohana ikaika nei mākou i nā lula e hele mai me ka mea hoʻohana Prometheus.
Aia nā manaʻo e hoʻomaikaʻi i ka hoʻokō pūʻulu. Ua wehewehe au iā lākou ma luna.
A makemake nui au e hoʻohaʻahaʻa i ka laʻana. I kā mākou hihia, pono ka downsampling wale nō no ka nānā ʻana i nā ʻano. ʻO ka ʻōlelo maʻamau, ua lawa hoʻokahi metric iaʻu i ka lā. Pono kēia mau ʻano no hoʻokahi makahiki, ʻekolu, ʻelima, ʻumi makahiki. A lawa loa ka waiwai metric hoʻokahi.
- Ua ʻike mākou i ka ʻeha, e like me kekahi o kā mākou mau hoa hana, i ka wā e hoʻohana ai iā Prometheus.
- Ua koho mākou iā VictoriaMetrics no mākou iho.
- Ua maikaʻi ka unahi ʻana ma ka ʻaoʻao a me ka pae.
- Hiki iā mākou ke hoʻokaʻawale i nā ʻāpana like ʻole i nā helu like ʻole o nā nodes i ka pūʻulu, kaupalena iā lākou ma ka hoʻomanaʻo, hoʻohui i ka hoʻomanaʻo, etc.
E hoʻohana mākou iā VictoriaMetrics ma ka home no ka mea makemake nui mākou. ʻO kēia ka mea i hala a me ka mea i lilo.
ʻElua mau code QR no ke kamaʻilio VictoriaMetrics, koʻu mau hoapili, LeroyMerlin technical radar.
Source: www.habr.com