VictoriaMetrics DBMS ce mai sauri kuma mai ƙima don adanawa da sarrafa bayanai ta hanyar tsarin lokaci (rikodi ya ƙunshi lokaci da saitin ƙimar daidai wannan lokacin, alal misali, ana samun ta ta hanyar jefa ƙuri'a na lokaci-lokaci na matsayin firikwensin ko tarin ma'auni).
Sunana Kolobaev Pavel. DevOps, SRE, LeroyMerlin, komai yana kama da lamba - duk game da mu ne: game da ni da kuma game da sauran ma'aikatan LeroyMerlin.
Akwai girgije bisa OpenStack. Akwai ƙaramin hanyar haɗi zuwa radar fasaha.
An gina shi akan kayan aikin Kubernetes, da kuma akan duk wasu ayyuka masu alaƙa don OpenStack da shiga.
Wannan shi ne makircin da muke da shi a cikin ci gaba. Lokacin da muke haɓaka duk waɗannan, muna da ma'aikacin Prometheus wanda ke adana bayanai a cikin gungu na K8s kanta. Kai tsaye ya sami abin da ake buƙatar gogewa ya sanya shi ƙarƙashin ƙafafunsa, yana magana.
Za mu buƙaci matsar da duk bayanan a waje da gungu na Kubernetes, saboda idan wani abu ya faru, muna buƙatar fahimtar menene kuma a ina.
Magani na farko shine muna amfani da tarayya lokacin da muke da Prometheus na ɓangare na uku, lokacin da muka je gungu na Kubernetes ta hanyar tsarin tarayya.
Amma akwai wasu ƙananan matsaloli a nan. A cikin yanayinmu, matsalolin sun fara ne lokacin da muke da ma'auni 250, kuma lokacin da akwai ma'auni 000, mun gane cewa ba za mu iya yin haka ba. Mun ƙara scrape_timeout zuwa daƙiƙa 400.
Me ya sa muka yi haka? Prometheus yana fara kirga lokacin ƙarewa daga farkon shinge. Ba komai har yanzu bayanan suna gudana. Idan a cikin wannan ƙayyadadden lokaci ba a haɗa bayanan ba kuma ba a rufe zaman ta hanyar http, to ana ganin zaman ya gaza kuma bayanan ba su shiga cikin Prometheus kanta ba.
Kowa ya san jadawali da muke samu lokacin da wasu bayanan suka ɓace. Jadawalin sun tsage kuma ba mu gamsu da wannan ba.
Zaɓin na gaba shine sharding dangane da Prometheus daban-daban guda biyu ta hanyar tsarin tarayya iri ɗaya.
Alal misali, ɗauka kawai su kuma raba su da suna. Ana iya amfani da wannan kuma, amma mun yanke shawarar ci gaba.
Yanzu za mu sarrafa waɗannan shards ko ta yaya. Kuna iya ɗaukar promxy, wanda ke zuwa yankin shard kuma ya ninka bayanai. Yana aiki tare da shards biyu azaman wurin shigarwa guda ɗaya. Ana iya aiwatar da wannan ta hanyar wakili, amma har yanzu yana da wahala.
Zabi na farko shine muna son yin watsi da tsarin tarayya saboda yana da hankali sosai.
Masu haɓakawa na Prometheus suna faɗi a sarari, "Guys, yi amfani da TimecaleDB daban saboda ba za mu goyi bayan adana awo na dogon lokaci ba." Wannan ba aikinsu bane.
Mun rubuta a takarda wanda har yanzu muna buƙatar saukewa a waje, don kada mu adana kome a wuri guda.
Matsala ta biyu ita ce amfani da ƙwaƙwalwar ajiya. Ee, na fahimci cewa mutane da yawa za su ce a cikin 2020 ma'aurata gigabytes na ƙwaƙwalwar ajiya suna kashe dinari, amma har yanzu.
Yanzu muna da yanayin dev da prod. A cikin dev yana da kusan gigabytes 9 don awoyi 350. A cikin samfurin yana da gigabytes 000 kuma kadan sama da awo 14. A lokaci guda, lokacin riƙe mu shine kawai mintuna 780. Wannan ba daidai ba ne. Kuma yanzu zan bayyana dalilin.
Muna yin lissafi, wato, tare da ma'auni miliyan daya da rabi, kuma mun riga mun kusanci su, a matakin zane muna samun 35-37 gigabytes na ƙwaƙwalwar ajiya. Amma tuni ma'auni miliyan 4 suna buƙatar kimanin gigabytes 90 na ƙwaƙwalwar ajiya. Wato, an ƙididdige shi ta amfani da dabarar da masu haɓaka Prometheus suka bayar. Mun kalli alaƙar kuma mun gane cewa ba ma son biyan miliyan biyu don sabar kawai don saka idanu.
Ba wai kawai za mu ƙara yawan injuna ba, muna kuma sanya ido kan injinan kama-da-wane da kansu. Sabili da haka, mafi kyawun injina, ƙarin itorican etitrics daban-daban, da sauransu zamu sami ci gaban musamman na tari game da awo.
Tare da sararin faifai, komai ba bakin ciki ba ne a nan, amma ina so in inganta shi. Mun sami jimillar gigabytes 15 a cikin kwanaki 120, wanda 100 aka matsa bayanai, 20 ne marasa matsawa, amma koyaushe muna son ƙasa.
Saboda haka, mun sake rubuta ƙarin batu guda ɗaya - wannan babban amfani ne na albarkatu, wanda har yanzu muna son adanawa, saboda ba ma son ƙungiyar sa ido ta cinye albarkatu fiye da rukunin mu, wanda ke sarrafa OpenStack.
Akwai wani ƙarin koma baya na Prometheus, wanda muka gano da kanmu, wannan shine aƙalla wasu nau'ikan ƙayyadaddun ƙwaƙwalwar ajiya. Tare da Prometheus, duk abin da ya fi muni a nan, saboda ba shi da irin wannan karkatarwa kwata-kwata. Amfani da iyaka a docker shima ba zaɓi bane. Idan ba zato ba tsammani RAF ɗinku ya faɗi kuma akwai gigabytes 20-30, to zai ɗauki lokaci mai tsawo don tashi.
Wannan wani dalili ne da ya sa Prometheus bai dace da mu ba, watau ba za mu iya iyakance amfani da ƙwaƙwalwar ajiya ba.
Zai yiwu a fito da irin wannan makirci. Muna buƙatar wannan tsarin don tsara tarin HA. Muna son ma'aunin mu ya kasance koyaushe kuma a ko'ina, koda sabar da ke adana waɗannan ma'aunin ta yi karo. Kuma ta haka ne za mu gina irin wannan makirci.
Wannan makirci ya ce za mu sami kwafi na shards, kuma, bisa ga haka, kwafin kuɗin albarkatun da aka cinye. Ana iya daidaita shi kusan a kwance, amma duk da haka amfani da albarkatun zai zama jahannama.
Rashin hasara a cikin tsari, a cikin hanyar da muka rubuta su don kanmu:
- Yana buƙatar loda awo a waje.
- Yawan amfani da albarkatu.
- Babu wata hanya ta iyakance amfani da ƙwaƙwalwar ajiya.
- Haɗaɗɗen aiwatar da kayan aiki mai ƙarfi na HA.
Don kanmu, mun yanke shawarar cewa muna ƙaura daga Prometheus azaman wurin ajiya.
Mun gano ƙarin buƙatun don kanmu waɗanda muke buƙata. Wannan:
- Wannan tallafin promql ne, saboda an riga an rubuta abubuwa da yawa don Prometheus: tambayoyi, faɗakarwa.
- Kuma a sa'an nan muna da Grafana, wanda aka riga aka rubuta a daidai wannan hanya ga Prometheus a matsayin baya. Ba na son sake rubuta dashboards.
- Muna son gina gine-ginen HA na yau da kullun.
- Muna so mu rage amfani da duk wani albarkatu.
- Akwai wani ƙaramin nuance. Ba za mu iya amfani da nau'ikan tsarin tattara awo na girgije iri-iri ba. Ba mu san abin da zai fada cikin waɗannan ma'aunin ba tukuna. Kuma tun da komai na iya tashi a can, dole ne mu iyakance kanmu ga wurin zama na gida.
Akwai ɗan zaɓi. Mun tattara duk abin da muka samu kwarewa da. Mun kalli shafin Prometheus a cikin sashin haɗin kai, mun karanta tarin labarai, kuma muka ga abin da ke can. Kuma ga kanmu, mun zaɓi VictoriaMetrics a matsayin maye gurbin Prometheus.
Me yasa? Domin:
- Ya san promql.
- Akwai tsarin gine-gine na zamani.
- Baya buƙatar canje-canje zuwa Grafana.
- Kuma mafi mahimmanci, ƙila za mu samar da ma'aunin ma'auni a cikin kamfaninmu a matsayin sabis, don haka muna sa ido gaba ga ƙuntatawa iri-iri don masu amfani su iya amfani da duk albarkatun gungu ta wata hanya mai iyaka, saboda akwai damar. cewa zai multitenancy.
Bari mu yi kwatancen farko. Muna ɗaukar Prometheus iri ɗaya a cikin gungu, Prometheus na waje yana zuwa gare ta. Ƙara ta hanyar nesaWrite VictoriaMetrics.
Nan da nan zan yi ajiyar wuri cewa a nan mun sami ɗan ƙara yawan amfani da CPU daga VictoriaMetrics. The VictoriaMetrics wiki yana gaya muku waɗanne sigogi ne mafi kyau. Mun duba su. Sun rage yawan amfani da CPU sosai.
A cikin yanayinmu, ƙwaƙwalwar ƙwaƙwalwar ajiya na Prometheus, wanda ke cikin gungu na Kubernetes, bai karu sosai ba.
Muna kwatanta tushen bayanai guda biyu na bayanai iri ɗaya. A cikin Prometheus muna ganin bayanan da suka ɓace. Komai yana da kyau a VictoriaMetrics.
Sakamakon gwajin sararin diski. Mu a Prometheus mun sami gigabytes 120 gabaɗaya. A VictoriaMetrics mun riga mun karɓi gigabytes 4 kowace rana. Akwai wata hanya ta ɗan bambanta fiye da abin da muke amfani da mu don gani a cikin Prometheus. Wato, an riga an matse bayanan da kyau a cikin yini ɗaya, cikin rabin sa'a. An riga an girbe su da kyau a cikin yini guda, a cikin rabin sa'a, duk da cewa bayanan za su ɓace daga baya. A sakamakon haka, mun adana akan sarari diski.
Muna kuma yin ajiya akan amfani da albarkatun ƙwaƙwalwar ajiya. A lokacin gwaji, an tura Prometheus akan na'ura mai mahimmanci - 8 cores, 24 gigabytes. Prometheus yana cin kusan komai. Ya fadi akan OOM Killer. A lokaci guda, ma'auni masu aiki 900 ne kawai aka zuba a ciki. Wannan shine kusan ma'auni 000-25 a sakan daya.
Mun gudu VictoriaMetrics akan na'ura mai kama da dual-core tare da 8 gigabytes na RAM. Mun yi nasarar samun VictoriaMetrics don yin aiki da kyau ta hanyar yin gyare-gyare da wasu abubuwa akan injin 8GB. A ƙarshe, mun ajiye shi zuwa 7 gigabytes. A lokaci guda, saurin isar da abun ciki, watau ma'auni, ya ma fi na Prometheus girma.
CPU ya zama mafi kyau idan aka kwatanta da Prometheus. Anan Prometheus yana cinye nau'ikan nau'ikan 2,5, kuma VictoriaMetrics yana cinye 0,25 kawai. A lokacin rani - 0,5 core. Yayin da yake haɗewa, ya kai cibiya ɗaya, amma wannan yana da matuƙar wuya.
A cikin yanayinmu, zaɓin ya faɗi akan VictoriaMetrics don dalilai masu ma'ana; muna son adana kuɗi kuma mun yi.
Bari mu ketare maki biyu nan da nan - loda ma'auni da yawan amfani da albarkatu. Kuma kawai mu yanke shawarar abubuwa biyu da har yanzu mun bar wa kanmu.
Anan zan yi ajiyar wuri nan da nan, muna ɗaukar VictoriaMetrics azaman ajiyar awo. Amma da yake za mu iya samar da VictoriaMetrics a matsayin ajiya ga duk Leroy, muna buƙatar iyakance waɗanda za su yi amfani da wannan gungu don kada su ba mu.
Akwai siga mai ban mamaki wanda ke ba ku damar iyakance ta lokaci, ta ƙarar bayanai da lokacin aiwatarwa.
Hakanan akwai kyakkyawan zaɓi wanda ke ba mu damar iyakance amfani da ƙwaƙwalwar ajiya, ta haka za mu iya samun ma'auni sosai wanda zai ba mu damar samun saurin aiki na yau da kullun da isasshen amfani da albarkatu.
Rage ƙarin maki ɗaya, watau ketare wurin - ba za ku iya iyakance yawan ƙwaƙwalwar ajiya ba.
A karo na farko, mun gwada VictoriaMetrics Single Node. Na gaba za mu matsa zuwa VictoriaMetrics Cluster Version.
Anan muna da hannun kyauta don raba ayyuka daban-daban a cikin VictoriaMetrics dangane da abin da za su yi amfani da su da kuma irin albarkatun da za su cinye. Wannan bayani ne mai sauƙi kuma mai dacewa. Mun yi amfani da wannan a kan kanmu.
Babban abubuwan da aka haɗa na VictoriaMetrics Cluster Version sune vmstsorage. Za a iya samun N lamba daga cikinsu. A wajenmu akwai 2 daga cikinsu ya zuwa yanzu.
Kuma akwai vminsert. Wannan ita ce uwar garken wakili wanda ke ba mu damar: shirya sharding tsakanin duk ma'ajiyar da muka ba da labari game da su, kuma tana ba da damar kwafi, watau za ku sami sharding da kwafi.
Vminsert yana goyan bayan OpenTSDB, Graphite, InfluxDB da ka'idojin Rubutun nesa daga Prometheus.
Akwai kuma vmselect. Babban aikinsa shine zuwa vmstorage, karɓar bayanai daga gare su, kwafin wannan bayanan kuma a ba abokin ciniki.
Akwai wani abu mai ban mamaki kamar vmagent. Muna sonta sosai. Yana ba ku damar saita daidai kamar Prometheus kuma har yanzu kuna yin komai daidai kamar Prometheus. Wato yana tattara ma'auni daga ƙungiyoyi da ayyuka daban-daban kuma yana aika su zuwa vminsert. Sannan komai ya dogara da kai.
Wani babban sabis ɗin shine vmalert, wanda ke ba ku damar amfani da VictoriaMetrics azaman abin baya, karɓar bayanan sarrafawa daga vminsert kuma aika zuwa vmselect. Yana aiwatar da faɗakarwar da kansu, da kuma ƙa'idodi. Game da faɗakarwa, muna samun faɗakarwa ta hanyar mai sarrafa faɗakarwa.
Akwai bangaren wmauth. Za mu iya ko a'a (ba mu yanke shawara kan wannan ba tukuna) mu yi amfani da shi azaman tsarin ba da izini ga nau'ikan tari masu yawa. Yana goyan bayan nisaWrite don Prometheus kuma yana iya ba da izini dangane da url, ko kuma bangarensa na biyu, inda zaka iya ko ba za ka iya rubutawa ba.
Hakanan akwai vmbackup, vmrestore. Wannan shi ne, a zahiri, maidowa da ajiyar duk bayanai. Mai iya S3, GCS, fayil.
An yi gyare-gyaren farko na tarin mu yayin keɓewa. A lokacin, babu kwafi, don haka bayanin mu ya ƙunshi gungu daban-daban guda biyu daban-daban kuma masu zaman kansu waɗanda muka karɓi bayanai ta hanyar rubutun nesa.
Anan zan yi ajiyar zuciya cewa lokacin da muka sauya daga VictoriaMetrics Single Node zuwa VictoriaMetrics Cluster Version, har yanzu mun kasance tare da albarkatun da aka cinye iri ɗaya, watau babban ɗayan shine ƙwaƙwalwar ajiya. Wannan shine kusan yadda aka rarraba bayananmu, watau amfani da albarkatun ƙasa.
An riga an ƙara kwafi anan. Mun haɗa duk waɗannan zuwa babban gungu guda ɗaya. Dukkan bayanan mu an share su kuma an kwafi su.
Gabaɗayan tarin yana da wuraren shigarwa N, watau Prometheus na iya ƙara bayanai ta hanyar HAPROXY. Ga mashigar mu. Kuma ta wannan hanyar shigar za ku iya shiga daga Grafana.
A cikin yanayinmu, HAPROXY ita ce kawai tashar jiragen ruwa da ke zaɓar, sakawa da sauran ayyuka a cikin wannan tari. A cikin yanayinmu, ba zai yiwu a yi adireshin ɗaya ba; dole ne mu sanya wuraren shigarwa da yawa, saboda na'urori masu kama da kansu waɗanda gungu na VictoriaMetrics ke gudana a cikin yankuna daban-daban na mai samar da girgije iri ɗaya, watau ba cikin girgijenmu ba, amma a waje. .
Muna da faɗakarwa. Muna amfani da shi. Muna amfani da mai sarrafa faɗakarwa daga Prometheus. Muna amfani da Opsgenie da Telegram azaman tashar isar da faɗakarwa. A cikin Telegram suna zuba daga dev, watakila wani abu daga prod, amma galibi wani abu na ƙididdiga, wanda injiniyoyi ke buƙata. Kuma Opsgenie yana da mahimmanci. Waɗannan su ne kira, sarrafa aukuwa.
Tambaya ta har abada: "Wane ne ke sa idanu?" A cikin yanayinmu, saka idanu yana sa ido kan kansa, saboda muna amfani da vmagent akan kowane kumburi. Kuma tunda an rarraba nodes ɗinmu a cikin cibiyoyin bayanai daban-daban na mai ba da sabis ɗaya, kowace cibiyar bayanai tana da tashar tata, suna da zaman kansu, kuma ko da tsagawar kwakwalwa ta zo, har yanzu za mu sami faɗakarwa. Haka ne, za a sami ƙarin su, amma yana da kyau a sami ƙarin faɗakarwa fiye da kowa.
Mun ƙare jerinmu tare da aiwatar da HA.
Kuma ina so in lura da ƙwarewar sadarwa tare da al'ummar VictoriaMetrics. Ya zama tabbatacce. Guys suna amsawa. Suna ƙoƙarin zurfafa cikin kowane harka da aka bayar.
Na fara batutuwa akan GitHub. An warware su cikin sauri. Akwai ƙarin wasu batutuwa guda biyu waɗanda ba a rufe su gaba ɗaya, amma na riga na iya gani daga lambar da ke aiki a cikin wannan hanyar.
Babban zafi a gare ni a lokacin maimaitawa shi ne cewa idan na rufe wani kumburi, sa'an nan na farko 30 seconds vminsert ba zai iya fahimtar cewa babu backend. Yanzu an yanke shawara. Kuma a zahiri a cikin daƙiƙa ɗaya ko biyu, ana ɗaukar bayanan daga duk kuɗaɗen da suka rage, kuma buƙatar ta daina jiran wannan kumburin da ya ɓace.
Mun so a wani lokaci daga VictoriaMetrics ya zama ma'aikacin VictoriaMetrics. Muka jira shi. Yanzu muna rayayye gina wani tsarin ga VictoriaMetrics afareta ya dauki duk pre-lissafi dokoki, da dai sauransu Prometheus, saboda muna sosai rayayye amfani da dokokin da suka zo tare da Prometheus afareta.
Akwai shawarwari don inganta aikin gungu. Na zayyana su a sama.
Kuma ina so in rage misali. A cikin yanayinmu, ana buƙatar saukar da samfur na musamman don kallon abubuwan da ke faruwa. Kusan magana, awo ɗaya ya ishe ni a rana. Ana buƙatar waɗannan halaye na shekara guda, uku, biyar, shekaru goma. Kuma ƙimar awo ɗaya ta isa sosai.
- Mun san zafi, kamar yadda wasu abokan aikinmu suka yi, lokacin amfani da Prometheus.
- Mun zabi VictoriaMetrics da kanmu.
- Yana yin ma'auni da kyau duka a tsaye da a kwance.
- Za mu iya rarraba sassa daban-daban zuwa lambobi daban-daban na nodes a cikin tari, iyakance su ta ƙwaƙwalwar ajiya, ƙara ƙwaƙwalwar ajiya, da sauransu.
Za mu yi amfani da VictoriaMetrics a gida, saboda muna son shi sosai. Wannan shi ne abin da ya kasance kuma abin da ya zama.
Lambobin QR guda biyu don hira ta VictoriaMetrics, lambobin sadarwa na, radar fasaha na LeroyMerlin.
source: www.habr.com