Kumasulira kwa nkhaniyi kunakonzedwa makamaka kwa ophunzira a maphunzirowo .
ndi wopanga mapulogalamu, Go otentheka, ndi kuthetsa mavuto. Ndiwoyang'anira Prometheus komanso woyambitsa nawo Kubernetes SIG zida. M'mbuyomu, anali injiniya wopanga ku SoundCloud ndipo adatsogolera gulu loyang'anira ku CoreOS. Pano akugwira ntchito ku Google.
- Infrastructure Engineer at Improbable. Iye ali ndi chidwi ndi matekinoloje atsopano ndi mavuto a machitidwe ogawidwa. Ali ndi chidziwitso chochepa cha mapulogalamu ku Intel, wothandizira ku Mesos, komanso luso lapadziko lonse lapansi lopanga SRE pa Improbable. Kudzipereka pakukweza dziko la microservices. Zokonda zake zitatu: Golang, gwero lotseguka ndi volleyball.
Kuyang'ana malonda athu odziwika bwino a SpatialOS, mutha kuganiza kuti Zosatheka zimafunikira mawonekedwe amtambo amphamvu kwambiri padziko lonse lapansi okhala ndi magulu ambiri a Kubernetes. Tinali m'modzi mwa oyamba kugwiritsa ntchito njira yowunikira . Prometheus imatha kutsata ma metric mamiliyoni munthawi yeniyeni ndipo imabwera ndi chilankhulo champhamvu chomwe chimakulolani kuti mutulutse zomwe mukufuna.
Kuphweka ndi kudalirika kwa Prometheus ndi chimodzi mwazabwino zake zazikulu. Komabe, titangodutsa sikelo inayake, tinakumana ndi zopinga zingapo. Kuti tithane ndi mavutowa tapanga ndi pulojekiti yotseguka yopangidwa ndi Improbable kuti isinthe mosasunthika masango a Prometheus kukhala njira imodzi yowunikira yokhala ndi zosungirako zakale zopanda malire. Thanos ikupezeka pa Github .
Zolinga zathu ndi Thanos
Pamlingo wina, pamakhala mavuto omwe sangathe kupitilira vanila Prometheus. Momwe mungasungire modalirika komanso mwachuma ma petabytes a mbiri yakale? Kodi izi zingatheke popanda kusokoneza nthawi yoyankha? Kodi ndizotheka kupeza ma metric onse omwe ali pa seva zosiyanasiyana za Prometheus ndi pempho limodzi la API? Kodi pali njira iliyonse yophatikizira zomwe zasonkhanitsidwa pogwiritsa ntchito Prometheus HA?
Kuti tithane ndi mavutowa, tidapanga Thanos. Zigawo zotsatirazi zikufotokoza momwe tinachitira ndi nkhanizi ndikufotokozera zolinga zathu.
Kufunsa za data kuchokera ku zochitika zingapo za Prometheus (funso lapadziko lonse lapansi)
Prometheus imapereka njira yogwira ntchito yopangira sharding. Ngakhale seva imodzi ya Prometheus imapereka scalability yokwanira kumasula ogwiritsa ntchito ku zovuta za sharding yopingasa pafupifupi pafupifupi milandu yonse yogwiritsira ntchito.
Ngakhale kuti iyi ndi chitsanzo chabwino chotumizira, nthawi zambiri zimakhala zofunikira kuti mupeze deta pa seva zosiyanasiyana za Prometheus kudzera mu API imodzi kapena UI - mawonedwe apadziko lonse. Zachidziwikire, ndizotheka kuwonetsa mafunso angapo mugawo limodzi la Grafana, koma funso lililonse limatha kuchitidwa pa seva imodzi ya Prometheus. Kumbali ina, ndi Thanos mutha kufunsa ndikuphatikiza zambiri kuchokera ku maseva angapo a Prometheus popeza onse amapezeka kumapeto amodzi.
M'mbuyomu, kuti tiwone padziko lonse lapansi mu Zosatheka, tidakonza zochitika zathu za Prometheus kukhala magawo angapo. . Izi zikutanthauza kupanga seva imodzi ya Prometheus meta yomwe imasonkhanitsa ma metrics kuchokera pa seva iliyonse yamasamba.
Njira imeneyi inali yovuta. Izi zapangitsa kuti pakhale masanjidwe ovuta kwambiri, kuwonjezeredwa kwa chinthu china chomwe chingathe kulephera, komanso kugwiritsa ntchito malamulo ovuta kuonetsetsa kuti mapeto a federal amangolandira deta yomwe ikufunika. Kuphatikiza apo, chitaganya chamtunduwu sichikulolani kuti muwone dziko lonse lapansi, popeza sizinthu zonse zomwe zimapezeka kuchokera ku pempho limodzi la API.
Zogwirizana kwambiri ndi izi ndikuwona kogwirizana kwa deta yomwe yasonkhanitsidwa pa ma seva apamwamba (HA) Prometheus. Chitsanzo cha Prometheus 'HA chimasonkhanitsa deta kawiri kawiri, chomwe chiri chophweka kwambiri sichingakhale chophweka. Komabe, kugwiritsa ntchito mawonedwe ophatikizika komanso ocheperako a mitsinje yonseyo kungakhale kothandiza kwambiri.
Zachidziwikire, pakufunika ma seva a Prometheus omwe amapezeka kwambiri. Pa Zosatheka, timatenga kuwunika kwa miniti ndi mphindi mozama, koma kukhala ndi chitsanzo chimodzi cha Prometheus pagulu lililonse ndikolephera. Cholakwika chilichonse cha kasinthidwe kapena kulephera kwa hardware kungayambitse kutayika kwa deta yofunikira. Ngakhale kutumiza kosavuta kungayambitse zosokoneza pang'ono pakusonkhanitsidwa kwa ma metrics chifukwa kuyambiranso kumatha kukhala kotalikirapo kuposa nthawi yolemba.
Kusungidwa kodalirika kwa mbiri yakale
Kusungirako zotsika mtengo, zachangu, zanthawi yayitali ndi maloto athu (ogawana ndi ogwiritsa ntchito ambiri a Prometheus). Mu Zosatheka, tinakakamizika kukonza nthawi yosungira ma metrics kukhala masiku asanu ndi anayi (ya Prometheus 1.8). Izi zimawonjezera malire oonekeratu kuti tingayang'ane kutali bwanji.
Prometheus 2.0 yayenda bwino pankhaniyi, popeza kuchuluka kwa nthawi sikukhudzanso magwiridwe antchito onse a seva (onani. ). Komabe, Prometheus amasunga deta pa disk yakomweko. Ngakhale kuponderezana kwa data kwapamwamba kumatha kuchepetsa kwambiri kugwiritsidwa ntchito kwa SSD kwanuko, pamapeto pake pamakhala malire pa kuchuluka kwa mbiri yakale yomwe ingasungidwe.
Kuphatikiza apo, pa Improbable timasamala za kudalirika, kuphweka komanso mtengo. Ma disks akulu am'deralo ndi ovuta kugwiritsa ntchito ndikusunga. Zimawononga ndalama zambiri ndipo zimafuna zida zowonjezera zowonjezera, zomwe zimabweretsa zovuta zosafunikira.
Kutsitsa zitsanzo
Titayamba kugwira ntchito ndi mbiri yakale, tidazindikira kuti pali zovuta zazikulu ndi-O zomwe zimapangitsa kuti mafunso azichedwetsa komanso pang'onopang'ono pamene tikugwira ntchito ndi masabata, miyezi, ndi zaka zambiri.
Njira yothetsera vutoli ingakhale (downsampling) - kuchepetsa kusanja kwa ma sikelo. Ndi downsampling, tikhoza "kutsika" ku nthawi yokulirapo ndikusunga chiwerengero chofanana cha zitsanzo, kusunga mafunso kuti ayankhe.
Kutsitsa deta yakale ndi chofunikira chosapeΕ΅eka cha njira iliyonse yosungiramo nthawi yayitali ndipo sikudutsa vanilla Prometheus.
Zolinga zowonjezera
Chimodzi mwazolinga zoyambilira za pulojekiti ya Thanos chinali kuphatikiza mosasunthika ndi makhazikitsidwe aliwonse a Prometheus omwe alipo. Cholinga chachiwiri chinali chosavuta kugwira ntchito ndi zopinga zochepa zolowera. Kudalira kulikonse kuyenera kukhutitsidwa mosavuta kwa ogwiritsa ntchito ang'onoang'ono ndi akulu, zomwe zikutanthauzanso mtengo wotsika.
Zomangamanga za Thanos
Titatchula zolinga zathu mu gawo lapitalo, tiyeni tigwiritse ntchito ndikuwona momwe Thanos amathetsera mavutowa.
Mawonedwe apadziko lonse lapansi
Kuti tiwone padziko lonse lapansi pazochitika za Prometheus, tifunika kulumikiza malo amodzi ofunsira ku maseva onse. Izi ndi zomwe gawo la Thanos limachita. . Imayikidwa pafupi ndi seva iliyonse ya Prometheus ndipo imagwira ntchito ngati projekiti, imagwiritsa ntchito data ya Prometheus yapafupi kudzera mu gRPC Store API, kulola kuti deta yanthawi yayitali ibwezedwe ndi tag ndi nthawi.
Kumbali inayi ndi gawo laling'ono, lopanda malire la Querier, lomwe silimangoyankha mafunso a PromQL kudzera pa Prometheus HTTP API. Querier, Sidecar ndi zigawo zina za Thanos zimalumikizana kudzera .
- Querier, atalandira pempho, amalumikizana ndi seva yofananira ya Store API, ndiko kuti, ku Sidecars yathu ndipo amalandira deta yanthawi kuchokera kumaseva a Prometheus.
- Pambuyo pake, imaphatikiza mayankho ndikuyankha funso la PromQL pa iwo. Querier akhoza kuphatikiza deta yosagwirizana ndi kubwereza deta kuchokera ku maseva a Prometheus HA.
Izi zimathetsa gawo lalikulu la chithunzi chathu - kuphatikiza deta kuchokera ku maseva a Prometheus akutali kukhala mawonekedwe amodzi. M'malo mwake, Thanos atha kugwiritsidwa ntchito pazomwezi. Palibe zosintha zomwe ziyenera kupangidwa ku ma seva a Prometheus omwe alipo!
Moyo wa alumali wopanda malire!
Komabe, posachedwa tidzafuna kusunga deta kupyola nthawi yosungira ya Prometheus. Tinasankha kusungirako zinthu kuti tisunge mbiri yakale. Imapezeka kwambiri mumtambo uliwonse komanso malo opangira deta ndipo imakhala yokwera mtengo kwambiri. Kuphatikiza apo, pafupifupi chilichonse chosungira zinthu chimapezeka kudzera mu S3 API yodziwika bwino.
Prometheus amalemba zambiri kuchokera ku RAM kupita ku disk pafupifupi maola awiri aliwonse. Deta yosungidwa yosungidwa ili ndi deta yonse kwa nthawi yokhazikika ndipo imakhala yosasinthika. Izi ndizothandiza kwambiri chifukwa Thanos Sidecar amatha kungoyang'ana bukhu la Prometheus ndipo, midadada yatsopano ikapezeka, imawayika mu ndowa zosungira zinthu.
Kulowetsa muzosungirako zinthu mutangolemba ku diski kumakupatsaninso mwayi wosunga kuphweka kwa scraper (Prometheus ndi Thanos Sidecar). Zomwe zimathandizira kuthandizira, mtengo ndi mapangidwe adongosolo.
Monga mukuonera, zosunga zobwezeretsera deta ndi losavuta. Koma bwanji zofunsa za data posungira zinthu?
Chigawo cha Thanos Store chimagwira ntchito ngati wothandizira kuti atengere deta kuchokera kusungirako zinthu. Monga Thanos Sidecar, imatenga nawo gawo pagulu la miseche ndikugwiritsa ntchito Store API. Mwanjira iyi, Querier yomwe ilipo imatha kuiona ngati Sidecar, ngati gwero lina lazinthu zanthawi - palibe kusinthika kwapadera komwe kumafunikira.
Ma block a nthawi amakhala ndi mafayilo akulu angapo. Kuziyika pakufunidwa sikungakhale kothandiza, ndipo kuzisunga kwanuko kungafune malo ambiri okumbukira ndi disk.
M'malo mwake, Store Gateway amadziwa momwe angagwiritsire ntchito mawonekedwe osungira a Prometheus. Chifukwa cha ndandanda yamafunso anzeru komanso kusungitsa magawo ofunikira a midadada, ndizotheka kuchepetsa mafunso ovuta kukhala ochepa opempha a HTTP kukana mafayilo osungira. Mwanjira imeneyi, mutha kuchepetsa kuchuluka kwa zopempha ndi maulamuliro anayi mpaka asanu ndi limodzi ndikukwaniritsa nthawi zoyankhira zomwe nthawi zambiri zimakhala zovuta kusiyanitsa kuchokera ku zopempha kupita ku data pa SSD yakomweko.
Monga momwe tawonetsera m'chithunzi pamwambapa, Thanos Querier amachepetsa kwambiri mtengo pafunso lililonse la deta yosungiramo zinthu mwa kugwiritsa ntchito mawonekedwe osungira a Prometheus ndikuyika deta yogwirizana mbali ndi mbali. Pogwiritsa ntchito njirayi, titha kuphatikiza zopempha zambiri zomwe zili m'modzi kukhala zochepa zogwirira ntchito zambiri.
compacting ndi downsampling
Dongosolo latsopano la nthawi likalowetsedwa bwino m'malo osungira zinthu, timawona ngati "mbiri yakale", yomwe imapezeka nthawi yomweyo kudzera pa Store Gateway.
Komabe, pakapita nthawi, midadada kuchokera ku gwero limodzi (Prometheus wokhala ndi Sidecar) amaunjikana ndipo sagwiritsanso ntchito kuthekera konse kolozera. Kuti tithane ndi vutoli, tidayambitsa gawo lina lotchedwa Compactor. Imangogwiritsa ntchito injini yophatikizira ya Prometheus kuzinthu zakale zosungira zinthu ndipo imatha kuyendetsedwa ngati ntchito yosavuta ya batch nthawi ndi nthawi.
Chifukwa cha kukanikiza koyenera, kufunsa zosungirako kwa nthawi yayitali sikubweretsa vuto potengera kukula kwa data. Komabe, mtengo womwe ungakhalepo wotsegulira mabiliyoni amtengo wapatali ndikuwuyendetsa kudzera pa purosesa yamafunso mosakayikira umabweretsa kuwonjezeka kwakukulu kwa nthawi yofunsa mafunso. Kumbali inayi, popeza pali mazana a mfundo za data pa pixel pazenera, zimakhala zosatheka ngakhale kuwonera detayo pakutha kwathunthu. Chifukwa chake, kutsitsa sikungotheka, komanso sikungabweretse kutayika kolondola.
Kuti muchepetse deta, Compactor imasonkhanitsa deta mosalekeza pamphindi zisanu ndi ola limodzi. Pa chunk iliyonse yaiwisi yosungidwa pogwiritsa ntchito TSDB XOR compression, mitundu yosiyanasiyana ya data yonse imasungidwa, monga min, max kapena sum pa block imodzi. Izi zimalola Querier kuti azisankha zokha zophatikiza zomwe zili zoyenera pafunso la PromQL.
Palibe masinthidwe apadera omwe amafunikira kuti wogwiritsa ntchito agwiritse ntchito deta yocheperako. Querier amasintha okha pakati pa zosintha zosiyanasiyana ndi data yaiwisi pomwe wogwiritsa ntchito amalowa ndi kutuluka. Ngati angafune, wogwiritsa ntchitoyo atha kuwongolera izi mwachindunji kudzera pagawo la "sitepe" mu pempho.
Popeza mtengo wosunga GB imodzi ndi wotsika, mokhazikika Thanos amasunga data yaiwisi, mphindi zisanu ndi ola limodzi. Palibe chifukwa chochotsa deta yoyambirira.
Malamulo ojambulira
Ngakhale ndi Thanos, malamulo ojambulira ndi gawo lofunikira kwambiri pakuwunika. Amachepetsa zovuta, latency, ndi mtengo wa mafunso. Ndiwosavuta kuti ogwiritsa ntchito apeze data yophatikizidwa ndi ma metric. Thanos imachokera pazochitika za vanilla Prometheus, kotero ndizovomerezeka kusunga malamulo ojambulira ndi malamulo ochenjeza pa seva ya Prometheus yomwe ilipo. Komabe, nthawi zina izi sizingakhale zokwanira:
- Chidziwitso chapadziko lonse lapansi ndi lamulo (mwachitsanzo, chenjezo ngati ntchito sikugwira ntchito pamagulu opitilira awiri mwa atatu).
- Lamulo la zomwe zili kunja kwa malo osungira.
- Chikhumbo chosunga malamulo onse ndi zidziwitso pamalo amodzi.
Pazochitika zonsezi, Thanos ikuphatikiza gawo lina lotchedwa Wolamulira, lomwe limawerengera malamulo ndi chenjezo kudzera pa Thanos Queries. Popereka StoreAPI yodziwika bwino, Query node imatha kupeza ma metric omwe angopangidwa kumene. Pambuyo pake amasungidwanso m'malo osungira zinthu ndikuperekedwa kudzera pa Store Gateway.
Mphamvu ya Thanos
Thanos ndi yosinthika mokwanira kuti ikhale yogwirizana ndi zosowa zanu. Izi ndizothandiza makamaka mukasamuka kuchokera ku Prometheus. Tiyeni tibwereze mwachangu zomwe taphunzira pazigawo za Thanos ndi chitsanzo chachangu. Umu ndi momwe mungatengere vanila Prometheus kudziko la "zosungirako zopanda malire":
- Onjezani Thanos Sidecar ku maseva anu a Prometheus - mwachitsanzo, chidebe cham'mbali mwa Kubernetes pod.
- Ikani zambiri za Thanos Querier kuti muwone zambiri. Pakadali pano ndizosavuta kukhazikitsa miseche pakati pa Scraper ndi Querier. Kuti muwone kugwirizana kwa zigawo, gwiritsani ntchito metric ya 'thanos_cluster_members'.
Masitepe awiriwa ndi okwanira kuti apereke mawonekedwe apadziko lonse lapansi komanso kuchotseratu deta mosasunthika kuchokera ku zofananira za Prometheus HA! Ingolumikizani ma dashboards anu kumapeto kwa Querier HTTP kapena gwiritsani ntchito Thanos UI mwachindunji.
Komabe, ngati mukufuna zosunga zobwezeretsera zama metrics ndikusunga kwa nthawi yayitali, muyenera kumaliza njira zina zitatu:
- Pangani chidebe cha AWS S3 kapena GCS. Konzani Sidecar kuti mukopere deta kuzidebe izi. Kusungidwa kwa data komweko kutha kuchepetsedwa.
- Tsegulani Store Gateway ndikulumikiza ndi gulu lanu lamiseche lomwe lilipo. Tsopano mutha kufunsa zomwe zasungidwa!
- Deploy Compactor kuti muwongolere bwino zamafunso pakanthawi yayitali pogwiritsa ntchito compaction ndi kutsitsa.
Ngati mukufuna kudziwa zambiri, musazengereze kuyang'ana pa athu ΠΈ !
M'masitepe asanu okha, tidatembenuza Prometheus kukhala njira yodalirika yowunikira ndikuwona padziko lonse lapansi, nthawi yosungirako zopanda malire komanso kupezeka kwakukulu kwa ma metric.
Kokani pempho: tikukufunani!
wakhala ntchito yotseguka gwero kuyambira pachiyambi. Kuphatikizika kosasunthika ndi Prometheus komanso kutha kugwiritsa ntchito gawo la Thanos kumapangitsa kukhala chisankho chabwino kwambiri pakuwongolera dongosolo lanu lowunikira mosavutikira.
Nthawi zonse timalandila Zopempha za GitHub Pull ndi Nkhani. Pakadali pano, omasuka kulumikizana nafe kudzera pa Github Issues kapena slackngati muli ndi mafunso kapena ndemanga, kapena mukufuna kugawana zomwe mwakumana nazo mukugwiritsa ntchito! Ngati mumakonda zomwe timachita ku Improbable, musazengereze kutilumikizana nafe - !
Source: www.habr.com