Ukuguqulelwa kwenqaku kulungiselelwe ngokukodwa abafundi bekhosi .
ngumphuhlisi wesoftware, Yiya ngokwenzondelelo, kunye nosombululi engxakini. Ukwangumgcini wePrometheus kunye nomseki we-Kubernetes SIG isixhobo. Ngaphambili, wayeyinjineli yemveliso kwi-SoundCloud kwaye wayekhokela iqela lokubeka iliso kwi-CoreOS. Okwangoku usebenza eGoogle.
-Injineli yeziseko zophuhliso kwi-Improbable. Unomdla kubuchwephesha obutsha kunye neengxaki zeenkqubo ezisasazwayo. Unamava okuprograma akwinqanaba elisezantsi e-Intel, amava omnikeli e-Mesos, kunye namava emveliso ye-SRE yodidi lwehlabathi kwi-Improbable. Inikezelwe ekuphuculeni ihlabathi leenkonzo ezincinci. Uthando lwakhe oluthathu: iGolang, umthombo ovulekileyo kunye nebhola yevoli.
Ukujonga imveliso yethu yeflegi ye-SpatialOS, unokuqikelela ukuba i-Improbable ifuna amandla amakhulu, isiseko selifu somgangatho wehlabathi kunye neqela lamaqela e-Kubernetes. Saba ngabokuqala ukusebenzisa inkqubo yokubeka iliso . I-Prometheus iyakwazi ukulandelela izigidi zeemethrikhi ngexesha lokwenyani kwaye iza nolwimi olunamandla lombuzo olukuvumela ukuba ukhuphe ulwazi oludingayo.
Ukulula kunye nokuthembeka kwePrometheus yenye yeenzuzo zayo eziphambili. Noko ke, sakuba sigqithile kumlinganiselo othile, siye sadibana nemiqobo emininzi. Ukusombulula ezi ngxaki siye saphuhlisa yiprojekthi yomthombo ovulekileyo owenziwe yi-Improbable ukuguqula ngokugqibeleleyo amaqela e-Prometheus asele ekhona kwinkqubo eyodwa yokubeka iliso kunye nokugcinwa kwedatha yembali engenamkhawulo. I-Thanos iyafumaneka kwi-Github .
Iinjongo zethu noThanos
Kwinqanaba elithile, kuvela iingxaki ezingaphaya kwamandla e-vanilla Prometheus. Indlela yokugcina ngokuthembekileyo nangoqoqosho iipetabytes zedatha yembali? Ngaba oku kunokwenziwa ngaphandle kokubeka esichengeni ixesha lokuphendula? Ngaba kuyenzeka ukufikelela kuzo zonke iimetrics ezibekwe kwiiseva ezahlukeneyo zePrometheus ngesicelo esinye se-API? Ngaba kukho nayiphi na indlela yokudibanisa idatha ephindwe kabini eqokelelwe usebenzisa i-Prometheus HA?
Ukujongana nale miba, senze iThanos. Amacandelo alandelayo achaza indlela esiyijonga ngayo le miba kwaye achaze iinjongo zethu.
Ukubuza idatha kwiziganeko ezininzi zePrometheus (umbuzo wehlabathi)
I-Prometheus inikezela ngendlela esebenzayo yokwahlula. Nokuba iseva enye ye-Prometheus ibonelela nge-scalability eyaneleyo yokukhulula abasebenzisi kubunzima be-sharding ethe tyaba phantse kuzo zonke iimeko zokusetyenziswa.
Nangona le imodeli enkulu yokuthunyelwa, kudla ngokuyimfuneko ukufikelela kwidatha kwiiseva ezahlukeneyo zePrometheus nge-API enye okanye i-UI - imbono yehlabathi. Ewe kunjalo, kuyenzeka ukubonisa imibuzo emininzi kwiqela elinye leGrafana, kodwa umbuzo ngamnye unokuphunyezwa kuphela kwiseva enye yePrometheus. Kwelinye icala, ngeThanos unokubuza kwaye udibanise idatha kwiiseva ezininzi zePrometheus kuba zonke zifikeleleka kwindawo enye.
Ngaphambili, ukufumana umbono wehlabathi kwi-Improbable, salungiselela iimeko zethu zePrometheus kumanqanaba amaninzi. . Oku kuthetha ukudala iseva enye ye-Prometheus yemeta eqokelela ezinye zeemetrics ukusuka kumncedisi wegqabi ngalinye.
Le ndlela yabonakala iyingxaki. Oku kubangele ulungelelwaniso olunzima ngakumbi, ukongezwa kwendawo eyongezelelweyo yokungaphumeleli, kunye nokusetyenziswa kwemithetho enzima ukuqinisekisa ukuba i-endpoint edibeneyo ifumana kuphela idatha eyidingayo. Ukongeza, olu hlobo lwe-federation alukuvumeli ukuba ufumane umbono wehlabathi lokwenyani, kuba ingeyiyo yonke idatha ekhoyo kwisicelo esinye se-API.
Okunxulumene ngokusondeleyo noku kumbono odibeneyo wedatha eqokelelwe kwi-high-availability (HA) iiseva zePrometheus. Imodeli ye-Prometheus 'HA iqokelela ngokuzimeleyo idatha kabini, into elula kakhulu ayinakuba lula. Nangona kunjalo, ukusebenzisa imboniselo edibeneyo kunye nephindwe kabini kuyo yomibini imijelo kuya kuba lula ngakumbi.
Ewe kunjalo, kukho imfuneko yeeseva zePrometheus ezifumaneka kakhulu. Kwi-Improbable, sithatha ukujonga idatha yomzuzu ngomzuzu nzulu ngokwenene, kodwa ukuba nomzekelo omnye wePrometheus kwiqela ngalinye linqaku elinye lokusilela. Nayiphi na impazamo yoqwalaselo okanye ukungaphumeleli kwehardware kunokukhokelela ekulahlekeni kwedatha ebalulekileyo. Nokuthunyelwa okulula kunokubangela ukuphazamiseka okuncinci kuqokelelo lweemetriki kuba ukuqalisa kwakhona kunokuba kude kakhulu kunexesha lokukhuhla.
Ukugcinwa okuthembekileyo kwedatha yembali
Ixabiso eliphantsi, elikhawulezayo, ixesha elide lokugcina iimetrics liphupha lethu (ekwabelwana ngalo ngabasebenzisi abaninzi bePrometheus). Kwi-Improbable, siye sanyanzeleka ukuba siqwalasele ixesha lokugcinwa kweemetrics ukuya kwiintsuku ezisithoba (kwi-Prometheus 1.8). Oku kongeza imida ecacileyo kwindlela esinokujonga ngayo umva.
I-Prometheus 2.0 iphuculwe kulo mba, kuba inani lexesha lexesha alisachaphazeli ukusebenza ngokubanzi kweseva (bona. ). Nangona kunjalo, i-Prometheus igcina idatha kwidiski yendawo. Nangona ukunyanzeliswa kwedatha ephezulu kunokunciphisa kakhulu ukusetyenziswa kwe-SSD yendawo, ekugqibeleni kusekho umda kwinani ledatha yembali enokugcinwa.
Ukongeza, kwi-Improbable sikhathalele ukuthembeka, ukulula kunye neendleko. Iidiski ezinkulu zengingqi zinzima kakhulu ukusebenza kunye nokugcina umva. Zixabisa ngaphezulu kwaye zifuna izixhobo zokuxhasa ngakumbi, ezikhokelela kubunzima obungeyomfuneko.
Ukuthoba iisampulu
Nje ukuba siqale ukusebenza ngedatha yembali, siye safumanisa ukuba kukho ubunzima obusisiseko nge-Big-O eyenza imibuzo icothe kwaye icothe njengoko sisebenza ngeeveki, iinyanga, kunye neminyaka yedatha.
Isisombululo esisemgangathweni kule ngxaki siya kuba (downsampling) - ukunciphisa umqondiso wesampulu frequency. Ngokuthotywa kwesampulu, singakwazi "ukwehlisa" ukuya kuluhlu lwexesha elikhulu kwaye sigcine inani elifanayo leesampuli, sigcina imibuzo iphendula.
Ukuthoba idatha endala kuyimfuno engenakuphepheka yaso nasiphi na isisombululo sokugcina ixesha elide kwaye ingaphaya kobubanzi be-vanilla Prometheus.
Usukelo olongezelelweyo
Enye yeenjongo zokuqala zeprojekthi yeThanos yayikukudibanisa ngaphandle komthungo nalo naluphi na ufakelo lwePrometheus olukhoyo. Injongo yesibini yayilula ukusebenza kunye nezithintelo ezincinci zokungena. Nakuphi na ukuxhomekeka kufuneka kwaneliseke ngokulula kubo bobabini abancinci nabasebenzisi abakhulu, nto leyo ethetha ixabiso elisezantsi.
Uyilo lweThanos
Emva kokudwelisa iinjongo zethu kwicandelo elidlulileyo, masisebenze ngazo kwaye sibone ukuba i-Thanos isombulula njani ezi ngxaki.
Imbono yehlabathi
Ukufumana umbono wehlabathi lonke ngaphezulu kweemeko ezikhoyo zePrometheus, kufuneka sidibanise indawo enye yokungena yesicelo kuzo zonke iiseva. Yile nto kanye eyenziwa licandelo leThanos. . Ifakwe ecaleni kweseva nganye ye-Prometheus kwaye isebenza njenge-proxy, ikhonza idatha ye-Prometheus yendawo nge-GRPC Store API, evumela ukuba idatha yoluhlu lwexesha ibuyiswe ngethegi kunye noluhlu lwexesha.
Kwelinye icala sisikali sokuphuma, icandelo le-Querier elingenammiselo, elenza okungaphezulu nje kokuphendula imibuzo yePromQL ngePrometheus HTTP API eqhelekileyo. Querier, Sidecar kunye namanye amacandelo eThanos anxibelelana nge .
- I-Querier, xa ifumana isicelo, idibanisa kwi-server ye-Store API ehambelanayo, oko kukuthi, kwii-Sidecars zethu kwaye ifumana idatha yochungechunge lwexesha kwiiseva ezihambelanayo ze-Prometheus.
- Emva koko, idibanisa iimpendulo kwaye iqhube umbuzo we-PromQL kubo. I-Querier inokudibanisa zombini i-disjoint data kunye nedatha ephindwe kabini kwiiseva ze-Prometheus HA.
Oku kusombulula iqhekeza eliphambili lephazili yethu - ukudibanisa idatha kwiiseva ezikwanti zePrometheus kwimbono enye. Ngapha koko, iThanos inokusetyenziselwa eli nqaku kuphela. Akukho tshintsho kufuneka lwenziwe kwiiseva ezikhoyo zePrometheus!
Ubomi beshelufa obungenamkhawulo!
Nangona kunjalo, ngokukhawuleza okanye kamva siya kufuna ukugcina idatha ngaphaya kwexesha eliqhelekileyo lokugcinwa kwePrometheus. Sakhetha ukugcinwa kwezinto zokugcina idatha yembali. Ifumaneka ngokubanzi kulo naliphi na ilifu kunye nakumaziko edatha akwindawo kwaye ixabisa kakhulu. Ukongeza, phantse nayiphi na into yokugcina into ifumaneka nge-S3 API eyaziwayo.
U-Prometheus ubhala idatha esuka kwi-RAM ukuya kwidiski malunga neeyure ezimbini. Ibhloko yedatha egciniweyo iqulethe yonke idatha yexesha elimiselweyo kwaye ayinakuguqulwa. Oku kulula kakhulu kuba i-Thanos Sidecar inokujonga ngokulula kwi-database ye-Prometheus kwaye, njengoko iibhloko ezintsha zifumaneka, zilayishe kwiibhakethi zokugcina izinto.
Ukulayisha kwindawo yokugcina into ngokukhawuleza emva kokubhala kwidiski kwakhona kukuvumela ukuba ugcine ukulula kwe-scraper (i-Prometheus kunye ne-Thanos Sidecar). Eyenza lula inkxaso, iindleko kunye noyilo lwenkqubo.
Njengoko ubona, ukugcinwa kwedatha kulula kakhulu. Kodwa kuthekani malunga nokubuza idatha kwindawo yokugcina into?
Icandelo le-Thanos Store lisebenza njenge-proxy yokubuyisela idatha kwindawo yokugcina into. NjengoThanos Sidecar, ithatha inxaxheba kwiqela le-gossip kwaye isebenzise i-Store API. Ngale ndlela, i-Querier ekhoyo inokuyiphatha njenge-Sidecar, njengomnye umthombo wedatha yochungechunge lwexesha - akukho lungiselelo lukhethekileyo olufunekayo.
Iibhloko zedatha yoluhlu lwexesha ziquka ezininzi iifayile ezinkulu. Ukuzilayisha kwimfuno akunakusebenza kakuhle, kwaye ukuzigcina kwindawo yangakini kuya kufuna isixa esikhulu sememori kunye nendawo yediski.
Endaweni yoko, iSango seSango liyayazi indlela yokuphatha ifomathi yokugcina iPrometheus. Ndiyabulela kwi-smart query scheduler kunye ne-caching kuphela iindawo zesalathisi eziyimfuneko zeebhloko, kunokwenzeka ukunciphisa imibuzo enzima kwinani elincinci lezicelo ze-HTTP kwiifayile zokugcina izinto. Ngale ndlela, unokunciphisa inani lezicelo ngee-odolo ezine ukuya kwezintandathu zobukhulu kwaye ufezekise amaxesha okuphendula ngokuqhelekileyo kunzima ukwahlula kwizicelo kwidatha kwi-SSD yendawo.
Njengoko kubonisiwe kumzobo ongentla, i-Thanos Querier inciphisa kakhulu indleko ngombuzo ngamnye wedatha yokugcina into ngokusebenzisa i-Prometheus ifomathi yokugcina kunye nokubeka idatha ehambelanayo ecaleni. Ngokusebenzisa le ndlela, sinokudibanisa izicelo ezininzi ezilodwa kwinani elincinci lemisebenzi emininzi.
Ukunyanzeliswa kunye nokuthotywa kwesampulu
Nje ukuba ibhloko entsha yedatha yoluhlu lwexesha ilayishwe ngempumelelo kwindawo yokugcina into, siyiphatha njengedatha "yembali", efumaneka ngokukhawuleza ngeSango leSitolo.
Nangona kunjalo, emva kwexesha elithile, iibhloko ezivela kumthombo omnye (i-Prometheus ene-Sidecar) ziqokelela kwaye azisayi kusebenzisa i-indexing epheleleyo. Ukusombulula le ngxaki, sazisa elinye icandelo elibizwa ngokuba yiCompactor. Isebenza ngokulula i-Prometheus 'injini yokudibanisa yendawo kwidatha yembali kwindawo yokugcina into kwaye inokuqhutywa njengomsebenzi olula webhetshi.
Ngombulelo kuxinzelelo olusebenzayo, ukubuza ugcino ixesha elide akubangeli ngxaki ngokobungakanani bedatha. Nangona kunjalo, iindleko ezinokubakho zokukhupha amaxabiso ebhiliyoni kunye nokuziqhuba ngeprosesa yombuzo ngokuqinisekileyo kuya kubangela ukonyuka okumangalisayo kwexesha lokwenziwa kombuzo. Kwelinye icala, kuba kukho amakhulukhulu eengongoma zedatha ngepikseli nganye kwiscreen, kuba nzima nokuba nomfanekiso ngqondweni wedatha kwisisombululo esipheleleyo. Ngaloo ndlela, i-downsampling ayinakwenzeka kuphela, kodwa kwakhona ayiyi kukhokelela ekulahlekeni okubonakalayo kokuchaneka.
Ukuthoba idatha, iCompactor iqhubeka idibanisa idatha kwisisombululo semizuzu emihlanu kunye neyure enye. Kwichunk nganye ekrwada ekhowudiweyo kusetyenziswa ucinezelo lwe-TSDB XOR, iindidi ezahlukeneyo zedatha edityanisiweyo zigcinwa, ezinje ngemin, max okanye isamba sebhloko enye. Oku kuvumela i-Querier ukuba ikhethe ngokuzenzekelayo iaggregate efanele umbuzo onikiweyo wePromQL.
Akukho lungelelwaniso lukhethekileyo olufunekayo ukuze umsebenzisi asebenzise idatha echanekileyo encitshisiweyo. Querier itshintsha ngokuzenzekelayo phakathi izisombululo ezahlukeneyo kunye nedatha ekrwada njengoko umsebenzisi zooms ngaphakathi nangaphandle. Ukuba unqwenela, umsebenzisi unokulawula oku ngokuthe ngqo nge "nyathelo" iparameter kwisicelo.
Ekubeni iindleko zokugcina i-GB enye iphantsi, ngokungagqibekanga i-Thanos igcina idatha eluhlaza, idatha yemizuzu emihlanu kunye neyure enye. Akukho mfuneko yokucima idatha yoqobo.
Imigaqo yokurekhoda
Ndikunye neThanos, imithetho yokurekhoda iyinxalenye ebalulekileyo yesitaki sokubeka iliso. Banciphisa ubunzima, ukubambezeleka, kunye neendleko zemibuzo. Zikwaluncedo kubasebenzisi ukufumana idatha edityanisiweyo ngeemethrikhi. I-Thanos isekelwe kwimizekelo ye-vanilla Prometheus, ngoko ke yamkelekile ngokugqibeleleyo ukugcina imithetho yokurekhoda kunye nemithetho yokulumkisa kwi-server ye-Prometheus ekhoyo. Nangona kunjalo, kwezinye iimeko oku akunakukwanela:
- Isilumkiso sehlabathi kunye nomthetho (umzekelo, isilumkiso xa inkonzo ingasebenzi ngaphezu kwesibini kwamaqela amathathu).
- Umthetho wedatha ngaphandle kwendawo yokugcina.
- Umnqweno wokugcina yonke imithetho kunye nezilumkiso kwindawo enye.
Kuzo zonke ezi meko, i-Thanos ibandakanya icandelo elahlukileyo elibizwa ngokuba nguMlawuli, odibanisa umthetho kunye nesilumkiso ngeThanos Queries. Ngokubonelela nge-StoreAPI eyaziwayo, i-Query node inokufikelela kwi-metrics esandul 'ikhompuyutha. Kamva zikwagcinwa kwindawo yokugcina izinto kwaye zenziwe zifumaneke nge-Store Gateway.
Amandla eThanos
I-Thanos ibhetyebhetye ngokwaneleyo ukuba ilungiselelwe ukuba ihambelane neemfuno zakho. Oku kuluncedo ngakumbi xa ufuduka kwi-Prometheus ethafeni. Masiphinde sibuyele ngokukhawuleza oko sikufundileyo malunga nezixhobo zeThanos ngomzekelo okhawulezayo. Nantsi indlela yokuthatha i-vanilla Prometheus yakho kwihlabathi "lokugcinwa kweemetriki ezingenamkhawulo":
- Yongeza i-Thanos Sidecar kwiiseva zakho ze-Prometheus-umzekelo, isikhongozeli se-sidecar kwi-Kubernetes pod.
- Sebenzisa iikopi ezininzi ze-Thanos Querier ukuze ukwazi ukujonga idatha. Kweli nqanaba kulula ukuseta intlebendwane phakathi kweScraper kunye neQuerier. Ukujonga intsebenziswano yamacandelo, sebenzisa i-'thanos_cluster_members' metric.
La manyathelo mabini anele ukubonelela ngembono yehlabathi kunye nokutsalwa kwedatha engenamthungo ukusuka kwi-replicas ye-Prometheus HA! Qhagamshela ngokulula iideshibhodi zakho kwisiphelo se-Querier HTTP okanye usebenzise i-Thanos UI ngqo.
Nangona kunjalo, ukuba ufuna i-metrics backup kunye nokugcinwa kwexesha elide, kuya kufuneka ugcwalise amanye amanyathelo amathathu:
- Yenza i-AWS S3 okanye i-GCS ibhakethi. Qwalasela iSidecar ukukhuphela idatha kula mabhakethi. Ukugcinwa kwedatha yendawo ngoku kungancitshiswa.
- Deploy Store Gateway kwaye uyidibanise neqela lakho elikhoyo lokuhleba. Ngoku ungabuza idatha egciniweyo!
- Sebenzisa iCompactor ukuphucula impumelelo yemibuzo kwixesha elide usebenzisa ukucudisa kunye nokuthoba iisampulu.
Ukuba ufuna ukwazi ngakumbi, unganqikazi ujonge yethu ΠΈ !
Ngamanyathelo amahlanu nje, siguqule i-Prometheus ibe yinkqubo ethembekileyo yokubeka iliso enokubonwa kwehlabathi, ixesha lokugcina elingenamkhawulo kunye nokufumaneka okuphezulu kweemetriki.
Tsala isicelo: siyakufuna!
ibiyiprojekthi yomthombo ovulekileyo ukusuka ekuqaleni. Udibaniso olungenamthungo kunye nePrometheus kunye nokukwazi ukusebenzisa nje inxalenye yeThanos kuyenza ibe lolona khetho lubalaseleyo lokukala inkqubo yakho yokubeka iliso ngaphandle komgudu.
Sihlala sisamkela izicelo zokutsalwa kweGitHub kunye neMiba. Okwangoku, zive ukhululekile ukuqhagamshelana nathi nge-Github Issues okanye i-slackukuba unemibuzo okanye impendulo, okanye ufuna ukwabelana ngamava akho usebenzisa! Ukuba uyayithanda into esiyenzayo kwi-Improbable, ungathandabuzi ukuqhagamshelana nathi - !
umthombo: www.habr.com