Ukuhunyushwa kwalesi sihloko kulungiselelwe ngokukhethekile abafundi bezifundo .
ungumthuthukisi wesofthiwe, Hamba ngokwedlulele, futhi umxazululi wezinkinga. Uphinde abe ngumnakekeli we-Prometheus kanye nomsunguli we-Kubernetes SIG instrumentation. Esikhathini esedlule, wayengunjiniyela wokukhiqiza e-SoundCloud futhi ehola ithimba lokuqapha e-CoreOS. Okwamanje usebenza kwa-Google.
- Ingqalasizinda Engineer at Improbable. Unentshisekelo kubuchwepheshe obusha kanye nezinkinga zezinhlelo ezisatshalaliswa. Unolwazi lokuhlela olusezingeni eliphansi kwa-Intel, isipiliyoni somnikeli e-Mesos, nolwazi lokukhiqiza lwe-SRE olusezingeni lomhlaba kwa-Improbable. Kunikezelwe ukuthuthukisa umhlaba wama-microservices. Izinto ezintathu azithandayo: i-Golang, umthombo ovulekile kanye ne-volleyball.
Uma ubheka umkhiqizo wethu ohamba phambili we-SpatialOS, ungaqagela ukuthi i-Improbable idinga ingqalasizinda yamafu esikali somhlaba wonke enamaqoqo amaningi e-Kubernetes. Sibe ngabokuqala ukusebenzisa uhlelo lokuqapha . I-Prometheus iyakwazi ukulandelela izigidi zamamethrikhi ngesikhathi sangempela futhi iza nolimi lombuzo olunamandla olukuvumela ukuthi ukhiphe ulwazi oludingayo.
Ubulula nokuthembeka kwe-Prometheus kungenye yezinzuzo zayo eziyinhloko. Nokho, lapho sesidlule esikalini esithile, sahlangabezana nezinkinga ezimbalwa. Ukuxazulula lezi zinkinga sithuthukile iphrojekthi yomthombo ovulekile edalwe i-Improbable ukuguqula kalula amaqoqo akhona e-Prometheus ibe uhlelo olulodwa lokuqapha olunomlando ongenamkhawulo wokugcina idatha. I-Thanos iyatholakala ku-Github .
Imigomo yethu ne-Thanos
Ngezinga elithile, kuphakama izinkinga ezingaphezu kwamandla e-vanilla Prometheus. Ungawagcina kanjani ngokuthembekile nangokwezomnotho ama-petabyte edatha yomlando? Ingabe lokhu kungenziwa ngaphandle kokuphazamisa isikhathi sokuphendula? Ingabe kungenzeka ukufinyelela wonke amamethrikhi atholakala kumaseva ahlukene e-Prometheus ngesicelo esisodwa se-API? Ingabe ikhona indlela yokuhlanganisa idatha ephindaphindiwe eqoqwe kusetshenziswa i-Prometheus HA?
Ukubhekana nalezi zinkinga, sidale i-Thanos. Izigaba ezilandelayo zichaza indlela esizisingatha ngayo lezi zinkinga futhi zichaza nemigomo yethu.
Ukubuza ngedatha evela kuzimo eziningi ze-Prometheus (umbuzo womhlaba wonke)
I-Prometheus inikeza indlela esebenzayo yokwabelana. Ngisho nesiphakeli se-Prometheus esisodwa sinikeza ukulinganisa okwanele ukukhulula abasebenzisi kusukela ezinkingeni ze-sharding enezingqimba cishe kuzo zonke izimo zokusetshenziswa.
Nakuba lena kuyimodeli enhle yokuthunyelwa, ngokuvamile kuyadingeka ukufinyelela idatha kumaseva ahlukene e-Prometheus nge-API eyodwa noma i-UI - ukubuka komhlaba wonke. Yebo, kungenzeka ukubonisa imibuzo eminingi kuphaneli eyodwa ye-Grafana, kodwa umbuzo ngamunye ungenziwa kuphela kuseva eyodwa ye-Prometheus. Ngakolunye uhlangothi, nge-Thanos ungabuza futhi uhlanganise idatha kusuka kumaseva amaningi e-Prometheus njengoba wonke efinyeleleka endaweni eyodwa yokugcina.
Ngaphambilini, ukuze uthole ukubuka komhlaba wonke ku-Improbable, sihlele izimo zethu ze-Prometheus zaba ngamaleveli amaningi. . Lokhu kusho ukudala iseva eyodwa ye-Prometheus meta eqoqa amanye ama-metrics kusuka kuseva yeqabunga ngalinye.
Le ndlela yabonakala iyinkinga. Lokhu kuye kwaholela ekucushweni okuyinkimbinkimbi kakhulu, ukwengezwa kwephuzu elingeziwe elingaba khona lokuhluleka, kanye nokusetshenziswa kwemithetho eyinkimbinkimbi ukuze kuqinisekiswe ukuthi indawo yokugcina ehlanganisiwe ithola kuphela idatha eyidingayo. Ngaphezu kwalokho, lolu hlobo lomfelandawonye alukuvumeli ukuthi uthole ukubuka kwangempela komhlaba wonke, njengoba kungeyona yonke idatha etholakala esicelweni esisodwa se-API.
Okuhlobene eduze nalokhu ukubuka okuhlanganisiwe kwedatha eqoqwe kumaseva atholakala kakhulu (HA) Prometheus. Imodeli ye-Prometheus 'HA iqoqa ngokuzimela idatha kabili, okulula kakhulu okungeke kube lula. Kodwa-ke, ukusebenzisa ukubuka okuhlanganisiwe nokuphindiwe kwakho kokubili ukusakaza kungaba lula kakhulu.
Vele, kunesidingo samaseva e-Prometheus atholakala kakhulu. Ku-Improbable, sithatha ukuqapha kwedatha komzuzu nomzuzu njengokubalulekile, kodwa ukuba nesenzakalo esisodwa se-Prometheus ngeqoqo ngalinye kuyiphuzu elilodwa lokwehluleka. Noma yiliphi iphutha lokumisa noma ukwehluleka kwehadiwe kungase kuholele ekulahlekelweni kwedatha ebalulekile. Ngisho nokuthunyelwa okulula kungabangela ukuphazamiseka okuncane ekuqoqweni kwamamethrikhi ngoba ukuqalisa kabusha kungase kube isikhathi eside kakhulu kunesikhawu sokuklwebha.
Ukugcinwa okuthembekile kwedatha yomlando
Isitoreji samamethrikhi esishibhile, esisheshayo, sesikhathi eside siyiphupho lethu (labiwe abasebenzisi abaningi be-Prometheus). Ku-Improbable, saphoqeleka ukuthi silungiselele isikhathi sokugcinwa kwamamethrikhi sibe izinsuku eziyisishiyagalolunye (ku-Prometheus 1.8). Lokhu kwengeza imingcele esobala yokuthi singabheka kude kangakanani.
I-Prometheus 2.0 isithuthukile kulokhu, njengoba inani lochungechunge lwesikhathi alisathinti ukusebenza kweseva kukonke (bona. ). Nokho, u-Prometheus ugcina idatha kudiski yendawo. Nakuba ukucindezela kwedatha okusebenza kahle kakhulu kunganciphisa kakhulu ukusetshenziswa kwasendaweni kwe-SSD, ekugcineni kusenomkhawulo enanini ledatha yomlando engagcinwa.
Ukwengeza, ku-Improbable siyakukhathalela ukwethembeka, ubulula kanye nezindleko. Amadiski endawo amakhulu anzima kakhulu ukuwasebenzisa nokwenza ikhophi yasenqolobaneni. Zibiza kakhulu futhi zidinga amathuluzi amaningi okulondoloza, okuholela enkingeni engadingekile.
Ukwehlisa isampula
Uma sesiqale ukusebenza ngedatha yomlando, sabona ukuthi kukhona ubunzima obuyisisekelo nge-big-O eyenza imibuzo ihambe kancane futhi ihambe kancane njengoba sisebenza ngamaviki, izinyanga, neminyaka yedatha.
Isixazululo esijwayelekile sale nkinga kungaba (downsampling) - ukunciphisa imvamisa yesampula yesignali. Ngesampula ephansi, singakwazi "ukwehlisela phansi" ebangeni lesikhathi elikhudlwana futhi sigcine inani elifanayo lamasampuli, sigcina imibuzo iphendula.
Ukwehlisa idatha endala kuyisidingo esingenakugwenywa sanoma yisiphi isisombululo sesikhathi eside futhi kungaphezu kobubanzi be-vanilla Prometheus.
Imigomo eyengeziwe
Enye yezinhloso zangempela zephrojekthi ye-Thanos bekuwukuhlanganisa ngaphandle komthungo nanoma yikuphi ukufakwa kwe-Prometheus okukhona. Umgomo wesibili bekuwukusebenza kalula nezithiyo ezincane zokungena. Noma yikuphi ukuncika kufanele kweneliseke kalula kubo bobabili abasebenzisi abancane nabakhulu, okusho futhi izindleko eziphansi.
I-Thanos Architecture
Ngemva kokufaka imigomo yethu esigabeni sangaphambilini, asisebenze ngayo futhi sibone ukuthi i-Thanos izixazulula kanjani lezi zinkinga.
Ukubuka komhlaba wonke
Ukuze uthole ukubuka komhlaba wonke ngaphezulu kwezimo ezikhona ze-Prometheus, sidinga ukuxhuma indawo yokufaka isicelo esisodwa kuwo wonke amaseva. Yilokhu kanye okwenziwa ingxenye ye-Thanos. . Isetshenziswa eduze kweseva ngayinye ye-Prometheus futhi isebenza njengommeleli, inikezela ngedatha yasendaweni ye-Prometheus nge-gRPC Store API, okuvumela idatha yochungechunge lwesikhathi ukuthi ibuyiswe ngomaki nebanga lesikhathi.
Ngakolunye uhlangothi i-scale-out, ingxenye ye-Querier engenasisekelo, eyenza okungaphezu nje kokuphendula imibuzo ye-PromQL nge-Prometheus HTTP API evamile. I-Querier, i-Sidecar nezinye izingxenye ze-Thanos zixhumana nge .
- I-Querier, lapho ithola isicelo, ixhumeka kuseva ye-Store API ehambisanayo, okungukuthi, ku-Sidecars yethu futhi ithola idatha yochungechunge lwesikhathi kusuka kumaseva ahambisanayo e-Prometheus.
- Ngemva kwalokho, ihlanganisa izimpendulo bese yenza umbuzo we-PromQL kuzo. I-Querier ingahlanganisa kokubili idatha engahlangani kanye nedatha eyimpinda evela kumaseva e-Prometheus HA.
Lokhu kuxazulula ingxenye enkulu yendida yethu - ukuhlanganisa idatha evela kumaseva e-Prometheus ahlukanisiwe ibe umbono owodwa. Eqinisweni, i-Thanos ingasetshenziselwa lesi sici kuphela. Azikho izinguquko okudingeka zenziwe kumaseva akhona e-Prometheus!
Impilo yeshelufu engenamkhawulo!
Nokho, ngokushesha noma kamuva sizofuna ukugcina idatha ngaphezu kwesikhathi esivamile sokugcinwa kwe-Prometheus. Sikhethe isitoreji sento ukuze sigcine idatha yomlando. Itholakala kabanzi kunoma yiliphi ifu kanye nasezikhungweni zedatha yasendaweni futhi ibiza kakhulu. Ngaphezu kwalokho, cishe noma yisiphi isitoreji sezinto sitholakala nge-S3 API eyaziwayo.
U-Prometheus ubhala idatha esuka ku-RAM iye kudiski cishe njalo emahoreni amabili. Ibhulokhi yedatha egciniwe iqukethe yonke idatha yesikhathi esinqunyiwe futhi ayinakuguqulwa. Lokhu kulula kakhulu ngoba i-Thanos Sidecar ingavele ibheke umkhombandlela wedatha we-Prometheus futhi, njengoba amabhlogo amasha etholakala, awalayishe kumabhakede okugcina izinto.
Ukulayisha endaweni yokugcina into ngokushesha ngemva kokubhala kudiski nakho kukuvumela ukuthi ugcine ubulula be-scraper (i-Prometheus ne-Thanos Sidecar). Okwenza kube lula ukwesekwa, izindleko kanye nokwakheka kwesistimu.
Njengoba ubona, isipele idatha ilula kakhulu. Kodwa kuthiwani ngokubuza idatha endaweni yokugcina into?
Ingxenye ye-Thanos Store isebenza njengommeleli wokubuyisa idatha ekugcinweni kwento. Njengo-Thanos Sidecar, ibamba iqhaza kuqoqo lenhlebo futhi isebenzisa i-Store API. Ngale ndlela, i-Querier ekhona ingayithatha njenge-Sidecar, njengomunye umthombo wedatha yochungechunge lwesikhathi - akukho ukucushwa okukhethekile okudingekayo.
Amabhulokhi edatha yochungechunge lwesikhathi aqukethe amafayela amakhulu ambalwa. Ukuzilayisha lapho kudingeka khona ngeke kungasebenzi kahle, futhi ukuzifaka kunqolobane endaweni kuzodinga inani elikhulu lememori nesikhala sediski.
Esikhundleni salokho, i-Store Gateway iyakwazi ukuphatha ifomethi yesitoreji se-Prometheus. Ngenxa yohlelo oluhlakaniphile lombuzo kanye nokufaka kunqolobane izingxenye zenkomba ezidingekayo zamabhulokhi kuphela, kungenzeka ukunciphisa imibuzo eyinkimbinkimbi ibe inani elincane lezicelo ze-HTTP zokuphikisa amafayela okugcina. Ngale ndlela, ungakwazi ukunciphisa inani lezicelo ngama-oda amane kuya kwayisithupha obukhulu futhi uzuze izikhathi zokuphendula ngokuvamile okunzima ukuzihlukanisa kusukela kuzicelo kuye kudatha ye-SSD yendawo.
Njengoba kuboniswe kumdwebo ongenhla, i-Thanos Querier yehlisa kakhulu izindleko ngombuzo ngamunye wedatha yesitoreji sento ngokusebenzisa ifomethi yesitoreji se-Prometheus nokubeka idatha ehlobene eduze. Ngokusebenzisa le ndlela, singahlanganisa izicelo eziningi ezizodwa zibe inani elincane lemisebenzi eyinqwaba.
Ukulinganisa kanye nokwehlisa isampula
Uma ibhulokhi entsha yochungechunge lwedatha ilayishwe ngempumelelo kusitoreji sento, siyiphatha njengedatha "engokomlando", etholakala ngokushesha nge-Store Gateway.
Nokho, ngemva kwesikhathi esithile, amabhulokhi asuka kumthombo owodwa (i-Prometheus ene-Sidecar) ayanqwabelana futhi awabe esasebenzisa amandla okukhomba agcwele. Ukuxazulula le nkinga, sethule enye ingxenye ebizwa ngeCompactor. Imane isebenzise injini yokuhlanganisa yendawo ka-Prometheus kudatha yomlando ekugcinweni kwento futhi ingasebenza njengomsebenzi we-batch wezikhathi ezithile olula.
Ngenxa yokucindezela okusebenzayo, ukubuza isitoreji isikhathi eside akubangeli inkinga ngosayizi wedatha. Kodwa-ke, izindleko ezingaba khona zokupakisha amanani ebhiliyoni nokuwasebenzisa ngokusebenzisa iphrosesa yemibuzo nakanjani kuzoholela ekwenyukeni okumangalisayo kwesikhathi sokwenza imibuzo. Ngakolunye uhlangothi, njengoba kunamakhulu amaphoyinti edatha ngephikseli ngayinye esikrinini, kuba nzima ngisho ukubona ngeso lengqondo idatha ngokulungiswa okugcwele. Ngakho-ke, ukwehlisa isampula akunakwenzeka kuphela, kodwa futhi ngeke kuholele ekulahlekelweni okuphawulekayo kokunemba.
Ukuze wehlise isampula yedatha, iCompactor ihlanganisa ngokuqhubekayo idatha ngokulungiswa kwemizuzu emihlanu nehora elilodwa. Ngesiqephu ngasinye esingahluziwe esifakwe ikhodi kusetshenziswa ukuminyanisa kwe-TSDB XOR, kugcinwa izinhlobo ezahlukene zedatha ehlanganisiwe, njengeminithi, ubuningi noma isamba sebhulokhi eyodwa. Lokhu kuvumela i-Querier ukuthi ikhethe ngokuzenzakalelayo isamba esifanele umbuzo onikeziwe we-PromQL.
Akukho ukulungiselelwa okukhethekile okudingekayo ukuze umsebenzisi asebenzise idatha enemba encishisiwe. I-Querer ishintsha ngokuzenzakalelayo phakathi kwesinqumo esihlukile nedatha eluhlaza njengoba umsebenzisi esondeza ngaphandle nangaphakathi. Uma efisa, umsebenzisi angakwazi ukulawula lokhu ngokuqondile ngepharamitha "yesinyathelo" esicelweni.
Njengoba izindleko zokugcina i-GB eyodwa ziphansi, ngokuzenzakalela i-Thanos igcina idatha eluhlaza, idatha yemizuzu emihlanu kanye nehora elilodwa. Asikho isidingo sokususa idatha yoqobo.
Imithetho yokuqopha
Ngisho ne-Thanos, imithetho yokurekhoda iyingxenye ebalulekile yesitaki sokuqapha. Banciphisa ubunkimbinkimbi, ukubambezeleka, kanye nezindleko zemibuzo. Futhi zilungele abasebenzisi ukuthi bathole idatha ehlanganisiwe ngamamethrikhi. I-Thanos isuselwe kuzimo ze-vanilla Prometheus, ngakho-ke kwamukeleka ngokuphelele ukugcina imithetho yokurekhoda nemithetho yokuxwayisa kuseva ekhona ye-Prometheus. Nokho, kwezinye izimo lokhu kungase kungenele:
- Isexwayiso somhlaba wonke kanye nomthetho (isibonelo, isexwayiso uma isevisi ingasebenzi kumaqoqo angaphezu kwamabili kwamathathu).
- Umthetho wedatha ngaphandle kwesitoreji sasendaweni.
- Isifiso sokugcina yonke imithetho nezixwayiso endaweni eyodwa.
Kuzo zonke lezi zimo, i-Thanos ihlanganisa ingxenye ehlukile ebizwa ngokuthi i-Ruler, ehlanganisa umthetho nesexwayiso nge-Thanos Queries. Ngokuhlinzeka nge-StoreAPI eyaziwa kakhulu, inodi Yombuzo ingafinyelela kumamethrikhi ekhompuyutha asanda kwenziwa. Kamuva zibuye zigcinwe endaweni yokugcina izinto futhi zenziwe zitholakale nge-Store Gateway.
Amandla e-Thanos
I-Thanos iguquguquka ngokwanele ukuthi yenziwe ngendlela oyifisayo ukuze ihambisane nezidingo zakho. Lokhu kuwusizo ikakhulukazi uma ufuduka usuka ethafeni le-Prometheus. Ake sibuyekeze ngokushesha esikufundile mayelana nezingxenye ze-Thanos ngesibonelo esisheshayo. Nansi indlela yokuyisa i-vanilla Prometheus yakho emhlabeni "wesitoreji samamethrikhi angenamkhawulo":
- Engeza i-Thanos Sidecar kumaseva akho e-Prometheus - isibonelo, isiqukathi se-sidecar ku-pod ye-Kubernetes.
- Sebenzisa izifaniso eziningi ze-Thanos Querier ukuze ukwazi ukubuka idatha. Kulesi sigaba kulula ukusetha inhlebo phakathi kwe-Scraper ne-Querier. Ukuze uhlole ukusebenzisana kwezingxenye, sebenzisa i-'thanos_cluster_members' metric.
Lezi zinyathelo ezimbili nje zanele ukuhlinzeka ngokubuka komhlaba wonke kanye nokudonswa kwedatha okungenamthungo kusuka kuma-replicas e-Prometheus HA angaba khona! Vele uxhume kalula amadeshibhodi akho endaweni yokugcina ye-Querier HTTP noma usebenzise i-Thanos UI ngokuqondile.
Nokho, uma udinga ikhophi yasenqolobaneni yamamethrikhi kanye nesitoreji sesikhathi eside, uzodinga ukuqedela izinyathelo ezintathu ezengeziwe:
- Dala ibhakede le-AWS S3 noma le-GCS. Lungiselela i-Sidecar ukuze ukopishele idatha kulawa mabhakede. Isitoreji sedatha yendawo manje singancishiswa.
- Sebenzisa i-Store Gateway futhi uyixhume kuqoqo lakho lenhlebo elikhona. Manje ungabuza ngedatha eyenziwe isipele!
- Sebenzisa iCompactor ukuze uthuthukise ukusebenza kahle kwemibuzo esikhathini eside usebenzisa ukuminyanisa nokusampula phansi.
Uma ufuna ukwazi okwengeziwe, ungangabazi ukubheka yethu ΠΈ !
Ngezinyathelo ezinhlanu nje, siguqule i-Prometheus yaba isistimu yokuqapha ethembekile enokubuka komhlaba wonke, isikhathi sesitoreji esingenamkhawulo kanye nokuba khona okuphezulu okungaba khona kwamamethrikhi.
Donsela isicelo: siyakudinga!
bekuyiphrojekthi yomthombo ovulekile kusukela ekuqaleni. Ukuhlanganiswa okungenamthungo ne-Prometheus kanye nekhono lokusebenzisa ingxenye nje ye-Thanos kuyenza ibe yisinqumo esihle kakhulu sokukala uhlelo lwakho lokuqapha kalula.
Samukela njalo Izicelo Zokudonsa I-GitHub Nezinkinga. Okwamanje, zizwe ukhululekile ukuxhumana nathi nge-Github Issues noma i-sleckuma unemibuzo noma impendulo, noma ufuna ukwabelana ngolwazi lwakho usebenzisa it! Uma uthanda esikwenzayo kwa-Improbable, ungangabazi ukusithinta - !
Source: www.habr.com