Kuturikirwa kwechinyorwa kwakagadzirirwa zvakananga vadzidzi vekosi .
inogadzira software, Go fanatic, uye dambudziko rekugadzirisa. Iye zvakare muchengeti wePrometheus uye co-muvambi weKubernetes SIG chiridzwa. Munguva yakapfuura, aive injiniya wekugadzira paSoundCloud uye akatungamira timu yekutarisa kuCoreOS. Parizvino anoshanda paGoogle.
- Infrastructure Injiniya paIsingagoneki. Anofarira matekinoroji matsva uye matambudziko ehurongwa hwakaparadzirwa. Ane yakaderera-level programming ruzivo kuIntel, mupiro ruzivo kuMesos, uye yepasi-kirasi SRE ruzivo rwekugadzira paIngagoneke. Yakatsaurirwa kuvandudza nyika ye microservices. Vatatu vake vanoda: Golang, yakavhurika sosi uye volleyball.
Uchitarisa pachigadzirwa chedu chemureza SpatialOS, unogona kufungidzira kuti Isingagoneki inoda yakasimba, yepasirese-mwero wegore zvivakwa zvine akawanda emasumbu eKubernetes. Isu taiva vekutanga kushandisa monitoring system . Prometheus inokwanisa kuronda mamirioni emametrics munguva chaiyo uye inouya neine simba remubvunzo mutauro unobvumidza iwe kuburitsa ruzivo rwaunoda.
Kureruka uye kuvimbika kwePrometheus ndeimwe yemabhenefiti ayo makuru. Zvisinei, kana tapfuura imwe chiyero, takasangana nezvipingamupinyi zvakawanda. Kugadzirisa matambudziko aya takagadzira ndeye yakavhurika sosi purojekiti yakagadzirwa neIngagoneke kushandura isina musono masumbu ePrometheus kuita imwechete yekutarisa sisitimu ine isina muganho wenhoroondo yekuchengetedza data. Thanos inowanikwa paGithub .
Zvinangwa zvedu naThanos
Pane imwe chiyero, matambudziko anomuka ari kunze kwekugona kwevanilla Prometheus. Nzira yekuchengetedza nekuvimbika uye nehupfumi petabytes yenhoroondo data? Izvi zvinogona kuitwa pasina kukanganisa nguva yekupindura? Zvinogoneka here kuwana ese metrics ari pane akasiyana Prometheus maseva ane imwechete API chikumbiro? Pane imwe nzira yekubatanidza yakadzokororwa data yakaunganidzwa uchishandisa Prometheus HA?
Kugadzirisa nyaya idzi, takagadzira Thanos. Zvikamu zvinotevera zvinotsanangura magadzirisiro atakaita nyaya idzi uye zvinotsanangura zvinangwa zvedu.
Kubvunza data kubva kune akawanda Prometheus zviitiko (mubvunzo wepasi rose)
Prometheus inopa nzira inoshanda kune sharding. Kunyangwe sevha imwe chete yePrometheus inopa scalability yakakwana yekusunungura vashandisi kubva kumatambudziko eiyo horizontal sharding mune dzinenge dzese kesi dzekushandisa.
Kunyange iyi iri hombe yekufambisa modhi, inowanzodikanwa kuwana data pane akasiyana Prometheus maseva kuburikidza neapi API kana UI - maonero epasirese. Ehe, zvinokwanisika kuratidza akawanda mibvunzo mune imwe Grafana pani, asi mubvunzo wega wega unogona kungoitwa pane imwe Prometheus server. Kune rimwe divi, neThanos unogona kubvunza uye kuunganidza data kubva kune akawanda Prometheus maseva sezvo ese anowanikwa kubva kune imwechete yekupedzisira.
Pakutanga, kuti tiwane maonero epasi rose muIngagoneke, takaronga zviitiko zvedu zvePrometheus kuita nhanho dzakawanda. . Izvi zvaireva kugadzira imwe Prometheus meta server iyo inounganidza mamwe metrics kubva kune yega mashizha server.
Iyi nzira yakaratidza dambudziko. Izvi zvakakonzera mamwe magadzirirwo akaomarara, kuwedzerwa kwechimwe chinhu chinogona kutadza, uye kushandiswa kwemitemo yakaoma kuona kuti iyo federated endpoint inogamuchira chete data yainoda. Uye zvakare, rudzi urwu rwemubatanidzwa harukubvumidze iwe kuti uwane yechokwadi yepasirese maonero, sezvo isiri yese data inowanikwa kubva kune imwechete API chikumbiro.
Zvine hukama zvakanyanya neizvi maonero akabatana eiyo data yakaunganidzwa pane yakakwirira-kuwanikwa (HA) Prometheus maseva. Prometheus 'HA modhi yakazvimiririra inounganidza data kaviri, iri nyore zvekuti haigone kuve nyore. Nekudaro, kushandisa maonerwo akasanganiswa uye akadhindwa enzizi ese ari maviri zvingave zviri nyore.
Ehe, pane kudiwa kweakanyanya kuwanikwa Prometheus maseva. At Improbable, isu tinotora miniti-ne-miniti yekutarisisa data zvakakomba, asi kuve nePrometheus imwe muenzaniso pane cluster ipfungwa imwe yekutadza. Chero kukanganisa kwekugadzirisa kana kutadza kwehardware kunogona kutungamirira mukurasikirwa kwemashoko anokosha. Kunyangwe kuendesa kuri nyore kunogona kukonzera kukanganisa kudiki mukuunganidza metrics nekuti restarts inogona kureba zvakanyanya kupfuura nguva yekukwenya.
Kuchengetedzwa kwakavimbika kwenhoroondo data
Yakachipa, inokurumidza, yenguva refu metrics kuchengetedza iroto redu (rakagovaniswa nevazhinji vashandisi vePrometheus). MuIngagoneke, isu takamanikidzwa kugadzirisa metrics yekuchengeta nguva kusvika mazuva mapfumbamwe (yePrometheus 1.8). Izvi zvinowedzera miganhu iri pachena kuti kure kwatinogona kutarisa kumashure.
Prometheus 2.0 yakagadzirika mune izvi, sezvo huwandu hwenguva dzakateedzana dzisingachakanganisa kuita kwese kwesevha (ona. ) Nekudaro, Prometheus inochengeta data pane yemuno dhisiki. Kunyangwe yakakwirira-inoshanda kudzvanya data kunogona kuderedza zvakanyanya kushandiswa kweSSD yemuno, pakupedzisira kuchine muganho kune huwandu hwenhoroondo data inogona kuchengetwa.
Pamusoro pezvo, paIsingagoneki isu tine hanya nekuvimbika, kuve nyore uye mutengo. Madhisiki makuru emunharaunda anonyanya kuoma kushanda nekuchengetedza. Vanodhura zvakanyanya uye vanoda mamwe maturusi ekuchengetedza, zvichikonzera kuoma kusingakoshi.
Downsampling
Patangotanga kushanda nenhoroondo yedata, takaona kuti kune matambudziko akakosha ne-hombe-O anoita kuti mibvunzo iite zvishoma uye zvishoma sezvatinoshanda nemavhiki, mwedzi, uye makore edata.
Iyo yakajairwa mhinduro yedambudziko iri ingave (downsampling) - kuderedza chiratidzo chekuenzanisira frequency. Nekudzikisa sampling, tinogona "kudzika" kune yakakura nguva renji uye kuchengetedza nhamba yakafanana yemasampuli, tichichengeta mibvunzo ichipindura.
Kudzikisa data rekare chinhu chisingadzivisiki chinodiwa chero chenguva refu yekuchengetedza mhinduro uye iri pamusoro peiyo vanilla Prometheus.
Zvimwe zvinangwa
Chimwe chezvinangwa zvepakutanga zveThanos purojekiti yaive yekubatanidza zvisina musono nechero iripo Prometheus kumisikidzwa. Chinangwa chechipiri chaiva nyore kushanda nezvipingamupinyi zvishoma kupinda. Chero kutsamira kunofanirwa kugutsikana zviri nyore kune vese vashandisi vadiki nevakuru, izvo zvinorevawo mutengo wakaderera.
Thanos architecture
Mushure mekunyora zvibodzwa zvedu muchikamu chakapfuura, ngatishandei mazviri toona kuti Thanos anogadzirisa sei matambudziko aya.
Global view
Kuti uwane maonero epasi rose pamusoro peiyo iripo Prometheus zviitiko, isu tinofanirwa kubatanidza imwe yekukumbira yekupinda nzvimbo kune ese maseva. Izvi ndizvo chaizvo zvinoitwa nechikamu cheThanos. . Inoiswa padivi pesevha yega yega yePrometheus uye inoita senge proxy, ichishandira yemuno Prometheus data kuburikidza negRPC Store API, ichibvumira nguva yakatevedzana data kuti itorerwe ne tag uye nguva renji.
Kune rimwe divi kune chiyero-kunze, chisingatauri Querier chikamu, chinoita zvishoma kupfuura kungopindura mibvunzo yePromQL kuburikidza neyakajairwa Prometheus HTTP API. Querier, Sidecar uye zvimwe zvikamu zveThanos zvinotaurirana kuburikidza .
- Querier, pakugamuchira chikumbiro, inobatanidza kune inoenderana Chitoro API server, kureva, kune yedu Sidecars uye inogamuchira yenguva yakatevedzana data kubva kune inoenderana Prometheus maseva.
- Mushure meizvozvo, inosanganisa mhinduro uye inoita mubvunzo wePromQL pavari. Querier inogona kusanganisa zvese disjoint data uye duplicate data kubva kuPrometheus HA maseva.
Izvi zvinogadzirisa chidimbu chikuru chepuzzle yedu - kusanganisa data kubva kune yakasarudzika Prometheus maseva kuita imwe chete maonero. Muchokwadi, Thanos inogona kushandiswa chete kune ichi chimiro. Hapana shanduko inoda kuitwa kune iripo Prometheus maseva!
Hupenyu hwesherufu husingagumi!
Nekudaro, nekukurumidza kana gare gare isu tichada kuchengetedza data kupfuura yakajairwa Prometheus nguva yekuchengetedza. Isu takasarudza kuchengetedza chinhu kuchengetedza nhoroondo data. Inowanikwa zvakanyanya mune chero gore pamwe ne-on-premises data centers uye inodhura zvakanyanya. Mukuwedzera, chero chinhu chekuchengetedza chinowanikwa kuburikidza neanozivikanwa S3 API.
Prometheus anonyora data kubva ku RAM kuenda ku diski anenge maawa maviri ega ega. Iyo yakachengetwa data block ine data rese kwenguva yakatarwa uye haichinjiki. Izvi zviri nyore nekuti Thanos Sidecar inogona kungotarisa iyo Prometheus data dhairekitori uye, sezvo mabhuraki matsva anowanikwa, anoaisa mumabhakiti ekuchengetera zvinhu.
Kurodha mukuchengetedza chinhu pakarepo mushure mekunyora kune diski zvakare inobvumidza iwe kuchengetedza kupusa kweiyo scraper (Prometheus uye Thanos Sidecar). Iyo inorerutsa rutsigiro, mutengo uye system dhizaini.
Sezvauri kuona, data backup iri nyore kwazvo. Asi zvakadini nekubvunza data mukuchengetedza chinhu?
Iyo Thanos Store chikamu chinoshanda semumiriri kutora data kubva mukuchengetedza chinhu. SaThanos Sidecar, inotora chikamu muboka remakuhwa uye inoshandisa Store API. Nenzira iyi, Querier iripo inogona kuibata seSidecar, seimwe sosi yenguva yakatevedzana data - hapana yakakosha kumisikidzwa inodiwa.
Nguva yakatevedzana data blocks ine akati wandei mafaera makuru. Kuaisa pazvinenge zvichidiwa kungave kusingashande, uye kuvachengeta munharaunda kunoda huwandu hukuru hwendangariro uye disk nzvimbo.
Pane kudaro, Store Gateway inoziva kubata iyo Prometheus yekuchengetedza fomati. Kutenda kune yakangwara yemubvunzo kuronga uye caching chete inodiwa indekisi zvikamu zvemabhuroko, zvinokwanisika kudzikisa yakaoma mibvunzo kune hushoma nhamba yeHTTP zvikumbiro zvekuramba mafaera ekuchengetedza. Nenzira iyi, unogona kudzikisa huwandu hwezvikumbiro nemaodha mana kusvika matanhatu ehukuru uye kuwana nguva dzekupindura dzinowanzo kunetsa kusiyanisa kubva kune zvikumbiro kune data pane yemuno SSD.
Sezvinoratidzwa mudhayagiramu iri pamusoro, Thanos Querier inoderedza zvakanyanya mutengo pamubvunzo wechinhu chekuchengetedza data nekushandisa iyo Prometheus yekuchengetedza fomati uye nekuisa inoenderana data parutivi. Tichishandisa nzira iyi, tinogona kusanganisa zvikumbiro zvakawanda muhuwandu hwehuwandu hwekushanda.
Compaction uye downsampling
Kamwe bhuroka nyowani yenguva yakatevedzana data yakanyatso kurodha mukuchengetedza chinhu, tinoitora se "nhoroondo" data, iyo inowanikwa nekukasika kuburikidza neChitoro Gateway.
Nekudaro, mushure menguva yakati, zvidhinha kubva kune imwe sosi (Prometheus ine Sidecar) inounganidza uye haichashandisi izere indexing kugona. Kugadzirisa dambudziko iri, takaunza chimwe chikamu chinonzi Compactor. Inongoshandisa Prometheus 'yemunharaunda compaction injini kune nhoroondo data mukuchengetedza chinhu uye inogona kumhanya seyakapusa periodic batch basa.
Nekuda kwekumanikidza kunoshanda, kubvunza chengetedzo kwenguva yakareba haipe dambudziko maererano nesaizi yedata. Nekudaro, mutengo ungangove wekuburitsa bhiriyoni kukosha uye nekuamhanyisa kuburikidza nemubvunzo processor zvinozoguma nekuwedzera kunoshamisa kwenguva yekubvunza mibvunzo. Kune rimwe divi, sezvo paine mazana emapoinzi edata papixel pachiratidziri, zvinove zvisingaite kuti utarise iyo data pakugadziriswa kuzere. Nokudaro, kudzikisa pasi hakugoneki chete, asiwo hakuzotungamiri pakurasikirwa kunooneka kwekururama.
Kuti udzikise data, Compactor inoramba ichiunganidza data pakugadziriswa kwemaminetsi mashanu neawa imwe. Kune yega yega chunk yakavharidzirwa uchishandisa TSDB XOR compression, marudzi akasiyana e data akaunganidzwa anochengetwa, senge min, max kana sum yebhuroka rimwe. Izvi zvinobvumira Querier kuti azvisarudzire otomatiki muunganidzwa wakakodzera kune yakapihwa PromQL mubvunzo.
Hapana kurongeka kwakakosha kunodiwa kuti mushandisi ashandise yakaderedzwa chaiyo data. Querier inoshandura otomatiki pakati pezvisarudzo zvakasiyana uye data mbishi sezvo mushandisi anokwevera mukati nekubuda. Kana zvichidikanwa, mushandisi anogona kudzora izvi zvakananga kuburikidza ne "danho" parameter muchikumbiro.
Sezvo mutengo wekuchengeta imwe GB wakadzikira, nekusarudzika Thanos inochengeta data mbishi, maminetsi mashanu neawa imwe yekugadzirisa data. Hapana chikonzero chekudzima data rekutanga.
Kurekodha mitemo
Kunyangwe neThanos, mitemo yekurekodha chikamu chakakosha chekutarisa stack. Ivo vanoderedza kuoma, latency, uye mutengo wemibvunzo. Izvo zvakare zviri nyore kune vashandisi kuwana yakaunganidzwa data nema metrics. Thanos yakavakirwa pane vanilla Prometheus zviitiko, saka zvinonyatso gamuchirwa kuchengeta mitemo yekurekodha uye yekuzivisa mitemo pane iripo Prometheus server. Zvisinei, mune dzimwe nguva izvi zvingasava zvakakwana:
- Chenjedzo yepasi rose uye mutemo (semuenzaniso, yambiro kana sevhisi isingashande pane anopfuura maviri ematatu masumbu).
- Mutemo wedata kunze kwekuchengetwa kwenzvimbo.
- Chishuwo chekuchengeta mitemo yese uye yambiro munzvimbo imwechete.
Pazviitiko zvese izvi, Thanos inosanganisira chikamu chakasiyana chinodaidzwa kuti Mutongi, icho chinoverengera mutemo uye yambiro kuburikidza neThanos Queries. Nekupa inozivikanwa StoreAPI, iyo Query node inogona kuwana ichangobva kuverengerwa metrics. Gare gare ivo zvakare vanochengetwa mukuchengetedza chinhu uye kuitwa kuwanikwa kuburikidza neStore Gateway.
Simba reThanos
Thanos inochinjika zvakakwana kuti igadziriswe kuti ienderane nezvido zvako. Izvi zvinonyanya kubatsira kana uchitama kubva kubani Prometheus. Ngatikurumidze kudzokorodza zvatakadzidza nezve Thanos zvikamu nemuenzaniso unokurumidza. Heano maitiro ekutora yako vanilla Prometheus munyika ye "isina muganho metrics chengetedzo":
- Wedzera Thanos Sidecar kumaseva ako ePrometheus - semuenzaniso, mudziyo wepadivi muKubernetes pod.
- Shandisa akawanda Thanos Querier replicas kuti ukwanise kuona data. Panguva ino zviri nyore kugadzira makuhwa pakati peScraper neQuerier. Kuti utarise kudyidzana kwezvikamu, shandisa iyo 'thanos_cluster_members' metric.
Aya matanho maviri chete akakwana kuti ape maonero epasirese uye isina musono data kudonhedza kubva kune inogona Prometheus HA replicas! Ingobatanidza madhibhodhi ako kune Querier HTTP endpoint kana kushandisa Thanos UI zvakananga.
Nekudaro, kana iwe uchida metrics backup uye kwenguva refu kuchengetedza, iwe unozofanirwa kupedzisa mamwe matanho matatu:
- Gadzira AWS S3 kana GCS bhaketi. Gadzirisa Sidecar kukopa data kune aya mabhakiti. Chengetedzo yedata yemunharaunda ikozvino inogona kudzikiswa.
- Deploy Store Gateway woibatanidza kune yako iripo makuhwa cluster. Iye zvino unogona kubvunza iyo yakatsigirwa data!
- Deploy Compactor kuti uvandudze kushanda zvakanaka kwemubvunzo kwenguva yakareba uchishandisa compaction uye downsampling.
Kana iwe uchida kuziva zvakawanda, usazeza kutarisa kune yedu ΠΈ !
Mumatanho mashanu chete, takashandura Prometheus kuita yakavimbika yekutarisisa sisitimu ine maonero epasirese, isina muganho yekuchengetedza nguva uye mukana wepamusoro kuwanikwa kwema metrics.
Dhonza chikumbiro: tinokuda iwe!
rave rakavhurika sosi purojekiti kubva pakutanga. Kubatanidzwa kusina musono nePrometheus uye kugona kushandisa chete chikamu cheThanos kunoita kuti ive sarudzo yakanaka yekuyera yako yekutarisa sisitimu zvisina basa.
Isu tinogara tichigamuchira GitHub Dhonza Zvikumbiro uye Matambudziko. Zvichakadaro, inzwa wakasununguka kutibata nesu kuburikidza neGithub Matambudziko kana kunonokakana uine mibvunzo kana mhinduro, kana uchida kugovera ruzivo rwako uchiishandisa! Kana iwe uchida zvatinoita paImprobable, usazeza kutibata - !
Source: www.habr.com