Cherechedza. transl.: Isu tinokupa kutarisiro yako ruzivo rwehunyanzvi nezve zvikonzero zvenguva pfupi yapfuura yekudzikira musevhisi yegore inochengetwa nevagadziri veGrafana. Uyu muenzaniso wekare wekuti chinhu chitsva uye chinoita sechinobatsira zvakanyanya chakagadzirirwa kuvandudza kunaka kwezvivakwa ... chinogona kukonzera kukuvadza kana ukasapa akawanda nuances ekushandiswa kwayo mune chokwadi chekugadzirwa. Zvakanaka kana zvinhu zvakaita seizvi zvichionekwa izvo zvinokutendera kuti udzidze kwete kubva mukukanganisa kwako chete. Tsanangudzo iri mushanduro yechinyorwa ichi kubva kumutevedzeri wemutungamiri wechigadzirwa kubva kuGrafana Labs.
NeChishanu, Chikunguru 19, iyo Hosting Prometheus sevhisi muGrafana Cloud yakamira kushanda kweanenge maminetsi makumi matatu. Ndinokumbira ruregerero kune vese vatengi vakakanganiswa nekudzima. Basa redu nderekupa maturusi ekutarisa aunoda, uye tinonzwisisa kuti kusave nawo kunogona kuita kuti hupenyu hwako hunyanye kuoma. Tinotora chiitiko ichi zvakanyanya. Ichi chinyorwa chinotsanangura zvakaitika, mapinduriro atakaita, uye zvatiri kuita kuona kuti hazviitike zvakare.
prehistory
Grafana Cloud Yakagadzirirwa Prometheus sevhisi yakavakirwa pa -CNCF purojekiti yekugadzira yakatwasuka, inowanikwa zvakanyanya, yakawanda-inoroja Prometheus sevhisi. Iyo Cortex architecture ine seti yemamicroservices ega ega, imwe neimwe inoita basa rayo: kudzokorora, kuchengetedza, mibvunzo, nezvimwe. Cortex iri pasi pekusimudzira uye inogara ichiwedzera maficha matsva nekuvandudza mashandiro. Isu tinogara tichitumira nyowani Cortex kuburitswa kumasumbu kuitira kuti vatengi vatore mukana weaya maficha - nerombo rakanaka, Cortex inogona kuvandudzwa pasina kudzika.
Kune zvigadziriso zvisina musono, iyo Ingester Cortex sevhisi inoda imwezve Ingester replica panguva yekuvandudza maitiro. (Cherechedza. transl.: - iyo yakakosha chikamu cheCortex. Basa rayo nderekuunganidza inogara ichiyerera yemasamples, kuaisa muPrometheus chunks uye kuichengeta mudhatabhesi seDynamoDB, BigTable kana Cassandra.) Izvi zvinobvumira yekare Ingesters kutumira data razvino kune Ingesters nyowani. Izvo zvakakosha kucherechedza kuti Ingesters iri kuda zviwanikwa. Kuti vashande, unofanirwa kuva ne 4 cores uye 15 GB yekuyeuka pane pod, i.e. 25% yesimba rekugadzirisa uye ndangariro yemuchina wepasi mune yedu Kubernetes masumbu. Kazhinji, isu tinowanzo kuve nezvakawanda zvisina kushandiswa zviwanikwa musumbu pane 4 cores uye 15 GB yendangariro, saka isu tinokwanisa nyore kupepeta aya ekuwedzera ekuwedzera panguva yekusimudzira.
Nekudaro, zvinowanzoitika kuti panguva yekushanda kwakajairika hapana kana imwe yemashini ine iyi 25% yezvinhu zvisina kushandiswa. Ehe, isu hatitomboedze: CPU uye ndangariro zvinogara zvichibatsira kune mamwe maitiro. Kuti tigadzirise dambudziko iri, takasarudza kushandisa . Iyo pfungwa ndeyekupa Ingesters chinzvimbo chepamusoro kupfuura mamwe (asina state) microservices. Kana isu tichida kumhanya yekuwedzera (N+1) Ingester, isu tinobvisa kwechinguva mamwe, mapodhi madiki. Aya mapodhi anotamirwa kune zviwanikwa zvemahara pane mamwe machina, achisiya yakakura yakakwana "gomba" kuti iite imwe Ingester.
NeChina, Chikunguru 18, takaburitsa matanho mana matsva ekutanga kumasumbu edu: kutsoropodza, high, pakati ΠΈ pasi. Ivo vakaedzwa pane yemukati cluster isina mutengi traffic kweinenge svondo rimwe. Nekumisikidza, mapodhi asina kutaurwa zvakakosha anogamuchirwa pakati pekutanga, kirasi yakagadzirirwa Ingesters ne yakakwira zvakakosha. Critical yakachengeterwa kuongororwa (Prometheus, Alertmanager, node-exporter, kube-state-metrics, nezvimwewo). Yedu config yakavhurika, uye iwe unogona kuona iyo PR .
Tsaona
NeChishanu, Chikunguru 19, mumwe weinjiniya akatanga itsva yakatsaurirwa Cortex cluster yemutengi mukuru. Iyo config yeiyi cluster haina kusanganisira zvitsva zvepod, saka mapodhi ese matsva akaiswa pamberi pekutanga - pakati.
Iyo Kubernetes cluster yanga isina zviwanikwa zvakakwana zveiyo Cortex cluster, uye iyo yaivepo yekugadzira Cortex cluster haina kuvandudzwa (Ingesters yakasara isina. high zvakakosha). Sezvo maIngesters echikwata chitsva nekusarudzika aive pakati zvakakosha, uye mapodhi aivepo mukugadzira akashanda pasina kukoshesa zvachose, iyo Ingesters yesumbu nyowani yakatsiva iyo Ingesters kubva kune iripo Cortex yekugadzira cluster.
ReplicaSet yeIngester yakadzingwa muchikwata chekugadzira yakaona iyo pod yakadzingwa uye yakagadzira imwe nyowani kuchengetedza nhamba yakatarwa yemakopi. Iyo pod itsva yakapihwa nekusarudzika pakati zvakakosha, uye imwe "yekare" Ingester mukugadzirwa yakarasikirwa nezviwanikwa zvayo. Mugumisiro wacho waiva Avalanche process, izvo zvakakonzera kutamiswa kwemapodhi ese kubva kuIngester yeCortex masumbu ekugadzira.
Ingesters ndeyekutaura uye chengetedza data kwemaawa gumi nemaviri apfuura. Izvi zvinotitendera kuti tizvitsigire zvine hungwaru tisati tazvinyorera kukuchengetedza kwenguva refu. Kuti uite izvi, Cortex shards data mhiri kwakatevedzana uchishandisa Distributed Hash Tafura (DHT) uye inodzokorora imwe neimwe nhevedzano mumatatu maIngesters uchishandisa Dynamo-maitiro quorum kuenderana. Cortex hainyore data kune Ingesters iyo yakaremara. Saka, kana nhamba huru yeIngesters ichibva muDHT, Cortex haigone kupa kudzokorora kwakakwana kwezvinyorwa, uye vanoputsika.
Kuongorora uye Kugadziriswa
Nyowani Prometheus zviziviso zvakavakirwa pa "kukanganisa bhajeti" (kukanganisa-bhajeti-yakavakirwa - ruzivo ruchaonekwa mune ramangwana chinyorwa) yakatanga kuridza alarm 4 maminetsi mushure mekutanga kudzima. Kwemaminetsi mashanu akatevera kana zvakadaro, takamhanyisa ongororo uye takakwidza iri pasi peKubernetes cluster kuti titore ese matsva uye aripo ekugadzira masumbu.
Mushure memamwe maminetsi mashanu, Ingesters yekare yakabudirira kunyora data ravo, zvitsva zvakatanga, uye mapoka eCortex akazowanikwa zvakare.
Mamwe maminetsi gumi akashandiswa kuongorora nekugadzirisa kunze-kwe-memory (OOM) zvikanganiso kubva kuchokwadi reverse proxies iri pamberi peCortex. Mhosho dzeOOM dzakakonzerwa nekuwedzera kakapetwa kagumi muQPS (tinotenda nekuda kwezvikumbiro zvehasha zvakanyanyisa kubva kumaseva ePrometheus emutengi).
Migumisiro
Nguva yese yekumisa yaive maminitsi makumi maviri nematanhatu. Hapana data yakarasika. Ingesters vakabudirira kurodha ese ari mu-memory data mukuchengetedza kwenguva refu. Panguva yekuvharwa, mutengi Prometheus maseva akavharwa akadzimwa (kure) rekodhi uchishandisa zvichibva paWAL (yakanyorwa na kubva kuGrafana Labs) uye akadzokorora akakundikana anonyora mushure mekuparara.
Production cluster kunyora mashandiro
zvakawanikwa
Zvakakosha kudzidza kubva pachiitiko ichi uye kutora matanho anodiwa kuti urege kudzokazve.
Tichifunga nezve, hatifanire kunge takaisa iyo default pakati pamberi kusvika ese maIngesters mukugadzira awana high chinhu chinokosha. Mukuwedzera, zvakanga zvakafanira kuvatarisira pachine nguva mberi kumusoro zvakakosha. Zvose zvinogadziriswa zvino. Tinovimba kuti ruzivo rwedu ruchabatsira mamwe masangano achifunga nezvekushandisa pod zvinonyanya kukosha muKubernetes.
Isu tichawedzera imwe nhanho yekutonga pamusoro pekutumirwa kwezvimwe zvinhu zvekuwedzera izvo zvigadziriso zviri pasi rose kune cluster. Kubva zvino zvichienda mberi, shanduko dzakadai dzichaongororwa bΠΎvanhu vakawanda. Pamusoro pezvo, kugadziridzwa kwakakonzera tsaona kwainzi kudiki kugwaro reprojekiti rakasiyana - rakangokurukurwa munyaya yeGitHub. Kubva zvino zvichienda mberi, shanduko dzese dzakadai kune configs dzichaperekedzwa nemagwaro akakodzera eprojekiti.
Chekupedzisira, isu tichagadzirisa saizi yeiyo yechokwadi reverse proxy kudzivirira kuwandisa kweOOM kwatakaona, uye tichaongorora Prometheus default marongero ane chekuita nekudonha uye kuyera kudzivirira nyaya dzakafanana mune ramangwana.
Kutadza kwacho kwakavewo nemimwe mhedzisiro yakanaka: mushure mekugamuchira zviwanikwa zvinodiwa, Cortex yakadzoreredza otomatiki pasina kumwe kupindira. Takawanawo ruzivo rwakakosha kushanda narwo - yedu itsva log aggregation system - iyo yakabatsira kuve nechokwadi chekuti maIngesters ese akazvibata nemazvo panguva uye mushure mekutadza.
PS kubva kumushanduri
Verenga zvakare pablog yedu:
- Β«";
- Β«";
- Β«";
- Β«".
Source: www.habr.com