Phawula. transl.: Sinikezela kwingqwalasela yakho iinkcukacha zobugcisa malunga nezizathu zokuthotywa kwamva nje kwinkonzo yelifu egcinwe ngabadali baseGrafana. Lo ngumzekelo weklasi wendlela into entsha kwaye ebonakala iluncedo kakhulu eyilelwe ukuphucula umgangatho weziseko ezingundoqo... inokubangela ingozi ukuba awuboneleli ngeenuances ezininzi zesicelo sayo kwizinto eziyinyani zemveliso. Kuhle xa kuvela izixhobo ezinje zikuvumela ukuba ufunde kungekuphela nje kwiimpazamo zakho. Iinkcukacha zikwinguqulelo yesi sicatshulwa sisuka kusekela mongameli wemveliso evela eGrafana Labs.
NgoLwesihlanu, umhla we-19 kaJulayi, inkonzo ye-Prometheus eHolisiweyo eGrafana Cloud yayeka ukusebenza malunga nemizuzu engama-30. Ndiyaxolisa kubo bonke abathengi abachatshazelwe kukucima. Umsebenzi wethu kukubonelela ngezixhobo zokubeka iliso ozifunayo, kwaye siyaqonda ukuba ukungabi nazo kunokwenza ubomi bakho bube nzima ngakumbi. Esi siganeko sisithatha njengento ebaluleke kakhulu. Eli nqaku licacisa okwenzekileyo, indlela esiphendule ngayo, kunye nento esiyenzayo ukuqinisekisa ukuba ayiphindi yenzeke.
ukubuzwa
I-Grafana Cloud ibanjwe inkonzo ye-Prometheus isekelwe - Iprojekthi ye-CNCF yokudala inkonzo ye-Prometheus ethe tye, efumaneka kakhulu, enabantu abaninzi. Uyilo lweCortex luneseti yee-microservices nganye, nganye eyenza umsebenzi wayo: ukuphindaphinda, ukugcinwa, imibuzo, njl. ICortex iphantsi kophuhliso olusebenzayo kwaye ihlala isongeza izinto ezintsha kunye nokuphucula ukusebenza. Sithumela rhoqo ukukhutshwa kweCortex kumaqela ukuze abathengi bathathe ithuba kwezi mpawu - ngethamsanqa, iCortex inokuhlaziywa ngaphandle kwexesha lokuphumla.
Uhlaziyo olungenamthungo, inkonzo ye-Ingester Cortex ifuna i-replica ye-Ingester eyongezelelweyo ngexesha lohlaziyo. (Phawula. transl.: - icandelo elisisiseko leCortex. Umsebenzi wayo kukuqokelela uthotho lweesampulu ezizingileyo, uzibeke ngokwamaqela kwiichunks zePrometheus kwaye uzigcine kwindawo egciniweyo efana neDynamoDB, iBigTable okanye iCassandra.) Oku kuvumela i-Ingesters endala ukuba ithumele idatha yangoku kwi-Ingesters entsha. Kuyaphawuleka ukuba i-Ingesters ifuna izixhobo. Ukuze basebenze, kufuneka ube ne-4 cores kunye ne-15 GB yememori nge-pod, i.e. I-25% yamandla okusebenza kunye nenkumbulo yomatshini wesiseko kwimeko yamaqela ethu e-Kubernetes. Ngokubanzi, sidla ngokuba nezixhobo ezininzi ezingasetyenziswanga kwiqela kune-4 cores kunye ne-15 GB yememori, ngoko sinokujikeleza ngokulula ezi zingenisi zongezelelweyo ngexesha lokuphucula.
Nangona kunjalo, kuyenzeka ukuba ngexesha lomsebenzi oqhelekileyo akukho namnye koomatshini abanale 25% yezibonelelo ezingasetyenziswanga. Ewe, asizange sizame: i-CPU kunye nememori iya kuhlala iluncedo kwezinye iinkqubo. Ukusombulula le ngxaki, sagqiba ekubeni sisebenzise . Umbono kukunika i-Ingesters indawo ephambili ephezulu kunezinye (ezingenammiselo) ezincinci. Xa sifuna ukuqhuba i-ingester eyongezelelweyo (N+1), sishiya okwethutyana ezinye, iipods ezincinci. Ezi pods zidluliselwa kwizibonelelo zamahhala kwezinye iimatshini, zishiya "umngxuma" omkhulu owaneleyo wokuqhuba i-Ingester eyongezelelweyo.
NgoLwesine, nge-18 kaJulayi, sikhuphe amanqanaba amane amatsha aphambili kumaqela ethu: ebalulekileyo, phezulu, umyinge ΠΈ ephantsi. Baye bavavanywa kwi-cluster yangaphakathi ngaphandle kwetrafikhi yabathengi malunga neveki enye. Ngokungagqibekanga, ii-pods ngaphandle kokubaluleka okuchaziweyo okufunyenweyo umyinge kuqala, iklasi yamiselwa Ingesters nge phezulu okubalulekileyo. Okubalulekileyo ibigcinelwe iliso (i-Prometheus, i-Alertmanager, i-node-exporter, i-kube-state-metrics, njl.). Uqwalaselo lwethu luvuliwe, kwaye unokujonga iPR .
Ukuhlaselwa
NgoLwesihlanu, nge-19 kaJulayi, enye yeenjineli yazisa iqela elitsha elizinikeleyo leCortex kumxhasi omkhulu. Uqwalaselo lweli qela alikhange liquke izinto ezintsha eziphambili ngokubaluleka, ngoko ke zonke iipod ezintsha zabelwa izinto eziphambili ngokubaluleka - umyinge.
Iqela leKubernetes alizange libe nezixhobo ezaneleyo zeqela elitsha leCortex, kwaye imveliso ekhoyo yeqela leCortex ayizange ihlaziywe (Ingesters ishiywe ngaphandle phezulu okubalulekileyo). Ekubeni i-Ingesters yeqela elitsha ngokungagqibekanga ine umyinge okuphambili, kunye neepods ezikhoyo kwimveliso zisebenze ngaphandle kokubeka phambili konke konke, i-Ingesters yeqela elitsha ithathe indawo ye-Ingesters ukusuka kwiqela lemveliso ekhoyo yeCortex.
I-ReplicaSet ye-Ingester ekhutshiweyo kwiqela lemveliso yafumanisa i-pod ekhutshiweyo kwaye yenza entsha ukugcina inani elichaziweyo leekopi. Ipod entsha yabelwa ngokungagqibekanga umyinge okuphambili, kunye nenye i-Ingester "endala" kwimveliso yalahlekelwa yimithombo yayo. Umphumo waba inkqubo ye-avalanche, okwakhokelela ekufudusweni kwazo zonke iipod ezivela kwi-Ingester ukwenzela amaqela emveliso yeCortex.
I-Ingesters isemthethweni kwaye igcina idatha kwiiyure ze-12 zangaphambili. Oku kusivumela ukuba sizicinezele ngokufanelekileyo ngaphambi kokuba sizibhale kwindawo yokugcina ixesha elide. Ukufezekisa oku, i-Cortex shards idatha kulo lonke uthotho usebenzisa i-Distributed Hash Table (DHT) kwaye iphinda-phinda uthotho ngalunye kwii-Ingesters ezintathu usebenzisa i-Dynamo-style quorum consistency. I-Cortex ayibhali idatha kwi-Ingesters ekhubazekileyo. Ngaloo ndlela, xa inani elikhulu le-Ingesters lishiya i-DHT, i-Cortex ayikwazi ukubonelela ngokuphindaphinda okwaneleyo kwamangeno, kwaye iyawa.
Ukufunyanwa kunye noLungiso
Izaziso ezitsha zePrometheus ezisekwe "kuhlahlo lwabiwo-mali lwempazamo" (imposiso-esekwe kuhlahlo lwabiwo-mali - iinkcukacha ziya kuvela kwinqaku elizayo) yaqala ukukhala i-alamu imizuzu emi-4 emva kokuqala kokuvalwa. Kwimizuzu emihlanu ezayo okanye kunjalo, siye saqhuba uxilongo kwaye sanyusa iqela le-Kubernetes elingaphantsi ukuze libambe omabini amacandelo amatsha kunye akhoyo emveliso.
Emva kweminye imizuzu emihlanu, i-Ingesters endala yabhala ngempumelelo idatha yabo, abatsha baqala, kwaye amaqela eCortex aphinde afumaneke kwakhona.
Eminye imizuzu ye-10 ichithwe ekuxilongeni nasekulungiseni iimpazamo ezingaphandle kwememori (OOM) ukusuka kwiiproksi ezibuyela umva ezibekwe phambi kweCortex. Iimpazamo ze-OOM zenziwe kukwanda kwe-QPS ngokuphindwe kalishumi (sikholelwa ukuba kungenxa yezicelo ezindlongondlongo ezisuka kumxhasi weeseva zePrometheus).
Iziphumo
Ixesha lokuphumla lilonke laliyimizuzu engama-26. Akukho datha ilahlekileyo. I-Ingesters ilayishe ngempumelelo yonke idatha yememori kwisitoreji sexesha elide. Ngexesha lokuvalwa, umxhasi we-Prometheus iiseva zivaliwe (ekude) ezirekhodiweyo usebenzisa isekelwe kwi-WAL (ibhalwe ngu ukusuka eGrafana Labs) kwaye uphindaphinda okusilelayo ubhala emva kwengozi.
Imisebenzi yokubhalwa kweqela lemveliso
ezifunyanisiweyo
Kubalulekile ukufunda kwesi siganeko kwaye uthathe amanyathelo ayimfuneko ukuze ugweme ukuphindaphinda kwayo.
Xa sijonga emva, bekungafanelanga ukuba sibeke ukungagqibeki umyinge okuphambili de yonke i-Ingesters kwimveliso ifunyenwe phezulu into ephambili. Ukongezelela, kwakuyimfuneko ukuba babanyamekele kwangaphambili phezulu okubalulekileyo. Yonke into ilungisiwe ngoku. Siyathemba ukuba amava ethu aya kunceda eminye imibutho ethathela ingqalelo ukusebenzisa izinto eziphambili kwi-pod eKubernetes.
Siza kongeza inqanaba elongezelelweyo lolawulo phezu kokuthunyelwa kwazo naziphi na izinto ezongezelelweyo ezilungelelwaniso zazo zihlabathi jikelele kwiqela. Ukususela ngoku ukuya phambili, olo tshintsho luya kuvavanywa bΠΎabantu abaninzi. Ukongeza, ukuguqulwa okubangele ukuphahlazeka kwathathwa njengento encinci kakhulu kuxwebhu lweprojekthi eyahlukileyo - yaxoxwa kuphela kumcimbi weGitHub. Ukusukela ngoku ukuya phambili, lonke olo tshintsho kuqwalaselo luya kukhatshwa ngamaxwebhu afanelekileyo eprojekthi.
Okokugqibela, siya kutshintsha ngokuzenzekela ubungakanani beproksi yokuqinisekisa umva ukunqanda ukugcwala kwe-OOM esikubonileyo, kwaye siya kuphonononga useto olungagqibekanga lwe-Prometheus olunxulumene nokubuyisela umva kunye nokukala ukunqanda imiba efanayo kwixa elizayo.
Ukungaphumeleli kwakhona kwaba nemiphumo emihle: emva kokufumana izixhobo eziyimfuneko, i-Cortex iphinde yafumana kwakhona ngaphandle kokungenelela okongeziweyo. Siye safumana namava axabisekileyo okusebenza nawo - Inkqubo yethu entsha yokudibanisa ilog - eyanceda ekuqinisekiseni ukuba zonke ii-Ingesters ziziphatha kakuhle ngexesha nasemva kokusilela.
PS evela kumguquleli
Funda nakwibhlog yethu:
- Β«";
- Β«";
- Β«";
- «».
umthombo: www.habr.com