Qaphela. transl.: Sikwethulela imininingwane yobuchwepheshe mayelana nezizathu zesikhathi sokuphumula sakamuva kusevisi yefu enakekelwa abadali be-Grafana. Lesi isibonelo sakudala sokuthi isici esisha nesibonakala siwusizo ngokwedlulele esidizayinelwe ukuthuthukisa ikhwalithi yengqalasizinda... singadala umonakalo uma ungahlinzekeli ngamanuance amaningi wokusetshenziswa kwaso ezintweni ezingokoqobo zokukhiqiza. Kuhle uma kuvela izinto ezinjengalezi ezikuvumela ukuthi ufunde hhayi emaphutheni akho kuphela. Imininingwane isekuhunyushweni kwalo mbhalo ovela kusekela mongameli womkhiqizo ovela e-Grafana Labs.
NgoLwesihlanu, Julayi 19, insizakalo ye-Hosted Prometheus e-Grafana Cloud yayeka ukusebenza cishe imizuzu engama-30. Ngiyaxolisa kuwo wonke amakhasimende athintekile ngokucisha. Umsebenzi wethu ukuhlinzeka ngamathuluzi okuqapha owadingayo, futhi siyaqonda ukuthi ukungatholakali kwawo kungenza impilo yakho ibe nzima kakhulu. Lesi sigameko sisithatha njengesibaluleke kakhulu. Leli nothi lichaza okwenzekile, ukuthi siphendule kanjani, nokuthi senzani ukuqinisekisa ukuthi akuphindi kwenzeke.
prehistory
I-Grafana Cloud Hosting Prometheus service isekelwe - Iphrojekthi ye-CNCF yokudala isevisi ye-Prometheus evundlile, etholakala kakhulu, eqasha abantu abaningi. Isakhiwo seCortex siqukethe isethi yama-microservices ngamanye, ngayinye eyenza umsebenzi wayo: ukuphindaphinda, ukugcinwa, imibuzo, njll. I-Cortex ingaphansi kokuthuthukiswa okusebenzayo futhi ihlala yengeza izici ezintsha futhi ithuthukisa ukusebenza. Sivame ukutshala ukukhishwa okusha kwe-Cortex kumaqoqo ukuze amakhasimende akwazi ukusizakala ngalezi zici - ngenhlanhla, i-Cortex ingabuyekezwa ngaphandle kokuphelelwa yisikhathi.
Ukuze uthole izibuyekezo ezingenamthungo, isevisi ye-Ingester Cortex idinga i-replica ye-Ingester eyengeziwe phakathi nenqubo yokubuyekeza. (Qaphela. transl.: - ingxenye eyisisekelo ye-Cortex. Umsebenzi wayo ukuqoqa uchungechunge oluqhubekayo lwamasampula, uwahlanganise abe yizingxenyana ze-Prometheus futhi uwagcine kusizindalwazi esinjenge-DynamoDB, BigTable noma i-Cassandra.) Lokhu kuvumela ama-Ingesters amadala ukuthi adlulisele phambili idatha yamanje kuma-Ingesters amasha. Kubalulekile ukuqaphela ukuthi i-Ingesters idinga izinsiza. Ukuze basebenze, udinga ukuba nama-cores angu-4 kanye ne-15 GB yememori ye-pod ngayinye, i.e. U-25% wamandla okucubungula kanye nenkumbulo yomshini oyisisekelo esimweni samaqoqo ethu e-Kubernetes. Ngokuvamile, sivamise ukuba nezinsiza eziningi ezingasetshenzisiwe kuqoqo kunama-cores angu-4 no-15 GB wememori, ngakho-ke singakwazi ukuphenya kalula lezi zingenisi ezengeziwe phakathi nokuthuthukiswa.
Kodwa-ke, kuvame ukwenzeka ukuthi ngesikhathi sokusebenza okujwayelekile akukho nowodwa umshini onalo 25% wezinsiza ezingasetshenzisiwe. Yebo, asizami nokuzabalaza: I-CPU nenkumbulo kuzohlala kuwusizo kwezinye izinqubo. Ukuxazulula le nkinga, sinqume ukusebenzisa . Umqondo uwukunikeza i-Ingesters into ebaluleke kakhulu kunamanye ama-microservices (angenasimo). Uma sidinga ukusebenzisa isingenisi (N+1) esengeziwe, sisusa okwesikhashana amanye ama-pods amancane. Lawa ma-pods adluliselwa kuzinsiza zamahhala kweminye imishini, eshiya "imbobo" enkulu ngokwanele ukuze kuqhutshwe i-Ingester eyengeziwe.
NgoLwesine, Julayi 18, sikhiphe amazinga amasha amane abalulekile kumaqoqo ethu: gxeka, phezulu, isilinganiso ΠΈ phansi. Ahlolwe kuqoqo langaphakathi elingenaso ithrafikhi yekhasimende cishe isonto elilodwa. Ngokuzenzakalelayo, ama-pod ngaphandle kokubalulekile okucacisiwe kwamukelwe isilinganiso Okubalulekile, ikilasi lalimiselwe i-Ingesters nge phezulu okubalulekile. Okubucayi ibigcinelwe ukuqapha (i-Prometheus, i-Alertmanager, i-node-exporter, i-kube-state-metrics, njll.). Ukulungiselelwa kwethu kuvuliwe, futhi ungakwazi ukubuka i-PR .
Ukuphahlazeka
NgoLwesihlanu, Julayi 19, omunye wonjiniyela wethula iqoqo elisha elizinikele leCortex lekhasimende elikhulu. Ukulungiselelwa kwaleli qoqo akuzange kuhlanganise izinto ezintsha eziza kuqala ze-pod, ngakho-ke wonke ama-pods amasha abelwa okubalulekile okuzenzakalelayo - isilinganiso.
Iqoqo le-Kubernetes alizange libe nezinsiza ezanele zeqoqo elisha le-Cortex, futhi iqoqo elikhona le-Cortex alizange libuyekezwe (Ama-Ingesters ashiywe ngaphandle phezulu okubalulekile). Njengoba i-Ingesters yeqoqo elisha ngokuzenzakalelayo yayinayo isilinganiso okubalulekile, kanye nama-pods akhona ekukhiqizeni asebenze ngaphandle kokubeka eqhulwini nhlobo, i-Ingesters yeqoqo elisha yathatha indawo ye-Ingesters kusukela kuqoqo lokukhiqiza elikhona le-Cortex.
I-ReplicaSet ye-Ingester ekhishiwe kuqoqo lokukhiqiza ithole i-pod ekhishiwe futhi yakha entsha ukuze kugcinwe inombolo ecacisiwe yamakhophi. I-pod entsha yabelwe ngokuzenzakalelayo isilinganiso okubalulekile, futhi enye i-Ingester βendalaβ ekukhiqizeni yalahlekelwa yizinsiza zayo. Umphumela waba inqubo ye-avalanche, okuholele ekususweni kwawo wonke ama-pods ku-Ingester kumaqoqo okukhiqiza e-Cortex.
I-Ingesters iyiqiniso futhi igcina idatha yamahora angu-12 adlule. Lokhu kusivumela ukuthi siwacindezele ngokuphumelelayo ngaphambi kokuwabhalela endaweni yokugcina isikhathi eside. Ukuze kuzuzwe lokhu, i-Cortex ishaba idatha kulo lonke uchungechunge isebenzisa Ithebula Le-Hash Elisabalalisiwe (i-DHT) futhi iphindaphinda uchungechunge ngalunye kuma-Ingesters amathathu isebenzisa ukungaguquguquki kwekhoramu yesitayela se-Dynamo. I-Cortex ayibhali idatha kuma-Ingesters akhutshaziwe. Ngakho-ke, lapho inani elikhulu lama-Ingesters lishiya i-DHT, i-Cortex ayikwazi ukunikeza ukuphindaphinda okwanele kokufakiwe, futhi ayaphahlazeka.
Ukutholwa Nokulungiswa
Izaziso ezintsha ze-Prometheus ezisekelwe "kubhajethi yephutha" (kusekelwe iphutha kwisabelomali - imininingwane izovela esihlokweni esizayo) yaqala ukukhala i-alamu imizuzu emi-4 ngemuva kokuqala kokuvalwa. Emizuzwini emihlanu elandelayo noma ngaphezulu, senze ukuxilonga futhi senyusa iqoqo le-Kubernetes elingaphansi ukuze libambe womabili amaqoqo okukhiqiza amasha nakhona.
Ngemva kweminye imizuzu emihlanu, ama-Ingesters amadala abhala ngempumelelo idatha yawo, amasha aqala, futhi amaqoqo eCortex aphinde atholakala.
Eminye imizuzu eyi-10 ichithelwe ekuxilongeni nasekulungiseni amaphutha angaphandle kwenkumbulo (OOM) kusuka kumaphroksi ahlehlayo okuqinisekisa atholakala ngaphambi kwe-Cortex. Amaphutha e-OOM adalwe ukwanda okuphindwe kashumi kwe-QPS (sikholwa ukuthi kungenxa yezicelo ezinolaka kakhulu ezivela kumaseva eklayenti e-Prometheus).
Imiphumela
Isikhathi esiphelele sokuphumula bekuyimizuzu engama-26. Ayikho idatha elahlekile. I-Ingesters ilayishe ngempumelelo yonke idatha esenkumbulweni kusitoreji sesikhathi eside. Ngesikhathi sokuvala shaqa, iklayenti Prometheus amaseva kubhafa asusiwe (ukude) okuqoshiwe usebenzisa isekelwe ku-WAL (ibhalwe ngu kusuka eGrafana Labs) futhi waphinda okuhlulekile kubhala ngemuva kwengozi.
Imisebenzi yokubhala iqoqo lokukhiqiza
okutholakele
Kubalulekile ukufunda kulesi sigameko futhi uthathe izinyathelo ezidingekayo ukuze ugweme ukuphindeka kwaso.
Uma sibheka emuva, bekungafanele sibeke okuzenzakalelayo isilinganiso okubalulekile kuze kube yilapho wonke ama-Ingesters ekukhiqizweni athola phezulu okubalulekile. Ngaphezu kwalokho, kwakudingekile ukubanakekela kusengaphambili phezulu okubalulekile. Konke sekulungisiwe manje. Sithemba ukuthi ulwazi lwethu luzosiza ezinye izinhlangano ezicabangela ukusebenzisa izinto eziza kuqala ku-Kubernetes.
Sizokwengeza ileveli eyengeziwe yokulawula phezu kokusetshenziswa kwanoma yiziphi izinto ezengeziwe okulungiselelwa kwazo kusemhlabeni wonke kuqoqo. Kusukela manje kuqhubeke, izinguquko ezinjalo zizohlolwa bΠΎabantu abengeziwe. Ukwengeza, ukuguqulwa okudale ukuphahlazeka kuthathwe njengokuncane kakhulu kudokhumenti yephrojekthi ehlukile - kwaxoxwa ngakho kuphela odabeni lwe-GitHub. Kusukela manje kuqhubeke, zonke izinguquko ezinjalo ezilungiselelweni zizohambisana nemibhalo yephrojekthi efanele.
Okokugcina, sizokwenza ngokuzenzakalelayo ukushintsha usayizi wommeleli ohlanekezela ubuqiniso ukuze sinqande ukugcwala kwe-OOM esikubonile, futhi sizobuyekeza izilungiselelo ezizenzakalelayo ze-Prometheus ezihlobene nokubuyela emuva nokukala ukuze kuvinjelwe izinkinga ezifanayo ngokuzayo.
Ukwehluleka futhi kwaba nemiphumela emihle: ngemva kokuthola izinsiza ezidingekayo, i-Cortex yalulama ngokuzenzakalelayo ngaphandle kokungenelela okwengeziwe. Siphinde sazuza ulwazi olubalulekile ekusebenzeni nalo - uhlelo lwethu olusha lokuhlanganisa amalogi - olusize ekuqinisekiseni ukuthi wonke ama-Ingesters aziphatha kahle ngesikhathi nangemva kokwehluleka.
I-PS evela kumhumushi
Funda futhi kubhulogi yethu:
- Β«";
- Β«";
- Β«";
- Β«".
Source: www.habr.com