Ukuqapha I-Kubernetes Cluster: Uhlolojikelele kanye nesingeniso ku-Prometheus

Cabanga ngomqondo wokuqapha kwe-Kubernetes, jwayelana nethuluzi le-Prometheus, futhi ukhulume ngokuxwayisa.

Isihloko sokuqapha sigcwele, asikwazi ukuhlakazwa esihlokweni esisodwa. Inhloso yalo mbhalo ukunikeza isifinyezo samathuluzi, imiqondo nezindlela.

Impahla ye-athikili iwukukhama kusuka inkulumo evulekile yesikole "Slurm". Uma ufuna ukwenza isifundo esigcwele - bhalisela isifundo ku Ingqalasizinda yokuqapha nokugawulwa kwemithi e-Kubernetes.

Yini egadwa kuqoqo le-Kubernetes

amaseva ngokomzimba. Uma iqoqo le-Kubernetes lisatshalaliswa kumaseva alo, udinga ukuqapha impilo yabo. I-Zabbix iphethe lo msebenzi; uma usebenza naye, awudingi ukwenqaba, ngeke kube nezingxabano. I-Zabbix eqapha isimo samaseva ethu.

Asiqhubekele ekuqapheni ezingeni leqoqo.

Izingxenye Zendiza Yokulawula: I-API, i-Scheduler nabanye. Okungenani, udinga ukwenza isiqiniseko sokuthi i-API yamaseva noma njlld inkulu kuno-0. I-Etcd ingabuyisa amamethrikhi amaningi: ngamadiski lapho iphotha khona, ngempilo yeqoqo layo njll.

Docker yavela kudala futhi wonke umuntu uzazi kahle izinkinga zayo: iziqukathi eziningi zikhiqiza amaqhwa nezinye izinkinga. Ngakho-ke, i-Docker ngokwayo, njengohlelo, kufanele futhi ilawulwe, okungenani ukuze itholakale.

I-DNS. Uma i-DNS iwe ku-cluster, yonke isevisi ye-Discovery izowa ngemva kwayo, izingcingo ezisuka kuma-pods ziye kuma-pods zizoyeka ukusebenza. Emsebenzini wami, azikho izinkinga ezinjalo, kodwa lokhu akusho ukuthi isimo se-DNS asidingi ukugadwa. Cela ukubambezeleka namanye amamethrikhi angalandelelwa ku-CoreDNS.

I-Ingress. Kuyadingeka ukulawula ukutholakala kwama-ingresses (kuhlanganise ne-Ingress Controller) njengamaphoyinti okungena kuphrojekthi.

Izingxenye eziyinhloko ze-cluster zihlakaziwe - manje ake sehlele ezingeni lokungafinyeleli.

Kungase kubonakale sengathi izinhlelo zokusebenza zisebenza ngama-pods, okusho ukuthi zidinga ukulawulwa, kodwa empeleni akunjalo. Ama-pods angama-ephemeral: namuhla agijima kuseva eyodwa, kusasa kwenye; namuhla kukhona abayi-10, kusasa 2. Ngakho-ke, akekho oqapha ama-pods. Ngaphakathi kwe-microservice architecture, kubaluleke kakhulu ukulawula ukutholakala kohlelo lokusebenza lulonke. Ikakhulukazi, hlola ukutholakala kwezindawo zokugcina zesevisi: ingabe kukhona okusebenzayo? Uma isicelo sitholakala, khona-ke kwenzekani ngemuva kwalo, zingaki izifaniso ezikhona manje - lena imibuzo yohlelo lwesibili. Asikho isidingo sokuqapha izimo ngazinye.

Ezingeni lokugcina, udinga ukulawula ukusebenza kohlelo lokusebenza ngokwalo, thatha amamethrikhi ebhizinisi: inani lama-oda, ukuziphatha komsebenzisi, njalonjalo.

Prometheus

Uhlelo olungcono kakhulu lokuqapha iqoqo yi Prometheus. Angazi noma yiliphi ithuluzi elingafanisa i-Prometheus ngokwekhwalithi nokusebenziseka kalula. Kuhle kungqalasizinda eguquguqukayo, ngakho-ke uma bethi "ukuqapha kwe-Kubernetes", ngokuvamile basho i-Prometheus.

Kunezinketho ezimbalwa zokuqalisa nge-Prometheus: usebenzisa i-Helm, ungafaka i-Prometheus evamile noma i-Prometheus Operator.

1. I-Prometheus evamile. Konke kuhamba kahle naye, kodwa udinga ukulungisa i-ConfigMap - empeleni, bhala amafayela wokucushwa asekelwe ematheksthini, njengoba senza ngaphambili, ngaphambi kwezakhiwo ze-microservice.
2. I-Prometheus Operator isabalele kancane, inkimbinkimbi kancane ngokuya nge-logic yangaphakathi, kodwa kulula ukusebenza nayo: kukhona izinto ezihlukene, izifinyezo zengezwa kuqoqo, ngakho-ke kulula kakhulu ukulawula nokulungisa.

Ukuze uqonde umkhiqizo, ngincoma ukuthi ufake i-Prometheus evamile kuqala. Kuzodingeka ulungiselele yonke into ngokusebenzisa i-config, kodwa lokhu kuzoba yinzuzo: uzothola ukuthi yini eyalokho nokuthi ihlelwe kanjani. Ku-Prometheus Operator, ukhuphuka ngokushesha uye phezulu, yize ungangena ekujuleni uma uthanda.

I-Prometheus ihlanganiswe kahle ne-Kubernetes: ingafinyelela futhi ihlanganyele neseva ye-API.

I-Prometheus idumile, yingakho inani elikhulu lezinhlelo zokusebenza nezilimi zokuhlela ziyisekela. Ukusekelwa kuyadingeka, njengoba i-Prometheus inefomethi yayo yamamethrikhi, futhi ukuze uyidlulise, udinga umtapo wezincwadi ongaphakathi kohlelo lokusebenza noma isithekeli esenziwe ngomumo. Futhi kukhona abambalwa abanjalo abathumela ngaphandle. Isibonelo, kukhona I-PostgreSQL Exporter: ithatha idatha ku-PostgreSQL futhi iyiguqulela kufomethi ye-Prometheus ukuze i-Prometheus isebenze nayo.

I-Prometheus Architecture

Iseva ye-Prometheus ingemuva, ubuchopho bukaPrometheus. Amamethrikhi agcinwa futhi acutshungulwe lapha.

Amamethrikhi agcinwa kusizindalwazi sochungechunge lwesikhathi (TSDB). I-TSDB ayiyona isizindalwazi esihlukile, kodwa iphakheji ngolimi lwe-Go elishumekwe ku-Prometheus. Uma sikhuluma nje, yonke into iku-binary eyodwa.

Ungagcini idatha ku-TSDB isikhathi eside

Ingqalasizinda ye-Prometheus ayifanele isitoreji sesikhathi eside samamethrikhi. Isikhathi sokugcina esizenzakalelayo yizinsuku eziyi-15. Ungakwazi ukweqa lo mkhawulo, kodwa khumbula: uma ugcina idatha eyengeziwe ku-TSDB futhi uma uyenza isikhathi eside, izosebenzisa izinsiza ezengeziwe. Ukugcina idatha yomlando ku-Prometheus kuthathwa njengomkhuba omubi.

Uma unethrafikhi enkulu, inani lamamethrikhi ngamakhulu ezinkulungwane ngomzuzwana, ngakho-ke kungcono ukukhawulela isitoreji sabo ngesikhala sediski noma ngezikhathi. Ngokuvamile, "idatha eshisayo" igcinwa ku-TSDB, amamethrikhi emahoreni ambalwa nje. Ukugcina isikhathi eside, isitoreji sangaphandle sisetshenziswa kulezo dathabhethi ezifanele ngempela lokhu, isibonelo, i-InfluxDB, i-ClickHouse, njalonjalo. Ngibone ukubuyekezwa okuhle okwengeziwe nge-ClickHouse.

I-Prometheus Server isebenza kumodeli Donsa: uhamba ngamamethrikhi kulawo maphuzu esimnike wona. Bathe: "iya kuseva ye-API", futhi uya lapho njalo ngenombolo ye-n-th yemizuzwana futhi athathe amamethrikhi.

Ezintweni ezinesikhathi esifushane sempilo (umsebenzi noma umsebenzi we-cron) ezingavela phakathi kwezikhathi zokuklwebha, kunengxenye ye-Pushgateway. Amamethrikhi avela ezintweni zesikhathi esifushane aphushelwa kuwo: umsebenzi ukhuphukile, wenze isenzo, wathumela amamethrikhi ku-Pushgateway futhi waqedwa. Ngemva kwesikhashana, u-Prometheus uzokwehla ngesivinini sakhe futhi athathe lawa mamethrikhi ku-Pushgateway.

Ukumisa izaziso ku-Prometheus kunengxenye ehlukile - Umphathi we-Alert. Nemithetho yokuxwayisa. Isibonelo, udinga ukudala isexwayiso uma iseva ye-API ingu-0. Uma umcimbi uvutha, isaziso sidluliselwa kumphathi wesexwayiso ukuze sithunyelwe okwengeziwe. Umphathi wesexwayiso unezilungiselelo zomzila ezivumelana nezimo: iqembu elilodwa lezaziso lingathunyelwa engxoxweni yocingo yabaphathi, elinye engxoxweni yonjiniyela, kanye nengxenye yesithathu engxoxweni yabasebenzi bengqalasizinda. Izaziso zingathunyelwa ku-Slack, Telegram, i-imeyili, nezinye iziteshi.

Futhi ekugcineni, ngizokutshela mayelana nesici sombulali we-Prometheus - Ukuthola. Lapho usebenza ne-Prometheus, awudingi ukucacisa amakheli athile ezinto zokuqapha, kwanele ukusetha uhlobo lwazo. Okungukuthi, awudingi ukubhala "nali ikheli le-IP, nali i-port - monitor", kunalokho, udinga ukunquma ukuthi yiziphi izimiso zokuthola lezi zinto (izinhloso - imigomo). I-Prometheus ngokwayo, kuye ngokuthi yiziphi izinto ezisebenzayo okwamanje, idonsa ezidingekayo futhi ingeze ekuqapheni.

Le ndlela ihambisana kahle nesakhiwo se-Kubernetes, lapho yonke into iphinde intante: namuhla kukhona amaseva angu-10, kusasa 3. Ukuze bangacacisi ikheli le-IP leseva isikhathi ngasinye, babhala kanye ukuthi bangayithola kanjani - futhi i-Discovering izokwenza. .

Ulimi lwe-Prometheus lubizwa ngokuthi I-PromQL. Usebenzisa lolu limi, ungathola amanani amamethrikhi athile bese uwaguqula, wakhe izibalo zokuhlaziya ngokusekelwe kuwo.

https://prometheus.io/docs/prometheus/latest/querying/basics/

Простой запрос

    container_memory_usage_bytes

Математические операции

    container_memory_usage_bytes / 1024 / 1024

Встроенные функции

    sum(container_memory_usage_bytes) / 1024 / 1024

Уточнение запроса

    100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)

I-Prometheus web interface

I-Prometheus ine-interface yayo yewebhu, ene-minimalistic. Ifanele kuphela ukulungisa iphutha noma ukuboniswa.

Emugqeni we-Expression, ungabhala umbuzo ngolimi lwe-PromQL.

Ithebhu Yezaziso iqukethe imithetho yokuxwayisa, futhi inezimo ezintathu:

1. engasebenzi - uma isixwayiso singasebenzi okwamanje, okungukuthi, konke kuhamba kahle ngakho, futhi akuzange kusebenze;
2. okulindile - lokhu kungukuthi uma isaziso sisebenzile, kodwa ukuthumela akukadluli. Ukubambezeleka kusethelwe ukunxephezela ukucwayiza kwenethiwekhi: uma isevisi eshiwo ikhuphuka phakathi neminithi, i-alamu akumele ikhale okwamanje;
3. ukudubula kuyisimo sesithathu lapho isexwayiso sikhanyisa futhi sithumela imiyalezo.

Kumenyu Yesimo uzothola ukufinyelela olwazini mayelana nokuthi uyini u-Prometheus. Kukhona nokudlulela kokuhlosiwe (okuhlosiwe), esikhulume ngakho ngenhla.

Ukuze uthole ukwaziswa okwengeziwe okuningiliziwe kwe-interface ye-Prometheus, bheka enkulumweni ka-Slurm yokuqapha iqoqo le-Kubernetes.

Ukuhlanganiswa noGrafana

Ku-interface yewebhu ye-Prometheus, ngeke uthole amagrafu amahle futhi aqondakalayo lapho ungathola khona isiphetho mayelana nesimo seqoqo. Ukuzakha, u-Prometheus uhlanganiswe noGrafana. Sithola amadeshibhodi anjalo.

Ukusetha ukuhlanganiswa kwe-Prometheus ne-Grafana akunzima neze, ungathola imiyalo kumadokhumenti: I-GRAFANA UKUSEKELWA KWE-PROMETHEUSHhayi-ke ngizophetha ngalokhu.

Ezihlokweni ezilandelayo, sizoqhubeka nesihloko sokuqapha: sizokhuluma ngokuqoqa nokuhlaziya izingodo sisebenzisa i-Grafana Loki namanye amathuluzi.

Umbhali: UMarcel Ibraev, umlawuli we-Kubernetes oqinisekisiwe, unjiniyela osebenzayo enkampanini I-Southbridge, isipikha kanye nonjiniyela wesifundo Slurm.

Source: www.habr.com