Kutarisisa Kubernetes Cluster: Kuongorora uye Sumo yePrometheus

Funga nezve pfungwa yeKubernetes yekutarisa, zivana nePrometheus chishandiso, uye taura nezve yambiro.

Musoro wekutarisa wakawanda, haugone kuparadzaniswa muchinyorwa chimwe. Chinangwa chechinyorwa ichi ndechekupa mucherechedzo wezvishandiso, pfungwa nemaitiro.

Zvinhu zvechinyorwa ndezvekusvinwa kubva hurukuro yakavhurika yechikoro "Slurm". Kana iwe uchida kutora kosi yakazara - nyoresa kune kosi pa Kuongorora uye kutema matanda muKubernetes.

Chii chinotariswa muKubernetes cluster

maseva emuviri. Kana iyo Kubernetes cluster ikaiswa pamaseva ayo, unofanirwa kutarisa hutano hwavo. Zabbix inobata basa iri; kana ukashanda naye, saka haufaniri kuramba, hapazovi nemakakatanwa. Iyo Zabbix inotarisa mamiriro emaseva edu.

Ngatienderere mberi kune monitoring pachikamu checluster.

Kudzora Ndege zvikamu: API, Scheduler nevamwe. Pazvishoma, iwe unofanirwa kuve nechokwadi chekuti API yemaseva kana etcd yakakura kupfuura 0. Etcd inogona kudzorera akawanda metrics: nemadhisiki pairi kutenderera, nehutano hweiyo etcd cluster, nezvimwe.

Docker yakaonekwa kare kare uye munhu wese anonyatsoziva matambudziko ayo: yakawanda yemidziyo inoburitsa chando uye mamwe matambudziko. Naizvozvo, Docker pachayo, sehurongwa, inofanirwawo kudzorwa, zvirinani kuti iwanikwe.

dns. Kana DNS ikadonha musumbu, ipapo iyo yese Discovery sevhisi inodonha mushure mayo, nhare kubva kumapodhi kuenda kumapodhi dzinomira kushanda. Mukuita kwangu, pakanga pasina matambudziko akadaro, asi izvi hazvirevi kuti mamiriro eDNS haafaniri kuongororwa. Kumbira latency uye mamwe metrics anogona kuteverwa paCoreDNS.

Ingress. Izvo zvinodikanwa kudzora kuwanikwa kweingresss (kusanganisira iyo Ingress Controller) senzvimbo dzekupinda purojekiti.

Izvo zvikamu zvikuru zvesumbu zvakabviswa - ikozvino ngatiendei kudzika kusvika padanho rekubvisa.

Zvingaita sekuti maapplication anomhanya mumapods, zvinoreva kuti anoda kudzorwa, asi muchokwadi haasi. Mapodhi ari ephemeral: nhasi anomhanya pane imwe server, mangwana pane imwe; nhasi kune gumi, mangwana 10. Naizvozvo, hapana anotarisa mapodhi. Mukati meiyo microservice architecture, zvakanyanya kukosha kudzora kuwanikwa kwechishandiso chakazara. Kunyanya, tarisa kuwanikwa kwesevhisi endpoints: pane chero chinhu chinoshanda? Kana iyo application iripo, zvino chii chinoitika kuseri kwaro, mangani makopi azvino - iyi mibvunzo yeiyo yechipiri. Hapana chikonzero chekutarisa zviitiko zvega.

Padanho rekupedzisira, iwe unofanirwa kudzora kushanda kwechikumbiro pachako, tora bhizinesi metrics: nhamba yemirairo, maitiro evashandisi, zvichingodaro.

Prometheus

Iyo yakanakisa sisitimu yekutarisa cluster ndeye Prometheus. Ini handizive chero chishandiso chinogona kuenderana nePrometheus maererano nemhando uye nyore kushandisa. Yakanakira zvivakwa zvinoshanduka, saka kana vachiti "Kubernetes yekutarisa", vanowanzoreva Prometheus.

Pane akati wandei sarudzo dzekutanga nePrometheus: uchishandisa Helm, unogona kuisa yakajairika Prometheus kana Prometheus Operator.

1. Nguva dzose Prometheus. Zvese zvakanaka naye, asi iwe unofanirwa kugadzirisa ConfigMap - kutaura zvazviri, nyora zvinyorwa-zvakavakirwa mafaera ekugadzirisa, sezvataiita kare, pamberi peiyo microservice architecture.
2. Prometheus Operator yakawedzera kupararira, yakawedzera kuomarara maererano nemukati mepfungwa, asi zviri nyore kushanda nayo: pane zvakaparadzana zvinhu, zvisungo zvinowedzerwa kusumbu, saka zviri nyore kudzora nekugadzirisa.

Kuti unzwisise chigadzirwa, ini ndinokurudzira kuisa yenguva dzose Prometheus kutanga. Iwe uchafanirwa kugadzirisa zvese kuburikidza neiyo config, asi izvi zvichabatsira: iwe unozoona kuti ndezvipi uye sei zvakagadziriswa. MuPrometheus Operator, iwe unobva wasimuka uchienda kumusoro, kunyangwe iwe uchigona zvakare kunyura mukadzika kana uchida.

Prometheus yakanyatsobatanidzwa neKubernetes: inogona kuwana uye kupindirana neiyo API Server.

Prometheus yakakurumbira, ndosaka nhamba huru yezvishandiso uye mitauro yekuronga ichiitsigira. Tsigiro inodiwa, sezvo Prometheus iine yayo metrics fomati, uye kuti uifambise, unoda raibhurari mukati mekushandisa kana yakagadzirira-yakagadzirwa kunze kunze. Uye kune vashoma vakadaro vatengesi kunze kwenyika. Semuenzaniso, kune PostgreSQL Exporter: inotora data kubva kuPostgreSQL uye inoshandura iyo kuPrometheus fomati kuitira kuti Prometheus ishande nayo.

Prometheus architecture

Prometheus Server ndiyo magumo ekumashure, uropi hwePrometheus. Metrics inochengetwa uye inogadziriswa pano.

Iwo metrics anochengetwa mune yenguva yakatevedzana dhatabhesi (TSDB). TSDB haisi dhatabhesi yakaparadzana, asi pasuru iri mumutauro weGo iyo yakadzikwa muPrometheus. Kutaura zvazviri, zvinhu zvose zviri mune imwe bhinary.

Usachengete data muTSDB kwenguva yakareba

Iyo Prometheus zvivakwa haina kukodzera kuchengetwa kwenguva refu kwema metrics. The default retention nguva ndeyemazuva gumi nemashanu. Iwe unogona kudarika uyu muganho, asi ramba uchifunga: iyo yakawanda data yaunochengeta muTSDB uye nekurebesa kwaunoiita, iyo yakawanda zviwanikwa yainopedza. Kuchengeta nhoroondo data muPrometheus inoonekwa seyakaipa tsika.

Kana iwe uine traffic yakakura, huwandu hwemetrics mazana ezviuru pasekondi, saka zviri nani kudzikisira kuchengetedza kwavo nedhisiki nzvimbo kana nenguva. Kazhinji, "hot data" inochengetwa muTSDB, metrics mumaawa mashoma. Kwekuchengetedza kwenguva refu, kuchengetedza kwekunze kunoshandiswa mune idzo dhatabhesi dzakanyatsokodzera izvi, semuenzaniso, InfluxDB, ClickHouse, zvichingodaro. Ndakaona mamwe mawongororo akanaka nezve ClickHouse.

Prometheus Server inoshanda pamuenzaniso pull: anoenda kumametrics kune iwo maendpoints atakamupa. Ivo vakati: "enda kune API Server", uye anoenda ikoko yega n-th nhamba yemasekonzi uye anotora metrics.

Kune zvinhu zvine hupenyu hupfupi (basa kana cron basa) zvinogona kuoneka pakati penguva yekurasa, pane Pushgateway chikamu. Metrics kubva kuzvinhu zvenguva pfupi inosundirwa mairi: basa rasimuka, rakaita chiito, rakatumira metrics kuPushgateway uye rakapedzwa. Mushure mechinguva, Prometheus achauya pasi nekumhanya kwayo uye otora aya metrics kubva kuPushgateway.

Kugadzirisa zviziviso muPrometheus pane chikamu chakasiyana - Alertmanager. Uye mitemo yekuzivisa. Semuenzaniso, iwe unofanirwa kugadzira yambiro kana sevha API iri 0. Kana chiitiko ichipisa, chenjedzo inopfuudzwa kune maneja wekuchenjerera kuti uwedzere kutumira. Maneja wekuzivisa ane anochinjika magadzirirwo enzira: rimwe boka rekuzivisa rinogona kutumirwa kune admins' telegraph chat, imwe kune vanogadzira chat, uye chetatu kune yevashandi vezvivakwa chat. Zviziviso zvinogona kutumirwa kuSlack, Teregiramu, email, uye mamwe chiteshi.

Uye pakupedzisira, ini ndinokuudza nezve Prometheus muurayi chimiro - Kuziva. Paunenge uchishanda nePrometheus, haufanire kutsanangura kero dzezvinhu zvekutarisa, zvakakwana kuseta mhando yavo. Kureva kuti, haufanirwe kunyora "heino kero yeIP, heino chiteshi chengarava", pane kudaro, iwe unofanirwa kuona kuti ndeapi misimboti yekuwana zvinhu izvi (targets - zvinangwa). Prometheus pachayo, zvichienderana nekuti zvinhu zvipi zviri kushanda izvozvi, inokwevera kumusoro izvo zvinodiwa uye inovawedzera pakutarisa.

Iyi nzira inonyatsoenderana neKubernetes chimiro, apo zvinhu zvose zvinoyangarara: nhasi kune 10 maseva, mangwana 3. Kuti varege kutsanangura IP kero yevhavha nguva imwe neimwe, vakanyora kamwe chete kuti vangaiwana sei - uye Discovering ichazviita. .

Mutauro wePrometheus unonzi PromQL. Uchishandisa mutauro uyu, unogona kuwana hunhu hwemametric chaiwo wobva waashandura, wovaka maverengero ekuongorora anoenderana nawo.

https://prometheus.io/docs/prometheus/latest/querying/basics/

Простой запрос

    container_memory_usage_bytes

Математические операции

    container_memory_usage_bytes / 1024 / 1024

Встроенные функции

    sum(container_memory_usage_bytes) / 1024 / 1024

Уточнение запроса

    100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)

Prometheus web interface

Prometheus ine yayo, yakaringana minimalistic web interface. Inokodzera chete kugadzirisa kana kuratidza.

Mumutsara weKuratidzira, unogona kunyora mubvunzo mumutauro wePromQL.

Iyo Alerts tab ine mitemo yekuzivisa, uye ine matatu matatu:

1. kusashanda - kana yambiro isiri kushanda panguva ino, ndiko kuti, zvinhu zvese zvakanaka nazvo, uye hazvina kushanda;
2. zvakamirira - izvi kana yambiro yakashanda, asi kutumira hakusati kwapfuura. Iko kunonoka kwakagadzirirwa kubhadhara kunetiweki blinking: kana sevhisi yakatsanangurwa yakakwira mukati meminiti, ipapo alarm haifanire kuridzwa;
3. kupfura inzvimbo yechitatu kana yambiro ichivheneka uye kutumira mameseji.

Mune iyo Status menyu iwe unowana kuwana ruzivo nezve chii Prometheus. Panewo kuchinja kune zvinangwa (targets), izvo zvatakataura pamusoro apa.

Kuti uwane rumwe ruzivo nezve Prometheus interface, ona muhurukuro yaSlurm yekutarisa boka reKubernetes.

Kubatanidzwa naGrafana

MuPrometheus web interface, hauzowana yakanaka uye inonzwisisika magirafu kubva kwaunogona kutora mhedziso nezve mamiriro esumbu. Kuvavaka, Prometheus inosanganiswa neGrafana. Tinowana dashboards akadaro.

Kumisikidza iyo Prometheus uye Grafana kubatanidzwa hakuna kuoma zvachose, unogona kuwana mirairo mune zvinyorwa: GRAFANA SUPPORT FOR PROMETHEUSZvakanaka, ndichapedzisa neizvi.

Muzvinyorwa zvinotevera, tichaenderera mberi nemusoro wekutarisa: isu tichataura nezvekuunganidza nekuongorora matanda tichishandisa Grafana Loki uye mamwe maturusi.

Munyori: Marcel Ibraev, akavimbiswa Kubernetes maneja, anodzidzira mainjiniya mukambani. Southbridge, mutauri uye kosi yekuvandudza Slurm.

Source: www.habr.com