Kuyang'anira Gulu la Kubernetes: Chidule ndi Chiyambi cha Prometheus

Tiyeni tiwone lingaliro la Kubernetes kuwunika, dziwani chida cha Prometheus, ndikulankhula za kuchenjeza.

Mutu wowunika ndi wochuluka, sungathe kufotokozedwa m'nkhani imodzi. Cholinga cha nkhaniyi ndikupereka mwachidule zida, malingaliro, ndi njira.

Zomwe zili m'nkhaniyi ndikufinya kuchokera phunziro lotseguka la sukulu "Slurm". Ngati mukufuna kuchita maphunziro athunthu - lembani maphunzirowa Kuwunikira ndikudula mitengo ku Kubernetes.

Zomwe zimayang'aniridwa mu gulu la Kubernetes

maseva akuthupi. Ngati gulu la Kubernetes likugwiritsidwa ntchito pa maseva ake, muyenera kuyang'anira thanzi lawo. Zabbix amayang'anira ntchitoyi; ngati mutagwira naye ntchito, ndiye kuti simuyenera kukana, sipadzakhala mikangano. Ndi Zabbix yomwe imayang'anira momwe ma seva athu alili.

Tiyeni tipitirire kuwunikira pamlingo wamagulu.

Zida Zowongolera Ndege: API, Scheduler ndi ena. Pang'ono ndi pang'ono, muyenera kuonetsetsa kuti API ya ma seva kapena etcd ndi yaikulu kuposa 0. Etcd akhoza kubwezera ma metrics ambiri: ndi ma disks omwe amazungulira, ndi thanzi la gulu lake etcd, ndi ena.

Docker adawonekera kalekale ndipo aliyense akudziwa bwino za zovuta zake: zotengera zambiri zimatulutsa amaundana ndi zovuta zina. Chifukwa chake, Docker palokha, ngati kachitidwe, iyeneranso kuwongoleredwa, kuti ipezeke.

dns. Ngati DNS igwera mgulu, ndiye kuti ntchito yonse ya Discovery idzagwa pambuyo pake, kuyimba kuchokera ku ma pods kupita ku ma pods kumasiya kugwira ntchito. Muzochita zanga, panalibe mavuto otero, koma izi sizikutanthauza kuti dziko la DNS siliyenera kuyang'aniridwa. Pemphani latency ndi ma metric ena amatha kutsatiridwa pa CoreDNS.

Ingress. Ndikofunikira kuyang'anira kupezeka kwa ingresses (kuphatikizapo Ingress Controller) monga malo olowera mu polojekitiyi.

Zigawo zazikulu za gululo zaphwasulidwa - tsopano tiyeni titsike pamlingo wochotsa.

Zitha kuwoneka kuti mapulogalamuwa amayendetsedwa m'matumba, zomwe zikutanthauza kuti amayenera kuwongolera, koma zenizeni sizili choncho. Ma pods ndi ephemeral: lero amayenda pa seva imodzi, mawa pa ina; lero alipo 10, mawa 2. Choncho, palibe amene amayang'anira makoko. M'kati mwazomangamanga za microservice, ndikofunikira kwambiri kuwongolera kupezeka kwa pulogalamu yonse. Makamaka, yang'anani kupezeka kwa ma endpoints: kodi pali chilichonse chimagwira ntchito? Ngati kugwiritsa ntchito kulipo, ndiye kuti chimachitika ndi chiyani kumbuyo kwake, ndi ma replicas angati omwe ali pano - awa ndi mafunso a dongosolo lachiwiri. Palibe chifukwa choyang'anira zochitika zapadera.

Pamlingo womaliza, muyenera kuyang'anira magwiridwe antchito a pulogalamuyo, tengani ma metric abizinesi: kuchuluka kwa malamulo, machitidwe a ogwiritsa ntchito, ndi zina zotero.

Prometheus

Njira yabwino kwambiri yowunikira masango ndi Prometheus. Sindikudziwa chida chilichonse chomwe chingafanane ndi Prometheus pankhani yaukadaulo komanso kugwiritsa ntchito mosavuta. Ndizothandiza pazitukuko zosinthika, kotero akamanena "Kubernetes monitoring", nthawi zambiri amatanthauza Prometheus.

Pali zosankha zingapo zoyambira ndi Prometheus: pogwiritsa ntchito Helm, mutha kukhazikitsa Prometheus kapena Prometheus Operator.

1. Prometheus nthawi zonse. Chilichonse chili bwino ndi iye, koma muyenera kukonza ConfigMap - makamaka, lembani mafayilo osinthika ozikidwa pamawu, monga tidachitira kale, mamangidwe a microservice asanachitike.
2. Prometheus Operator ndi wofalikira pang'ono, wovuta pang'ono malinga ndi malingaliro amkati, koma ndi osavuta kugwira nawo ntchito: pali zinthu zosiyana, zotsalira zimawonjezeredwa kumagulu, kotero zimakhala zosavuta kuziwongolera ndikusintha.

Kuti mumvetsetse malonda, ndikupangira kukhazikitsa Prometheus wokhazikika poyamba. Muyenera kukonza zonse kudzera mu config, koma izi zidzakhala zopindulitsa: mudzazindikira zomwe zili ndi momwe zimapangidwira. Mu Prometheus Operator, nthawi yomweyo mumakwera pamwamba, ngakhale mutha kuzamanso mwakuya ngati mukufuna.

Prometheus imaphatikizidwa bwino ndi Kubernetes: imatha kulowa ndikulumikizana ndi API Server.

Prometheus ndi yotchuka, ndichifukwa chake kuchuluka kwa mapulogalamu ndi zilankhulo zamapulogalamu zimathandizira. Thandizo likufunika, popeza Prometheus ali ndi mawonekedwe ake a metrics, ndipo kuti muwasamutse, mufunika laibulale mkati mwa pulogalamuyi kapena wogulitsa kunja wokonzeka. Ndipo pali ochepa otere otumiza kunja. Mwachitsanzo, pali PostgreSQL Exporter: imatengera deta kuchokera ku PostgreSQL ndikuisintha kukhala mtundu wa Prometheus kuti Prometheus athe kugwira nawo ntchito.

Zomangamanga za Prometheus

Seva ya Prometheus - ichi ndi gawo la seva, ubongo wa Prometheus. Apa ndi pamene ma metrics amasungidwa ndi kukonzedwa.

Ma metrics amasungidwa mu database ya nthawi (TSDB). TSDB sinkhokwe yosiyana, koma phukusi la Go lomwe limapangidwa ku Prometheus. Mwachidule, zonse zili mu binary imodzi.

Osasunga deta mu TSDB kwa nthawi yayitali

Zomangamanga za Prometheus sizoyenera kusungirako nthawi yayitali ma metric. Nthawi yosungira yosasinthika ndi masiku 15. Mutha kupyola malire awa, koma kumbukirani: mukasunga zambiri mu TSDB komanso mukamachita nthawi yayitali, zimawononga zambiri. Kusunga mbiri yakale ku Prometheus kumawonedwa kukhala koyipa.

Ngati muli ndi magalimoto ambiri, kuchuluka kwa ma metrics ndi mazana masauzande pamphindikati, ndiye kuti ndi bwino kuchepetsa kusungidwa kwawo ndi disk space kapena nthawi. Nthawi zambiri, "data yotentha" imasungidwa mu TSDB, ma metrics m'maola ochepa chabe. Posungirako nthawi yayitali, kusungirako kwakunja kumagwiritsidwa ntchito m'ma database omwe ali oyeneradi izi, mwachitsanzo, InfluxDB, ClickHouse, ndi zina zotero. Ndawona ndemanga zabwino zambiri za ClickHouse.

Prometheus Server imagwira ntchito pachitsanzo Kokani: iye mwini amapita ku ma metric mpaka kumapeto komwe tidamupatsa. Iwo adati: "pita ku Seva ya API," ndipo imapita kumeneko nambala iliyonse yamasekondi ndikutenga ma metric.

Kwa zinthu zomwe zimakhala ndi moyo waufupi (ntchito kapena ntchito ya cron) zomwe zimatha kuwoneka pakati pa nthawi yolemba, pali gawo la Pushgateway. Ma metrics ochokera kuzinthu zazifupi amakankhidwira mmenemo: ntchito yakwera, yachitapo kanthu, inatumiza ma metrics ku Pushgateway ndikumaliza. Patapita kanthawi, Prometheus adzatsika pa liwiro lake ndi kutenga ma metrics ku Pushgateway.

Kukonza zidziwitso ku Prometheus pali gawo lina - Woyang'anira Alert. Ndipo malamulo ochenjeza. Mwachitsanzo, muyenera kupanga chenjezo ngati seva API ndi 0. Pamene chochitikacho chikuyambitsidwa, chenjezo limaperekedwa kwa woyang'anira tcheru kuti atumizenso. Woyang'anira zidziwitso ali ndi makonda osinthika: gulu limodzi la zidziwitso litha kutumizidwa ku macheza a telegalamu a admins, lina kumacheza a omanga, ndipo lachitatu kumacheza a ogwira ntchito. Zidziwitso zitha kutumizidwa ku Slack, Telegraph, imelo ndi njira zina.

Ndipo potsiriza, ndikuwuzani za Prometheus wakupha - Kupeza. Mukamagwira ntchito ndi Prometheus, simuyenera kufotokoza maadiresi enieni a zinthu zowunikira, ndikwanira kukhazikitsa mtundu wawo. Ndiye kuti, simuyenera kulemba "ayi adilesi ya IP, nayi doko - monitor", m'malo mwake, muyenera kudziwa ndi mfundo ziti zomwe mungapeze zinthu izi (zolinga - zolinga). Prometheus mwiniwake, kutengera ndi zinthu zomwe zikugwira ntchito pano, amakoka zofunikira ndikuziwonjezera kuwunika.

Njirayi ikugwirizana bwino ndi kapangidwe ka Kubernetes, komwe chirichonse chimayandamanso: lero pali ma seva 10, mawa 3. Kuti tisasonyeze adilesi ya IP ya seva nthawi zonse, tinalembapo kale momwe tingapezere - ndipo Discovering idzachita.

Chilankhulo cha Prometheus chimatchedwa Mtengo wa PromQL. Pogwiritsa ntchito chilankhulochi, mutha kupeza ma metrics enaake ndikusintha ndikupanga mawerengedwe owunikira potengera iwo.

https://prometheus.io/docs/prometheus/latest/querying/basics/

Простой запрос

    container_memory_usage_bytes

Математические операции

    container_memory_usage_bytes / 1024 / 1024

Встроенные функции

    sum(container_memory_usage_bytes) / 1024 / 1024

Уточнение запроса

    100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)

Pulogalamu yapaintaneti ya Prometheus

Prometheus ili ndi mawonekedwe ake ake, m'malo mwa minimalistic ukonde. Oyenera kokha kuwongolera kapena kuwonetsera.

Pa mzere wa Expression, mutha kulemba funso muchilankhulo cha PromQL.

Tabu ya Zidziwitso ili ndi malamulo ochenjeza, ndipo ali ndi magawo atatu:

1. osagwira ntchito - ngati chenjezo silikugwira ntchito panthawiyi, ndiye kuti, zonse zili bwino ndi izo, ndipo sizinagwire ntchito;
2. podikirira - izi ndi ngati chenjezo linagwira ntchito, koma kutumiza sikunapitebe. Kuchedwa kwakhazikitsidwa kuti kulipirire kuphatikizika kwa maukonde: ngati ntchito yomwe yafotokozedwayo yakwera mkati mwa mphindi imodzi, ndiye kuti alamu siyenera kuyimba;
3. kuwombera ndi udindo wachitatu, pamene chenjezo likuwunikira ndikutumiza mauthenga.

Mu Status menyu mupeza mwayi wodziwa zambiri za Prometheus. Palinso kusintha kwa zolinga (zolinga), zomwe takambirana pamwambapa.

Kuti mumve zambiri za mawonekedwe a Prometheus, onani munkhani ya Slurm yowunika gulu la Kubernetes.

Kuphatikiza ndi Grafana

Mu mawonekedwe a tsamba la Prometheus simupeza ma graph okongola komanso omveka omwe mutha kudziwa momwe gululi lilili. Kuti amange, Prometheus akuphatikizana ndi Grafana. Awa ndi ma dashboard omwe timapeza.

Kukhazikitsa kuphatikiza kwa Prometheus ndi Grafana sikovuta konse; malangizo angapezeke pazolembedwa: Thandizo la GRAFANA KWA PROMETHEUSChabwino, ndimaliza ndi izi.

M'nkhani zotsatirazi tipitiliza mutu wowunika: tikambirana za kutolera ndi kusanthula zipika pogwiritsa ntchito Grafana Loki ndi zida zina.

Wolemba: Marcel Ibraev, woyang'anira wovomerezeka wa Kubernetes, woyeserera pakampaniyo Southbridge, wokamba komanso woyambitsa maphunziro Slurm.

Source: www.habr.com