Saib xyuas Kubernetes Cluster: Tus Txheej Txheem thiab Taw Qhia rau Prometheus

Xav txog lub tswv yim ntawm Kubernetes saib xyuas, paub txog cov cuab yeej Prometheus, thiab tham txog kev ceeb toom.

Lub ntsiab lus ntawm kev saib xyuas yog voluminous, nws tsis tuaj yeem disassembled hauv ib tsab xov xwm. Lub hom phiaj ntawm cov ntawv no yog los muab cov ntsiab lus ntawm cov cuab yeej, cov tswv yim thiab cov txheej txheem.

Cov khoom ntawm cov ntawv yog nyem los ntawm qhib kev qhuab qhia ntawm lub tsev kawm ntawv "Slurm". Yog tias koj xav mus kawm tag nrho - sau npe rau chav kawm ntawm Kev saib xyuas thiab kev nkag mus rau cov txheej txheem hauv Kubernetes.

Dab tsi yog saib xyuas hauv Kubernetes pawg

lub cev servers. Yog tias Kubernetes pawg raug xa mus rau nws cov servers, koj yuav tsum tau saib xyuas lawv txoj kev noj qab haus huv. Zabbix tuav txoj haujlwm no; yog tias koj ua haujlwm nrog nws, ces koj tsis tas yuav tsis kam, yuav tsis muaj kev tsis sib haum xeeb. Nws yog Zabbix uas saib xyuas lub xeev ntawm peb cov servers.

Cia peb mus saib xyuas ntawm pawg pawg.

Control Plane Cheebtsam: API, Scheduler thiab lwm yam. Yam tsawg kawg nkaus, koj yuav tsum ua kom paub tseeb tias API ntawm servers lossis lwm yam yog ntau dua 0. Etcd tuaj yeem rov qab tau ntau qhov kev ntsuas: los ntawm cov disks uas nws tig, los ntawm kev noj qab haus huv ntawm nws pawg etcd, thiab lwm yam.

docker tau tshwm sim ntev dhau los thiab txhua tus paub zoo txog nws cov teeb meem: ntau lub thawv tsim kom khov thiab lwm yam teeb meem. Yog li ntawd, Docker nws tus kheej, raws li ib tug system, kuj yuav tsum tau tswj, tsawg kawg yog rau muaj.

dns ua. Yog tias DNS poob rau hauv pawg, ces tag nrho cov kev pabcuam Discovery yuav poob tom qab nws, hu xov tooj los ntawm pods mus rau pods yuav tsum tsis ua haujlwm. Hauv kuv qhov kev coj ua, tsis muaj teeb meem zoo li no, tab sis qhov no tsis txhais tau tias lub xeev DNS tsis tas yuav tsum tau saib xyuas. Thov latency thiab qee qhov kev ntsuas tuaj yeem taug qab ntawm CoreDNS.

Ingress. Nws yog ib qho tsim nyog los tswj qhov muaj ntawm ingresses (xws li Ingress Controller) raws li cov ntsiab lus nkag rau qhov project.

Cov ntsiab lus tseem ceeb ntawm pawg tau raug rhuav tshem - tam sim no cia peb mus rau theem ntawm abstractions.

Nws yuav zoo li tias cov ntawv thov khiav hauv pods, uas txhais tau tias lawv yuav tsum tau tswj, tab sis qhov tseeb lawv tsis yog. Pods yog ephemeral: hnub no lawv khiav ntawm ib lub server, tag kis rau lwm tus; hnub no muaj 10 leej, tag kis 2. Yog li ntawd, tsis muaj leej twg saib xyuas lub kaus mom. Tsis pub dhau lub microservice architecture, nws tseem ceeb dua los tswj qhov muaj ntawm daim ntawv thov tag nrho. Tshwj xeeb, tshawb xyuas qhov muaj cov kev pabcuam kawg: puas muaj dab tsi ua haujlwm? Yog tias daim ntawv thov muaj, ces dab tsi tshwm sim tom qab nws, muaj pes tsawg replicas tam sim no - cov no yog cov lus nug ntawm qhov kev txiav txim thib ob. Tsis tas yuav saib xyuas cov xwm txheej ntawm tus kheej.

Nyob rau theem kawg, koj yuav tsum tswj hwm kev ua haujlwm ntawm daim ntawv thov nws tus kheej, coj cov kev ntsuas kev lag luam: tus naj npawb ntawm kev txiav txim, tus neeg siv tus cwj pwm, thiab lwm yam.

Prometheus

Qhov zoo tshaj plaws rau kev saib xyuas ib pawg yog Prometheus. Kuv tsis paub txog cov cuab yeej twg uas tuaj yeem phim Prometheus ntawm qhov zoo thiab yooj yim ntawm kev siv. Nws yog qhov zoo rau kev hloov pauv yooj yim, yog li thaum lawv hais tias "Kubernetes saib xyuas", lawv feem ntau txhais tau tias Prometheus.

Muaj ob peb txoj hauv kev los pib nrog Prometheus: siv Helm, koj tuaj yeem nruab Prometheus lossis Prometheus Operator.

1. Tsis tu ncua Prometheus. Txhua yam zoo nrog nws, tab sis koj yuav tsum teeb tsa ConfigMap - qhov tseeb, sau cov ntaub ntawv raws li kev teeb tsa, raws li peb tau ua ua ntej, ua ntej microservice architecture.
2. Prometheus Tus neeg teb xov tooj yog me ntsis nthuav tawm, qhov nyuaj me ntsis ntawm cov logic sab hauv, tab sis nws yooj yim dua los ua haujlwm nrog nws: muaj cov khoom sib cais, abstractions ntxiv rau pawg, yog li lawv yooj yim dua rau kev tswj thiab teeb tsa.

Yuav kom nkag siab txog cov khoom, kuv xav kom txhim kho Prometheus li niaj zaus ua ntej. Koj yuav tau teeb tsa txhua yam los ntawm kev teeb tsa, tab sis qhov no yuav muaj txiaj ntsig: koj yuav paub tias dab tsi yog dab tsi thiab nws tau teeb tsa li cas. Hauv Prometheus Operator, koj tam sim ntawd nce mus rau qhov tsis txaus ntseeg siab dua, txawm hais tias koj tuaj yeem nkag mus rau qhov tob yog tias koj xav tau.

Prometheus tau zoo nrog Kubernetes: nws tuaj yeem nkag mus thiab cuam tshuam nrog API Server.

Prometheus yog nrov, uas yog vim li cas ib tug loj tus naj npawb ntawm daim ntawv thov thiab programming lus txhawb nws. Kev txhawb nqa yog xav tau, txij li Prometheus muaj nws tus kheej metrics hom, thiab hloov nws, koj xav tau ib lub tsev qiv ntawv hauv daim ntawv thov lossis tus xa tawm npaj txhij. Thiab muaj ob peb yam xws li exporters. Piv txwv li, muaj PostgreSQL Exporter: nws siv cov ntaub ntawv los ntawm PostgreSQL thiab hloov nws mus rau Prometheus hom kom Prometheus tuaj yeem ua haujlwm nrog nws.

Prometheus architecture

Prometheus Server yog qhov kawg, lub hlwb ntawm Prometheus. Kev ntsuas tau muab khaws cia thiab ua tiav ntawm no.

Cov kev ntsuas tau muab khaws cia rau hauv lub sijhawm series database (TSDB). TSDB tsis yog cov ntaub ntawv cais, tab sis ib pob hauv Go lus uas tau kos hauv Prometheus. Roughly hais lus, txhua yam yog nyob rau hauv ib tug binary.

Tsis txhob khaws cov ntaub ntawv hauv TSDB ntev

Prometheus infrastructure tsis haum rau kev khaws cia ntev ntawm kev ntsuas. Lub sijhawm khaws cia ua ntej yog 15 hnub. Koj tuaj yeem tshaj qhov txwv no, tab sis nco ntsoov: ntau cov ntaub ntawv koj khaws hauv TSDB thiab ntev koj ua nws, cov peev txheej ntau nws yuav siv. Khaws cov ntaub ntawv keeb kwm hauv Prometheus yog suav tias yog kev coj tsis zoo.

Yog tias koj muaj tsheb loj loj, tus lej ntawm cov ntsuas yog ntau pua txhiab tus ib ob, ces nws yog qhov zoo dua los txwv lawv qhov chaw cia los ntawm qhov chaw disk lossis los ntawm lub sijhawm. Feem ntau, "cov ntaub ntawv kub" khaws cia hauv TSDB, kev ntsuas hauv ob peb teev xwb. Rau kev cia ntev dua, kev cia sab nraud yog siv rau hauv cov ntaub ntawv uas tsim nyog rau qhov no, piv txwv li, InfluxDB, ClickHouse, thiab lwm yam. Kuv pom ntau qhov kev tshuaj xyuas zoo txog ClickHouse.

Prometheus Server ua haujlwm ntawm tus qauv rub: nws mus rau metrics rau cov endpoints uas peb muab rau nws. Lawv hais tias: "mus rau API Server", thiab nws mus qhov ntawd txhua n-th tus lej ntawm vib nas this thiab siv qhov ntsuas.

Rau cov khoom uas muaj lub neej luv luv (txoj haujlwm lossis txoj haujlwm cron) uas tuaj yeem tshwm sim ntawm lub sijhawm scraping, muaj Pushgateway tivthaiv. Kev ntsuas los ntawm cov khoom siv luv luv tau raug thawb mus rau hauv nws: txoj haujlwm tau nce, ua haujlwm, xa cov ntsuas mus rau Pushgateway thiab ua tiav. Tom qab ib pliag, Prometheus yuav nqis los ntawm nws tus kheej thiab khaws cov kev ntsuas no los ntawm Pushgateway.

Txhawm rau teeb tsa cov ntawv ceeb toom hauv Prometheus muaj qhov sib cais - Alertmanager. Thiab cov cai ceeb toom. Piv txwv li, koj yuav tsum tsim ib qho kev ceeb toom yog tias tus neeg rau zaub mov API yog 0. Thaum qhov kev tshwm sim hluav taws kub, qhov kev ceeb toom raug xa mus rau tus thawj tswj kev ceeb toom kom xa mus ntxiv. Tus neeg saib xyuas kev ceeb toom muaj qhov hloov pauv tau yooj yim heev: ib pawg ntawm cov lus ceeb toom tuaj yeem xa mus rau admins 'telegram tham, lwm tus rau cov neeg tsim khoom sib tham, thiab ib feem peb ntawm cov neeg ua haujlwm hauv kev sib tham. Cov ntawv ceeb toom tuaj yeem xa mus rau Slack, Telegram, email, thiab lwm yam.

Thiab thaum kawg, kuv yuav qhia koj txog Prometheus killer feature - discovering. Thaum ua haujlwm nrog Prometheus, koj tsis tas yuav qhia qhov chaw nyob tshwj xeeb ntawm cov khoom rau kev saib xyuas, nws txaus los teeb lawv hom. Ntawd yog, koj tsis tas yuav sau "ntawm no yog qhov chaw nyob IP, ntawm no yog qhov chaw nres nkoj - saib", hloov pauv, koj yuav tsum txiav txim siab los ntawm cov ntsiab lus twg los nrhiav cov khoom no (lub hom phaj - cov hom phiaj). Prometheus nws tus kheej, nyob ntawm seb cov khoom twg tam sim no nquag, rub cov khoom tsim nyog thiab ntxiv rau kev saib xyuas.

Txoj hauv kev no haum zoo nrog cov qauv Kubernetes, qhov twg txhua yam kuj ntab: hnub no muaj 10 servers, tag kis 3. Txhawm rau kom tsis txhob qhia tus IP chaw nyob ntawm tus neeg rau zaub mov txhua zaus, lawv tau sau ib zaug yuav nrhiav tau li cas - thiab Tshawb nrhiav yuav ua li cas. .

Cov lus Prometheus hu ua PromQL. Siv cov lus no, koj tuaj yeem tau txais cov txiaj ntsig ntawm cov metrics tshwj xeeb thiab tom qab ntawd hloov lawv, tsim cov kev ntsuas ntsuas raws li lawv.

https://prometheus.io/docs/prometheus/latest/querying/basics/

Простой запрос

    container_memory_usage_bytes

Математические операции

    container_memory_usage_bytes / 1024 / 1024

Встроенные функции

    sum(container_memory_usage_bytes) / 1024 / 1024

Уточнение запроса

    100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)

Prometheus web interface

Prometheus muaj nws tus kheej, haum minimalistic web interface. Tsuas yog tsim nyog rau kev debug lossis ua qauv qhia.

Hauv kab Lus Qhia, koj tuaj yeem sau cov lus nug hauv PromQL hom lus.

Lub Alerts tab muaj cov cai ceeb toom, thiab lawv muaj peb yam xwm txheej:

1. tsis ua haujlwm - yog tias qhov kev ceeb toom tsis ua haujlwm tam sim no, uas yog, txhua yam zoo nrog nws, thiab nws tsis ua haujlwm;
2. tos - qhov no yog qhov kev ceeb toom ua haujlwm, tab sis kev xa tsis tau dhau mus. Qhov ncua sij hawm yog teem los them rau lub network blinking: yog tias qhov kev pabcuam tau teev tseg tau nce hauv ib feeb, ces lub tswb yuav tsum tsis tau suab;
3. Kev tua hluav taws yog qhov thib peb thaum lub suab ceeb toom teeb thiab xa cov lus.

Hauv cov ntawv qhia xwm txheej koj yuav pom kev nkag mus rau cov ntaub ntawv hais txog Prometheus yog dab tsi. Kuj tseem muaj kev hloov pauv mus rau lub hom phiaj (lub hom phiaj), uas peb tau tham txog saum toj no.

Rau cov ncauj lus kom ntxaws ntxiv ntawm Prometheus interface, saib hauv Slurm cov lus qhuab qhia ntawm kev saib xyuas Kubernetes pawg.

Kev koom ua ke nrog Grafana

Hauv lub vev xaib Prometheus, koj yuav tsis pom cov duab zoo nkauj thiab nkag siab tau los ntawm qhov koj tuaj yeem kos cov lus xaus txog lub xeev ntawm pawg. Txhawm rau tsim lawv, Prometheus tau koom nrog Grafana. Peb tau txais cov dashboards zoo li no.

Kev teeb tsa kev koom ua ke ntawm Prometheus thiab Grafana tsis yog qhov nyuaj, koj tuaj yeem nrhiav cov lus qhia hauv cov ntaub ntawv: GRAFANA SUPPORT RAU PROMETHEUSZoo, kuv mam li xaus nrog qhov no.

Hauv cov ntawv hauv qab no, peb yuav txuas ntxiv lub ntsiab lus ntawm kev saib xyuas: peb yuav tham txog kev sau thiab tshuaj xyuas cov cav siv Grafana Loki thiab lwm yam cuab yeej.

Tus sau: Marcel Ibraev, tau lees paub tus thawj coj Kubernetes, xyaum ua haujlwm hauv lub tuam txhab sab qab teb choj, tus neeg hais lus thiab chav kawm tsim Slurm.

Tau qhov twg los: www.hab.com