Nco tseg. txhais.: Peb nthuav qhia rau koj cov lus qhia tshwj xeeb txog cov laj thawj rau lub sijhawm tsis ntev los no hauv huab kev pabcuam tswj hwm los ntawm tus tsim ntawm Grafana. Qhov no yog ib tug classic piv txwv ntawm yuav ua li cas ib tug tshiab thiab zoo li tsis tshua muaj txiaj ntsig feature tsim los txhim kho qhov zoo ntawm infrastructure ... tuaj yeem ua rau muaj kev phom sij yog tias koj tsis muab rau ntau qhov nuances ntawm nws daim ntawv thov hauv qhov tseeb ntawm ntau lawm. Nws zoo heev thaum cov ntaub ntawv zoo li no tshwm sim uas tso cai rau koj kawm tsis yog los ntawm koj qhov yuam kev xwb. Cov ntsiab lus muaj nyob rau hauv kev txhais cov ntawv no los ntawm tus lwm thawj ntawm cov khoom los ntawm Grafana Labs.
Hnub Friday, Lub Xya Hli 19, qhov kev pabcuam Hosted Prometheus hauv Grafana Huab tau nres ua haujlwm li ntawm 30 feeb. Kuv thov txim rau txhua tus neeg siv khoom cuam tshuam los ntawm qhov hluav taws xob. Peb txoj haujlwm yog muab cov cuab yeej saib xyuas koj xav tau, thiab peb nkag siab tias tsis muaj lawv tuaj yeem ua rau koj lub neej nyuaj dua. Peb coj qhov xwm txheej no hnyav heev. Daim ntawv no piav qhia txog qhov tshwm sim, peb teb li cas, thiab peb tab tom ua dab tsi kom ntseeg tau tias nws tsis tshwm sim dua.
prehistory
Grafana Cloud Hosted Prometheus kev pabcuam yog nyob ntawm - CNCF qhov project los tsim kab rov tav scalable, muaj ntau, ntau tus neeg xauj tsev Prometheus. Lub Cortex architecture muaj cov txheej txheem ntawm tus kheej microservices, txhua tus ua nws txoj haujlwm: rov ua dua, khaws cia, lus nug, thiab lwm yam. Cortex nyob rau hauv kev txhim kho nquag thiab niaj hnub ntxiv cov yam ntxwv tshiab thiab txhim kho kev ua haujlwm. Peb niaj hnub siv Cortex tshiab tso tawm rau pawg kom cov neeg siv khoom tuaj yeem ua kom zoo dua ntawm cov yam ntxwv no - hmoov zoo, Cortex tuaj yeem hloov kho yam tsis muaj sijhawm poob.
Rau kev hloov kho tsis tu ncua, Ingester Cortex cov kev pabcuam yuav tsum muaj Ingester replica ntxiv thaum lub sijhawm hloov tshiab. (Nco tseg. txhais.: - Qhov tseem ceeb ntawm Cortex. Nws txoj haujlwm yog los sau cov kwj mus tas li ntawm cov qauv, muab lawv tso rau hauv Prometheus chunks thiab khaws cia rau hauv cov ntaub ntawv xws li DynamoDB, BigTable lossis Cassandra.) Qhov no tso cai rau Ingesters qub xa cov ntaub ntawv tam sim no mus rau Ingesters tshiab. Nws yog ib qho tsim nyog sau cia tias Ingesters yog cov peev txheej xav tau. Rau lawv ua hauj lwm, koj yuav tsum muaj 4 cores thiab 15 GB ntawm lub cim xeeb rau ib pod, i.e. 25% ntawm kev ua lub zog thiab kev nco ntawm lub hauv paus tshuab nyob rau hauv rooj plaub ntawm peb Kubernetes pawg. Feem ntau, peb feem ntau muaj ntau yam khoom siv tsis siv hauv pawg tshaj li 4 cores thiab 15 GB ntawm lub cim xeeb, yog li peb tuaj yeem yooj yim tig cov ingesters ntxiv no thaum hloov kho dua tshiab.
Txawm li cas los xij, feem ntau tshwm sim thaum lub sijhawm ua haujlwm ib txwm tsis muaj lub tshuab muaj 25% ntawm cov khoom siv tsis siv. Yog, peb tsis txawm siv zog: CPU thiab lub cim xeeb yuav ib txwm muaj txiaj ntsig rau lwm cov txheej txheem. Txhawm rau daws qhov teeb meem no, peb txiav txim siab siv . Lub tswv yim yog muab Ingesters qhov tseem ceeb dua li lwm yam (tsis muaj xeev) microservices. Thaum peb yuav tsum tau khiav ib qho ntxiv (N + 1) Ingester, peb hloov chaw ib ntus, me me pods. Cov pods no raug xa mus rau cov peev txheej pub dawb ntawm lwm lub tshuab, tawm hauv qhov "qhov" loj txaus los khiav ib qho Ingester ntxiv.
Hnub Thursday, Lub Xya Hli 18, peb tau nthuav tawm plaub theem tseem ceeb tshiab rau peb pawg: tseem ceeb, siab, nruab nrab ΠΈ tsawg. Lawv tau sim ntawm ib pawg sab hauv uas tsis muaj cov neeg siv khoom siv tau li ib lub lis piam. Los ntawm lub neej ntawd, pods tsis muaj qhov tseem ceeb tau txais nruab nrab qhov tseem ceeb, chav kawm tau teem rau Ingesters nrog siab qhov tseem ceeb. Tseem ceeb tau tshwj tseg rau kev saib xyuas (Prometheus, Alertmanager, node-exporter, kube-state-metrics, thiab lwm yam). Peb config qhib, thiab koj tuaj yeem saib PR .
Kev sib tsoo
Hnub Friday, Lub Xya Hli 19, ib tus kws tshaj lij tau tsim kho tshiab Cortex pawg rau cov neeg siv khoom loj. Lub config rau pawg no tsis suav nrog qhov tseem ceeb ntawm cov pods tshiab, yog li tag nrho cov pods tshiab tau muab qhov tseem ceeb tshaj plaws - nruab nrab.
Cov pab pawg Kubernetes tsis muaj peev txheej txaus rau Cortex pawg tshiab, thiab cov khoom tsim tawm Cortex uas twb muaj lawm tsis tau hloov kho (Ingesters tau tso tseg yam tsis muaj siab qhov tseem ceeb). Txij li thaum Ingesters ntawm pawg tshiab los ntawm lub neej ntawd muaj nruab nrab qhov tseem ceeb, thiab cov pods uas twb muaj lawm hauv kev tsim khoom tau ua haujlwm yam tsis muaj qhov tseem ceeb ntawm txhua qhov, Ingesters ntawm pawg tshiab hloov Ingesters los ntawm pawg Cortex uas twb muaj lawm.
ReplicaSet rau qhov raug ntiab tawm Ingester nyob rau hauv pawg ntau lawm tau kuaj pom cov pods raug tshem tawm thiab tsim ib qho tshiab los tuav cov ntawv teev tseg. Lub pod tshiab tau muab los ntawm lub neej ntawd nruab nrab qhov tseem ceeb, thiab lwm qhov "laus" Ingester hauv kev tsim khoom poob nws cov peev txheej. Qhov tshwm sim yog txheej txheem avalanche, uas ua rau kev hloov pauv ntawm tag nrho cov pods los ntawm Ingester rau Cortex ntau lawm pawg.
Ingesters yog qhov tseem ceeb thiab khaws cov ntaub ntawv rau 12 teev dhau los. Qhov no tso cai rau peb kom compress lawv zoo dua ua ntej sau lawv mus rau lub sijhawm ntev. Txhawm rau ua tiav qhov no, Cortex shards cov ntaub ntawv hla cov koob siv Distributed Hash Table (DHT) thiab rov ua dua txhua qhov hla peb Ingesters siv Dynamo-style pawg sib xws. Cortex tsis sau cov ntaub ntawv rau Ingesters uas tsis taus. Yog li, thaum muaj coob tus Ingesters tawm ntawm DHT, Cortex tsis tuaj yeem muab cov ntaub ntawv rov qab txaus, thiab lawv poob.
Kev tshawb nrhiav thiab kho
Cov ntawv ceeb toom tshiab Prometheus raws li "kev siv nyiaj yuam kev" (yuam kev-pob nyiaj-raws li - cov ntsiab lus yuav tshwm sim hauv tsab xov xwm yav tom ntej) pib suab lub tswb 4 feeb tom qab pib kaw. Tshaj li tsib feeb tom ntej no los yog li ntawd, peb tau khiav qee qhov kev kuaj mob thiab ntsuas lub hauv paus Kubernetes pawg los tuav cov pawg tsim khoom tshiab thiab tam sim no.
Tom qab tsib feeb ntxiv, cov qub Ingesters tau sau lawv cov ntaub ntawv tiav, cov tshiab tau pib, thiab Cortex pawg tau rov muaj dua.
Lwm 10 feeb tau siv tshuaj ntsuam xyuas thiab kho qhov tsis nco qab (OOM) qhov yuam kev los ntawm kev lees paub rov qab cov npe nyob rau pem hauv ntej ntawm Cortex. OOM yuam kev tau tshwm sim los ntawm kev nce kaum npaug ntawm QPS (peb ntseeg tias vim muaj kev thov nruj dhau los ntawm tus neeg siv khoom Prometheus servers).
Qhov yuav tshwm sim
Tag nrho downtime yog 26 feeb. Tsis muaj ntaub ntawv poob lawm. Ingesters tau ua tiav tag nrho cov ntaub ntawv hauv lub cim xeeb rau hauv kev cia ntev. Thaum lub sijhawm kaw, tus neeg siv khoom Prometheus servers buffered deleted (nyob deb) cov ntaub ntawv siv raws li WAL (sau los ntawm los ntawm Grafana Labs) thiab rov sau cov ntawv ua tsis tiav tom qab kev sib tsoo.
Kev tsim khoom pawg sau ua haujlwm
tshawb pom
Nws yog ib qho tseem ceeb kom kawm los ntawm qhov xwm txheej no thiab ua cov kauj ruam tsim nyog kom tsis txhob rov tshwm sim dua.
Nyob rau hauv hindsight, peb yuav tsum tsis tau teem lub neej ntawd nruab nrab qhov tseem ceeb kom txog thaum tag nrho Ingesters hauv ntau lawm tau txais siab qhov tseem ceeb. Tsis tas li ntawd, nws yuav tsum tau saib xyuas lawv ua ntej siab qhov tseem ceeb. Txhua yam yog kho tam sim no. Peb cia siab tias peb qhov kev paub dhau los yuav pab tau lwm lub koom haum xav txog kev siv cov pods tseem ceeb hauv Kubernetes.
Peb yuav ntxiv ib theem ntxiv ntawm kev tswj hwm kev xa tawm ntawm ib qho khoom ntxiv uas nws cov kev teeb tsa yog thoob ntiaj teb rau pawg. Txij no mus, cov kev hloov no yuav raug ntsuas bΠΎntau tus neeg. Tsis tas li ntawd, qhov kev hloov kho uas ua rau kev sib tsoo tau suav tias yog me me rau cov ntaub ntawv sib cais - nws tsuas yog tham hauv GitHub qhov teeb meem. Txij ntawm no mus, tag nrho cov kev hloov pauv rau kev teeb tsa yuav raug nrog rau cov ntaub ntawv tsim nyog.
Thaum kawg, peb yuav automate resizing ntawm authentication reverse proxy los tiv thaiv lub overload OOM peb pom, thiab yuav tshuaj xyuas Prometheus default settings ntsig txog kev poob rov qab thiab scaling los tiv thaiv cov teeb meem zoo sib xws yav tom ntej.
Qhov tsis ua hauj lwm kuj muaj qee qhov txiaj ntsig zoo: tau txais cov peev txheej tsim nyog, Cortex tau txais txiaj ntsig zoo yam tsis muaj kev cuam tshuam ntxiv. Peb kuj tau txais txiaj ntsig kev ua haujlwm nrog - peb lub cav sib sau ua ke tshiab - uas tau pab xyuas kom txhua tus Ingesters coj tus cwj pwm zoo thaum lub sijhawm thiab tom qab tsis ua haujlwm.
PS los ntawm tus txhais lus
Nyeem kuj ntawm peb blog:
- «»;
- «»;
- «»;
- Β«".
Tau qhov twg los: www.hab.com