Zindikirani. transl.: Tikukudziwitsani zambiri zazomwe zachititsa kuti mtambo usamayende bwino ndi omwe amapanga Grafana. Ichi ndi chitsanzo chodziwika bwino cha momwe chinthu chatsopano komanso chowoneka ngati chothandiza kwambiri chopangidwira kukonza magwiridwe antchito ... chingayambitse vuto ngati simupereka zambiri zamagwiritsidwe ake pazomwe zimapangidwira. Ndibwino kuti zinthu ngati izi ziwoneka zomwe zimakupatsani mwayi wophunzirira osati pazolakwa zanu zokha. Tsatanetsatane ndi kumasulira kwa mawuwa kuchokera kwa wachiwiri kwa purezidenti wazogulitsa ku Grafana Labs.
Lachisanu, Julayi 19, ntchito yaHost Prometheus ku Grafana Cloud idasiya kugwira ntchito kwa mphindi pafupifupi 30. Pepani kwa makasitomala onse omwe akhudzidwa ndi kuzimitsa. Ntchito yathu ndikupereka zida zowunikira zomwe mukufuna, ndipo tikumvetsetsa kuti kusakhala nazo kungapangitse moyo wanu kukhala wovuta. Timaona chochitikachi mozama kwambiri. Cholembachi chikufotokoza zomwe zidachitika, momwe tidayankhira, ndi zomwe tikuchita kuti zisachitikenso.
prehistory
Grafana Cloud host Prometheus service idakhazikitsidwa - Pulojekiti ya CNCF yopangira ntchito ya Prometheus yokhazikika, yopezeka kwambiri, yokhala ndi anthu ambiri. Zomangamanga za Cortex zimakhala ndi ma microservices angapo, iliyonse imagwira ntchito yake: kubwereza, kusungirako, kufunsa, ndi zina. Cortex ikukula mwachangu ndipo imangowonjezera zatsopano ndikuwongolera magwiridwe antchito. Nthawi zonse timatumiza zatsopano za Cortex m'magulu kuti makasitomala athe kutengapo mwayi pazinthu izi - mwamwayi, Cortex ikhoza kusinthidwa popanda kutsika.
Pazosintha zopanda msoko, ntchito ya Ingester Cortex imafuna chowonjezera cha Ingester panthawi yokonzanso. (Zindikirani. transl.: - chigawo choyambirira cha Cortex. Ntchito yake ndikutolera zitsanzo zingapo, kuziyika m'magulu a Prometheus ndikuzisunga mu database ngati DynamoDB, BigTable kapena Cassandra.) Izi zimalola Ingesters akale kutumiza deta yamakono ku Ingesters yatsopano. Ndizofunikira kudziwa kuti Ingesters ndiyofunika kwambiri. Kuti agwire ntchito, muyenera kukhala ndi ma cores 4 ndi 15 GB ya kukumbukira pa pod, i.e. 25% ya mphamvu yopangira ndi kukumbukira makina oyambira pamagulu athu a Kubernetes. Nthawi zambiri, nthawi zambiri timakhala ndi zinthu zambiri zosagwiritsidwa ntchito pagulu kuposa ma cores 4 ndi 15 GB ya kukumbukira, kotero titha kusuntha mosavuta izi powonjezera.
Komabe, nthawi zambiri zimachitika kuti pakugwira ntchito bwino palibe makina omwe ali ndi 25% yazinthu zosagwiritsidwa ntchito. Inde, sitiyesetsa ngakhale pang'ono: CPU ndi kukumbukira zidzakhala zothandiza pazinthu zina. Kuti tithetse vutoli, tinaganiza zogwiritsa ntchito . Lingaliro ndikupatsa Ingesters patsogolo kwambiri kuposa ma microservices ena (osawerengeka). Tikafuna kuyendetsa chowonjezera (N+1) Ingester, timachotsa kwakanthawi ma pod ena ang'onoang'ono. Ma pod awa amasamutsidwa kuzinthu zaulere pamakina ena, ndikusiya "dzenje" lalikulu lokwanira kuyendetsa Ingester yowonjezera.
Lachinayi, pa Julayi 18, tidatulutsa magawo anayi atsopano kumagulu athu: chotsutsa, mkulu, pafupifupi ΠΈ otsika. Iwo adayesedwa pagulu lamkati lopanda makasitomala kwa pafupifupi sabata imodzi. Mwachikhazikitso, ma pods opanda choyambirira chinalandiridwa pafupifupi choyambirira, kalasi idakhazikitsidwa kwa Ingesters ndi mkulu chofunika kwambiri. Zovuta idasungidwa kuti iwunikire (Prometheus, Alertmanager, node-exporter, kube-state-metrics, etc.). Kusintha kwathu ndi kotseguka, ndipo mutha kuwona PR .
Kuwonongeka
Lachisanu, Julayi 19, m'modzi mwa mainjiniya adayambitsa gulu latsopano la Cortex kwa kasitomala wamkulu. Kukonzekera kwa gululi sikunaphatikizepo zofunikira zatsopano za pod, kotero ma pod onse atsopano adapatsidwa mwayi wokhazikika - pafupifupi.
Gulu la Kubernetes linalibe zida zokwanira za gulu latsopano la Cortex, ndipo gulu lomwe linalipo la Cortex silinasinthidwe (Ingesters idasiyidwa popanda apamwamba patsogolo). Popeza Ingesters ya gulu latsopano mwachisawawa anali pafupifupi Chofunika kwambiri, ndipo ma pod omwe analipo popanga adagwira ntchito popanda kufunikira konse, a Ingesters a gulu latsopanolo adalowa m'malo mwa Ingesters kuchokera kugulu lakupanga la Cortex.
ReplicaSet ya Ingester yemwe adathamangitsidwa mgulu lopanga adazindikira poto wothamangitsidwa ndikupanga ina yatsopano kuti asunge kuchuluka kwa makope. Pod yatsopano idaperekedwa mwachisawawa pafupifupi chofunika kwambiri, ndipo Ingester wina "wakale" pakupanga adataya chuma chake. Zotsatira zake zinali ndondomeko ya chigumukire, zomwe zidapangitsa kuti ma pod onse achoke ku Ingester kwamagulu opanga Cortex.
Ingesters ndi yodalirika ndipo imasunga zidziwitso zamaola 12 apitawa. Izi zimatipangitsa kuti tiziwapanikiza bwino tisanawalembe kuti asungidwe kwa nthawi yayitali. Kuti akwaniritse izi, Cortex amadula zidziwitso pamndandanda pogwiritsa ntchito Distributed Hash Table (DHT) ndikubwereza mndandanda uliwonse pama Ingesters atatu pogwiritsa ntchito mawonekedwe a Dynamo quorum kusasinthika. Cortex silemba deta ku Ingesters yomwe ili yolemala. Chifukwa chake, ambiri a Ingesters akachoka ku DHT, Cortex sangathe kupereka kubwereza kokwanira kwa zolembazo, ndipo zimawonongeka.
Kuzindikira ndi Kukonzanso
Zidziwitso Zatsopano za Prometheus zochokera pa "bajeti yolakwika" (zolakwika-zotengera bajeti - zambiri zidzawonekera m'nkhani yamtsogolo) adayamba kulira mphindi 4 pambuyo poyambira kutseka. Kwa mphindi zisanu kapena kupitilira apo, tidachita zowunikira ndikukulitsa gulu la Kubernetes kuti tipeze magulu atsopano komanso omwe alipo.
Pambuyo pa mphindi zisanu, a Ingesters akale adalemba bwino deta yawo, atsopanowo adayamba, ndipo magulu a Cortex adapezekanso.
Mphindi zina za 10 zidagwiritsidwa ntchito pozindikira ndi kukonza zolakwika zakunja kwa memory (OOM) kuchokera ku ma proxies otsimikizira omwe ali kutsogolo kwa Cortex. Zolakwa za OOM zidayamba chifukwa cha kuchuluka kwa QPS kakhumi (tikukhulupirira chifukwa chopempha mwaukali kwambiri kuchokera kwa makasitomala a Prometheus).
Zotsatira
Nthawi yonse yopuma inali mphindi 26. Palibe deta yomwe yatayika. Ingesters adatsitsa bwino zonse zomwe zili mumemory posungira nthawi yayitali. Panthawi yotseka, ma seva a kasitomala a Prometheus adachotsedwa (kutali) kujambula pogwiritsa ntchito kutengera WAL (yolembedwa ndi kuchokera ku Grafana Labs) ndikubwereza zolemba zomwe zidalephera pambuyo pa ngoziyi.
Zolemba zamagulu opangira
anapezazo
Ndikofunika kuphunzira kuchokera ku chochitika ichi ndikuchitapo kanthu kuti mupewe kubwereza.
Poyang'ana m'mbuyo, sitinayenera kuyika zokhazikika pafupifupi patsogolo mpaka ma Ingesters onse opanga atalandira mkulu chofunika kwambiri. Komanso, kunali koyenera kuwasamalira pasadakhale mkulu chofunika kwambiri. Zonse zakonzedwa tsopano. Tikukhulupirira kuti zomwe takumana nazo zithandiza mabungwe ena poganizira kugwiritsa ntchito zoyambira ku Kubernetes.
Tidzawonjezera gawo lowonjezera laulamuliro pa kutumizidwa kwa zinthu zina zowonjezera zomwe masinthidwe ake ali padziko lonse lapansi kumagulu. Kuyambira pano, zosintha zotere zidzawunikidwa bΠΎanthu ambiri. Kuphatikiza apo, kusinthidwa komwe kudayambitsa ngoziku kunkawonedwa ngati kochepa kwambiri kuti pakhale chikalata chosiyana cha polojekiti - kudangokambidwa munkhani ya GitHub. Kuyambira pano, zosintha zonse zotere zimayendera limodzi ndi zolemba zoyenera za polojekiti.
Pomaliza, tidzasinthanso makulidwe a projekiti yotsimikizira kuti tipewe kuchulukira kwa OOM komwe tidawona, ndikuwunikanso zosintha za Prometheus zokhudzana ndi kubweza ndi makulitsidwe kuti tipewe zovuta zofananira mtsogolo.
Kulepheraku kunalinso ndi zotsatira zabwino: atalandira zofunikira, Cortex adachira popanda kulowererapo kwina. Tinaphunziranso zinthu zofunika kwambiri pogwira ntchito - dongosolo lathu latsopano lophatikiza zipika - lomwe lathandizira kuwonetsetsa kuti ma Ingesters onse amachita bwino panthawi komanso atalephera.
PS kuchokera kwa womasulira
Werenganinso pa blog yathu:
- Β«";
- Β«";
- Β«";
- Β«".
Source: www.habr.com