{"id":36706,"date":"2019-10-31T22:13:15","date_gmt":"2019-10-31T19:13:15","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs\/"},"modified":"2019-10-31T22:13:15","modified_gmt":"2019-10-31T19:13:15","slug":"kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs","title":{"rendered":"Si prioritetet e pod'\u00ebve n\u00eb Kubernetes shkaktuan ndales\u00eb n\u00eb Grafana Labs","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Sh\u00ebn. p\u00ebrkth.<\/b>: Ju sjellim p\u00ebr ju detajet teknike mbi arsyet e fundit t\u00eb ndaljes n\u00eb funksionimin e sh\u00ebrbimit cloud, t\u00eb mbajtur nga krijuesit e Grafana. Ky \u00ebsht\u00eb nj\u00eb shembull klasik se si nj\u00eb mund\u00ebsi e re, duksh\u00ebm e dobishme, e krijuar p\u00ebr t\u00eb p\u00ebrmir\u00ebsuar cil\u00ebsin\u00eb e infrastruktur\u00ebs... mund t\u00eb d\u00ebmtoj\u00eb, n\u00ebse nuk parashikohet numri i madh i nuancave t\u00eb aplikimit t\u00eb saj n\u00eb realitetin e prodhimit. \u00cbsht\u00eb e shk\u00eblqyer kur shfaqen materiale t\u00eb tilla, q\u00eb lejojn\u00eb t\u00eb m\u00ebsojm\u00eb jo vet\u00ebm nga gabimet tona. Detajet jan\u00eb n\u00eb p\u00ebrkthimin e k\u00ebtij teksti nga n\u00ebn-presidenti p\u00ebr produktet n\u00eb Grafana Labs.<\/i><\/p>\n

\"Si
\n
\nN\u00eb t\u00eb premten, 19 korrik, sh\u00ebrbimi Hosted Prometheus n\u00eb Grafana Cloud ndaloi t\u00eb funksiononte p\u00ebr rreth 30 minuta. M\u00eb vjen keq p\u00ebr t\u00eb gjith\u00eb klient\u00ebt q\u00eb u preken nga ky nd\u00ebrprerje. Detyra jon\u00eb \u00ebsht\u00eb t\u00eb ofrojm\u00eb mjetet e duhura p\u00ebr monitorim, dhe ne e kuptojm\u00eb se mosdisponueshm\u00ebria e tyre e komplikon jet\u00ebn tuaj. Ne e marrim shum\u00eb seriozisht k\u00ebt\u00eb incident. N\u00eb k\u00ebt\u00eb sh\u00ebnim shpjegohet se \u00e7far\u00eb ndodhi, si reaguam dhe \u00e7far\u00eb po b\u00ebjm\u00eb q\u00eb kjo t\u00eb mos ndodh\u00eb m\u00eb.<\/a><\/noindex><\/p>\n

Historia e m\u00ebparshme<\/h2>\n

\nSh\u00ebrbimi Grafana Cloud Hosted Prometheus bazohet n\u00eb Cortex<\/a><\/noindex> \u2014 projektin CNCF p\u00ebr krijimin e nj\u00eb sh\u00ebrbimi Prometheus me shkall\u00ebzim horizontal, me disponueshm\u00ebri t\u00eb lart\u00eb dhe shum\u00eb-qiramarr\u00ebs (multi-tenant). Arkitektura Cortex p\u00ebrb\u00ebhet nga nj\u00eb grup mikrosh\u00ebrbimesh, secili prej t\u00eb cil\u00ebve kryen funksionin e tij: riplikimi, ruajtja, k\u00ebrkesat etj. Cortex \u00ebsht\u00eb n\u00eb zhvillim aktiv, vazhdimisht po shfaqen mund\u00ebsi t\u00eb reja dhe rritet performanca. Ne rregullisht nxjerrim versione t\u00eb reja t\u00eb Cortex n\u00eb klaster\u00eb, n\u00eb m\u00ebnyr\u00eb q\u00eb klient\u00ebt t\u00eb mund t\u00eb p\u00ebrfitojn\u00eb nga k\u00ebto mund\u00ebsi \u2014 fatmir\u00ebsisht, Cortex di t\u00eb p\u00ebrdit\u00ebsohet pa nd\u00ebrprerje.<\/p>\n

P\u00ebr p\u00ebrdit\u00ebsime pa paqt\u00eb, sh\u00ebrbimi Ingester i Cortex k\u00ebrkon nj\u00eb replik\u00eb shtes\u00eb t\u00eb Ingester gjat\u00eb procesit t\u00eb p\u00ebrdit\u00ebsimit. (Sh\u00ebn. p\u00ebrkth.<\/b>: Ingester<\/a><\/noindex> \u2014 komponenti baz\u00eb i Cortex. Detyra e tij \u00ebsht\u00eb t\u00eb mbledh\u00eb nj\u00eb rrjedh\u00eb t\u00eb vazhdueshme sample, t'i grupoj\u00eb ato n\u00eb chunks p\u00ebr Prometheus dhe t'i ruaj\u00eb n\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash si DynamoDB, BigTable ose Cassandra.)<\/i> Kjo i lejon Ingester-\u00ebt e vjet\u00ebr t\u00eb d\u00ebrgojn\u00eb t\u00eb dh\u00ebnat aktuale te Ingester-\u00ebt e rinj. Vlen t\u00eb p\u00ebrmendet se Ingester-\u00ebt k\u00ebrkojn\u00eb burime t\u00eb larta. P\u00ebr t\u00eb funksionuar, secili ka nevoj\u00eb p\u00ebr 4 b\u00ebrthama dhe 15 GB memorie n\u00eb pod, q\u00eb do t\u00eb thot\u00eb 25% e kapacitetit t\u00eb procesorit dhe memories t\u00eb makin\u00ebs baz\u00eb n\u00eb rastin e klastrave tan\u00eb Kubernetes. N\u00eb p\u00ebrgjith\u00ebsi, ne zakonisht kemi shum\u00eb m\u00eb tep\u00ebr burime t\u00eb pap\u00ebrdorura n\u00eb klaster sesa 4 b\u00ebrthama dhe 15 GB memorie, prandaj mund ta ekzekutojm\u00eb leht\u00ebsisht k\u00ebto Ingester shtes\u00eb gjat\u00eb p\u00ebrdit\u00ebsimeve.<\/p>\n

Megjithat\u00eb, shpesh ndodh q\u00eb gjat\u00eb pun\u00ebs normale asnj\u00eb nga makinat nuk ka k\u00ebto 25% burime t\u00eb pap\u00ebrdorura. Po ashtu, ne nuk p\u00ebrpiqemi: CPU dhe memoria gjithmon\u00eb do t\u00eb jen\u00eb t\u00eb nevojshme p\u00ebr procese t\u00eb tjera. P\u00ebr t\u00eb zgjidhur k\u00ebt\u00eb problem, ne vendos\u00ebm t\u00eb p\u00ebrdorim Prioritetet e Kubernetes Pod<\/a><\/noindex>. Ideja \u00ebsht\u00eb t'u jepet Ingester-\u00ebve nj\u00eb prioritet m\u00eb t\u00eb lart\u00eb se mikrosh\u00ebrbimeve t\u00eb tjera (stateless). Kur na nevojitet t\u00eb lan\u00e7ojm\u00eb nj\u00eb Ingester shtes\u00eb (N+1), ne p\u00ebrkoh\u00ebsisht z\u00ebvend\u00ebsojm\u00eb pod-et e tjera, m\u00eb t\u00eb vogla. K\u00ebto pod-e transferohen n\u00eb burime t\u00eb lira n\u00eb makina t\u00eb tjera, duke l\u00ebn\u00eb nj\u00eb \"hap\u00ebsir\u00eb\" t\u00eb mjaftueshme p\u00ebr t\u00eb lan\u00e7uar Ingester-in shtes\u00eb.<\/p>\n

M\u00eb 18 korrik, ne lan\u00e7uam kat\u00ebr nivele t\u00eb reja prioriteti n\u00eb klaster\u00ebt tan\u00eb: kritik<\/b>, nj\u00eb<\/b>, mesatar<\/b> dhe i ul\u00ebt<\/b>. Ato jan\u00eb testuar n\u00eb nj\u00eb klas\u00ebr t\u00eb brendsh\u00ebm pa trafik klient\u00ebsh p\u00ebr rreth nj\u00eb jav\u00eb. Si rregull, pod-et pa prioritet t\u00eb caktuar merrnin mesatar<\/b> prioritet, n\u00ebse p\u00ebr Ingester-\u00ebt ishte caktuar nj\u00eb klas\u00eb me t\u00eb lart\u00eb<\/b> Rregulli 4a: N\u00ebse detyra p\u00ebrdor t\u00eb gjith\u00eb dritaren e saj t\u00eb caktuar kohore, at\u00ebher\u00eb prioriteti i saj Kritik<\/b> ishte rezervuar p\u00ebr monitorimin (Prometheus, Alertmanager, node-exporter, kube-state-metrics etj.). Konfigurat tona jan\u00eb t\u00eb hapura, dhe mund t\u00eb shihni PR-n\u00eb k\u00ebtu<\/a><\/noindex>.<\/p>\n

Aksidente<\/h2>\n

\nM\u00eb 19 korrik, nj\u00eb nga inxhinier\u00ebt lan\u00e7oi nj\u00eb klas\u00ebr t\u00eb ri t\u00eb dedikuar p\u00ebr Cortex p\u00ebr nj\u00eb klient t\u00eb madh. Konfigurimi p\u00ebr k\u00ebt\u00eb klas\u00ebr nuk p\u00ebrfshinte prioritetet e reja t\u00eb pod-eve, k\u00ebshtu q\u00eb t\u00eb gjith\u00eb pod-et e rinj merrnin prioritetin e paracaktuar \u2014 mesatar<\/b>.<\/p>\n

Nuk kishte mjaft burime n\u00eb klastrin Kubernetes p\u00ebr klasrin e ri t\u00eb Cortex, dhe klastri aktual i prodhimit Cortex nuk ishte p\u00ebrdit\u00ebsuar (Ingester-\u00ebt mbet\u00ebn pa t\u00eb lart\u00eb<\/b> prioritet). Duke qen\u00eb se Ingester-\u00ebt e klasrit t\u00eb ri sipas paracaktimit kishin mesatar<\/b> prioritet, nd\u00ebrsa pod-et ekzistuese n\u00eb prodhim funksiononin pa prioritet fare, Ingester-\u00ebt e klasrit t\u00eb ri z\u00ebvend\u00ebsuan Ingester-\u00ebt nga klastri ekzistues i Cortex n\u00eb prodhim.<\/p>\n

ReplicaSet p\u00ebr Ingester-in e z\u00ebvend\u00ebsuar n\u00eb klastrin e prodhimit zbuloi pod-in e z\u00ebvend\u00ebsuar dhe krijoi nj\u00eb t\u00eb ri p\u00ebr t\u00eb mbajtur numrin e caktuar t\u00eb kopjeve. Pod-it t\u00eb ri, sipas paracaktimit, iu caktua mesatar<\/b> prioriteti, dhe Ingester-i \u00abi vjet\u00ebr\u00bb n\u00eb production humbi burimet. Rezultati ishte nj\u00eb proces avullues<\/b>, i cili \u00e7oi n\u00eb z\u00ebvend\u00ebsimin e t\u00eb gjith\u00eb pod-eve me Ingester p\u00ebr klastrat e prodhimit t\u00eb Cortex.<\/p>\n

Ingesterat ruajn\u00eb nj\u00eb gjendje (stateful) dhe mbajn\u00eb t\u00eb dh\u00ebna p\u00ebr 12 or\u00ebt e kaluara. Kjo na lejon t\u00eb kompakt\u00ebsojm\u00eb m\u00eb efektivisht ato p\u00ebrpara se t'i regjistronim n\u00eb ruajtjen afatgjat\u00eb. P\u00ebr k\u00ebt\u00eb, Cortex realizon ndarjen e t\u00eb dh\u00ebnave n\u00eb seri, duke p\u00ebrdorur nj\u00eb tabel\u00eb heshash t\u00eb shp\u00ebrndar\u00eb (Distributed Hash Table, DHT), dhe riprodhon \u00e7do seri n\u00eb tri Ingester-a me konsistenc\u00eb kuorumi n\u00eb stilin Dynamo. Cortex nuk shkruan t\u00eb dh\u00ebna n\u00eb Ingester-at q\u00eb jan\u00eb \u00e7aktivizuar. K\u00ebshtu, kur nj\u00eb num\u00ebr i madh Ingester-ash largohet nga DHT, Cortex nuk mund t\u00eb siguroj\u00eb nj\u00eb riprodhim t\u00eb mjaftuesh\u00ebm t\u00eb regjistrimeve, dhe ato \"bjer\u00eb\".<\/p>\n

Zbulimi dhe eliminimi<\/h2>\n

\nNjoftimet e reja t\u00eb Prometheus q\u00eb bazohen n\u00eb \u201cbuxhetin e gabimeve\u201d (error-budget-based<\/i> \u2014 detajet do t\u00eb shfaqen n\u00eb nj\u00eb artikull t\u00eb ardhsh\u00ebm) filluan t\u00eb japin alarmin pas 4 minutash nga fillimi i fikjes. Gjat\u00eb pes\u00eb minutave t\u00eb ardhshme, ne realizuam diagnostikimin dhe rrit\u00ebm klasterin Kubernetes t\u00eb posht\u00ebm p\u00ebr t\u00eb mbajtur si klasterat production t\u00eb rinj ashtu edhe ata ekzistues.<\/p>\n

Paks minuta m\u00eb von\u00eb, Ingesterat e vjet\u00ebr regjistruan me sukses t\u00eb dh\u00ebnat e tyre, nd\u00ebrsa t\u00eb rinjt\u00eb filluan, dhe klasteret Cortex u b\u00ebn\u00eb p\u00ebrs\u00ebri t\u00eb aksesueshme.<\/p>\n

Nj\u00eb tjet\u00ebr 10 minuta u desh p\u00ebr diagnostikimin dhe rregullimin e gabimeve out-of-memory (OOM) nga server\u00ebt e p\u00ebrparm\u00eb t\u00eb autentikimit q\u00eb ndodheshin p\u00ebrpara Cortex. Gabimet OOM ishin shkaktuar nga rritja dhjet\u00ebfish e QPS (si\u00e7 besojm\u00eb, p\u00ebr shkak t\u00eb k\u00ebrkesave tep\u00ebr agresive nga server\u00ebt Prometheus t\u00eb klient\u00ebve).<\/p>\n

Pasojat<\/h2>\n

\nKoh\u00ebzgjatja totale e ndales\u00ebs ishte 26 minuta. T\u00eb dh\u00ebnat nuk u humb\u00ebn. Ingesterat transferuan me sukses t\u00eb gjitha t\u00eb dh\u00ebnat in-memory n\u00eb ruajtjen afatgjat\u00eb. Gjat\u00eb \u00e7aktivizimit, serverat Prometheus t\u00eb klient\u00ebve ruajt\u00ebn n\u00eb buffer t\u00eb dh\u00ebnat e larguara. (remote)<\/i> p\u00ebrmes API t\u00eb ri remote_write<\/a><\/noindex> t\u00eb bazuar n\u00eb WAL (t\u00eb autor\u00ebsis\u00eb Callum Styan<\/a><\/noindex> nga Grafana Labs) dhe p\u00ebrs\u00ebrit\u00ebn regjistrimet e d\u00ebshtuar pas \u00e7rregullimit.<\/p>\n

\"Si
\nOperacionet e shkruar t\u00eb klasterit production<\/i><\/p>\n

P\u00ebrfundimet<\/h2>\n

\n\u00cbsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb nxirren m\u00ebsime nga ky incident dhe t\u00eb nd\u00ebrmerren veprimet e nevojshme p\u00ebr t\u00eb parandaluar p\u00ebrs\u00ebritjen e tij.<\/p>\n

Duke shikuar prapa, duhet t\u00eb pranojm\u00eb se nuk duhej t\u00eb vendosnim p\u00ebrpar\u00ebsi si parazgjedhje mesatar<\/b> prioritet, derisa t\u00eb gjith\u00eb Ingesterat n\u00eb production t\u00eb merrnin nj\u00eb<\/b> prioritet. P\u00ebr m\u00eb tep\u00ebr, duhej t\u00eb ishim kujdesur p\u00ebr prioritetin e tyre t\u00eb lart\u00eb.<\/b> Tani gjith\u00e7ka \u00ebsht\u00eb rregulluar. Shpresojm\u00eb q\u00eb p\u00ebrvoja jon\u00eb do t\u00eb ndihmoj\u00eb organizatat e tjera q\u00eb po shqyrtojn\u00eb mund\u00ebsin\u00eb e p\u00ebrdorimit t\u00eb prioriteteve t\u00eb pod\u00ebve n\u00eb Kubernetes.<\/p>\n

Ne do t\u00eb shtojm\u00eb nj\u00eb nivel t\u00eb shtuar kontrolli mbi implementimin e \u00e7do objekti shtes\u00eb, konfigurimet e t\u00eb cil\u00ebve jan\u00eb globale p\u00ebr klasterin. N\u00eb t\u00eb ardhmen, ndryshime t\u00eb tilla do t\u00eb vler\u00ebsohen ngao<\/i>nj\u00eb num\u00ebr m\u00eb i madh njer\u00ebzish. P\u00ebr m\u00eb tep\u00ebr, modifikimi q\u00eb \u00e7oi n\u00eb d\u00ebshtim u konsiderua tep\u00ebr i par\u00ebnd\u00ebsish\u00ebm p\u00ebr nj\u00eb dokument projektimi t\u00eb ve\u00e7ant\u00eb \u2014 u diskutua vet\u00ebm n\u00eb GitHub issue. Nga ky moment e tutje, t\u00eb gjitha ndryshimet e tilla t\u00eb konfigurove do t\u00eb shoq\u00ebrohen me dokumentacionin p\u00ebrkat\u00ebs t\u00eb projektit.<\/p>\n

N\u00eb fund, do t\u00eb automatizojm\u00eb rritjen e serverit t\u00eb p\u00ebrpar\u00ebsis\u00eb p\u00ebr t\u00eb parandaluar OOM gjat\u00eb mbingarkes\u00ebs, t\u00eb cil\u00ebt ishim d\u00ebshmitar\u00eb, dhe do t\u00eb analizojm\u00eb parametrat e Prometheus-it n\u00eb parazgjedhje, q\u00eb lidhen me rikthimin dhe dimensionimin, p\u00ebr t\u00eb parandaluar probleme t\u00eb ngjashme n\u00eb t\u00eb ardhmen.<\/p>\n

D\u00ebshtimi q\u00eb kaluam kishte edhe disa pasoja pozitive: duke pasur burimet e nevojshme, Cortex u rikuperua automatikisht pa nd\u00ebrhyrje t\u00eb m\u00ebtejshme. Ne gjithashtu mor\u00ebm p\u00ebrvoj\u00eb t\u00eb vyer n\u00eb pun\u00ebn me Grafana Loki<\/a><\/noindex> \u2014 sistemi yn\u00eb i ri i agregimit t\u00eb logeve, \u2014 i cili ndihmoi t\u00eb sigurohemi q\u00eb t\u00eb gjith\u00eb Ingesterat t\u00eb silleshin si\u00e7 duhet gjat\u00eb dhe pas nd\u00ebrprerjes.<\/p>\n

P.S. nga p\u00ebrkthyesi<\/h2>\n

\nLexoni gjithashtu n\u00eb blogun ton\u00eb:<\/p>\n