{"id":36706,"date":"2019-10-31T22:13:15","date_gmt":"2019-10-31T19:13:15","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs\/"},"modified":"2019-10-31T22:13:15","modified_gmt":"2019-10-31T19:13:15","slug":"kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-prioritety-podov-v-kubernetes-stali-prichinoj-prostoya-v-grafana-labs","title":{"rendered":"Si prioritetet e pod\u00ebve n\u00eb Kubernetes shkaktuan nd\u00ebrprerje n\u00eb Grafana Labs","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Sh\u00ebn. p\u00ebrk.<\/b>: Po prezantoni detajet teknike mbi shkaqet e fundit t\u00eb nd\u00ebrprerjes s\u00eb sh\u00ebrbimit t\u00eb cloud, i sh\u00ebrbyer nga krijuesit e Grafana. Ky \u00ebsht\u00eb nj\u00eb shembull klasik se si nj\u00eb mund\u00ebsi e re dhe, p\u00ebr sa duket, tejet e dobishme, e krijuar p\u00ebr t\u00eb p\u00ebrmir\u00ebsuar cil\u00ebsin\u00eb e infrastruktur\u00ebs... mund t\u00eb d\u00ebmtoj\u00eb, n\u00ebse nuk merren parasysh nuancat e shumta t\u00eb p\u00ebrdorimit t\u00eb saj n\u00eb realitetin e produksionit. K\u00ebto materiale jan\u00eb t\u00eb shk\u00eblqyera, duke na lejuar t\u00eb m\u00ebsojm\u00eb jo vet\u00ebm nga gabimet tona. Detajet jan\u00eb n\u00eb p\u00ebrkthimin e k\u00ebtij teksti nga zv.presidenti p\u00ebr produktin nga Grafana Labs.<\/i><\/p>\n

\"Si
\n
\nN\u00eb t\u00eb premten, m\u00eb 19 korrik, sh\u00ebrbimi Hosted Prometheus n\u00eb Grafana Cloud ndaloi funksionimin p\u00ebr rreth 30 minuta. K\u00ebrkoj ndjes\u00eb nga t\u00eb gjith\u00eb klient\u00ebt e prekur nga kjo nd\u00ebrprerje. Detyra jon\u00eb \u00ebsht\u00eb t\u00eb ofrojm\u00eb mjetet e nevojshme p\u00ebr monitorim, dhe ne kuptojm\u00eb se mungesa e tyre e b\u00ebn jet\u00ebn tuaj m\u00eb t\u00eb v\u00ebshtir\u00eb. Ne i marrim shum\u00eb seriozisht k\u00ebto incidente. N\u00eb k\u00ebt\u00eb sh\u00ebnim shpjegohet se \u00e7far\u00eb ndodhi, si reaguam dhe \u00e7far\u00eb po b\u00ebjm\u00eb p\u00ebr t\u00eb siguruar q\u00eb nj\u00eb gj\u00eb e till\u00eb t\u00eb mos ndodh\u00eb m\u00eb.<\/a><\/noindex><\/p>\n

Pas historia<\/h2>\n

\nSh\u00ebrbimi Grafana Cloud Hosted Prometheus \u00ebsht\u00eb i bazuar n\u00eb Cortex<\/a><\/noindex> \u2014 projektin CNCF p\u00ebr krijimin e nj\u00eb sh\u00ebrbimi Prometheus horizontalisht t\u00eb shkall\u00ebzuar, me disponueshm\u00ebri t\u00eb lart\u00eb dhe multitenant. Arsyetimi i Cortex-it p\u00ebrb\u00ebhet nga nj\u00eb grup mikrosh\u00ebrbesh t\u00eb ndara, secili nga t\u00eb cilat kryen funksionin e tij: replikimin, ruajtjen, k\u00ebrkesat etj. Cortex po zhvillohet aktivisht, me mund\u00ebsi t\u00eb reja dhe rritje t\u00eb performanc\u00ebs q\u00eb shfaqen vazhdimisht. Ne rregullisht b\u00ebnim zhvillime t\u00eb reja t\u00eb Cortex n\u00eb klastere, n\u00eb m\u00ebnyr\u00eb q\u00eb klient\u00ebt t\u00eb mund t\u00eb p\u00ebrfitojn\u00eb nga k\u00ebto mund\u00ebsi \u2014 fatkeq\u00ebsisht, Cortex \u00ebsht\u00eb n\u00eb gjendje t\u00eb p\u00ebrdit\u00ebsohet pa nd\u00ebrprerje.<\/p>\n

P\u00ebr p\u00ebrdit\u00ebsime pa shk\u00ebputje, sh\u00ebrbimi Ingester Cortex k\u00ebrkon nj\u00eb replik\u00eb shtes\u00eb t\u00eb Ingester gjat\u00eb procesit t\u00eb p\u00ebrdit\u00ebsimit. (Sh\u00ebn. p\u00ebrk.<\/b>: Ingester<\/a><\/noindex> \u2014 komponenti baz\u00eb i Cortex. Detyra e tij \u00ebsht\u00eb t\u00eb mbledh\u00eb nj\u00eb rrjedh\u00eb t\u00eb vazhdueshme sample-esh, t'i grupoj\u00eb ato n\u00eb chunks p\u00ebr Prometheus dhe t'i ruaj\u00eb n\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash si DynamoDB, BigTable ose Cassandra.)<\/i> Kjo lejon Ingester-\u00ebt e vjet\u00ebr t\u00eb d\u00ebrgojn\u00eb t\u00eb dh\u00ebnat aktuale te Ingester-\u00ebt e rinj. Duhet t\u00eb theksohet se Ingester-\u00ebt jan\u00eb k\u00ebrkues ndaj burimeve. P\u00ebr funksionimin e tyre nevojiten 4 b\u00ebrthama dhe 15 GB memorie p\u00ebr pod, q\u00eb do t\u00eb thot\u00eb 25% e fuqis\u00eb procesorike dhe memories s\u00eb makin\u00ebs baz\u00eb n\u00eb rastin e klaster\u00ebve tan\u00eb Kubernetes. N\u00eb p\u00ebrgjith\u00ebsi, ne zakonisht kemi shum\u00eb m\u00eb tep\u00ebr burime t\u00eb pa p\u00ebrdorura n\u00eb klaster sesa 4 b\u00ebrthama dhe 15 GB memorie, prandaj mund t'i aktivizojm\u00eb leht\u00ebsisht k\u00ebta Ingester\u00eb shtes\u00eb gjat\u00eb p\u00ebrdit\u00ebsimeve.<\/p>\n

Megjithat\u00eb, shpesh ndodh q\u00eb gjat\u00eb pun\u00ebs normale, asnj\u00eb nga makinat nuk ka k\u00ebto 25% burime t\u00eb padobishme. Ne anche nuk jemi t\u00eb interesuar p\u00ebr to: CPU dhe memoria do t\u00eb jen\u00eb gjithmon\u00eb t\u00eb nevojshme p\u00ebr procese t\u00eb tjera. P\u00ebr t\u00eb zgjidhur k\u00ebt\u00eb problem, ne vendos\u00ebm t\u00eb shfryt\u00ebzojm\u00eb Kubernetes Pod Priorities<\/a><\/noindex>. Ideja \u00ebsht\u00eb t'u jepet Ingester-\u00ebve prioritet m\u00eb t\u00eb lart\u00eb se mikrosherbimeve (stateless) t\u00eb tjera. Kur na nevojitet t\u00eb aktivizojm\u00eb nj\u00eb Ingester shtes\u00eb (N+1), p\u00ebrkoh\u00ebsisht z\u00ebvend\u00ebsojm\u00eb pod-\u00ebt e tjer\u00eb m\u00eb t\u00eb vegj\u00ebl. K\u00ebta pod prishen n\u00eb burime t\u00eb lira n\u00eb makina t\u00eb tjera, duke l\u00ebn\u00eb nj\u00eb \"hapje\" t\u00eb mjaftueshme p\u00ebr t\u00eb aktivizuar nj\u00eb Ingester shtes\u00eb.<\/p>\n

N\u00eb t\u00eb enjten, m\u00eb 18 korrik, ne vendos\u00ebm kat\u00ebr nivele t\u00eb reja prioritetesh n\u00eb klasteret tona: kritik<\/b>, niveli<\/b>, mesatar<\/b> dhe i ul\u00ebt<\/b>. Ato jan\u00eb testuar n\u00eb klasterin internal pa trafik nga klient\u00ebt p\u00ebr rreth nj\u00eb jav\u00eb. N\u00eb m\u00ebnyr\u00eb t\u00eb parazgjedhur, pod-\u00ebt pa prioritet t\u00eb caktuar merrnin mesatar<\/b> prioritet, p\u00ebr Ingester-\u00ebt u vendos nj\u00eb klas\u00eb me t\u00eb lart\u00eb<\/b> prioritet. Kritik<\/b> u rezervua p\u00ebr monitorimin (Prometheus, Alertmanager, node-exporter, kube-state-metrics etj.). Konfigurimi yn\u00eb \u00ebsht\u00eb i hapur, dhe mund t\u00eb shihni PR-n\u00eb k\u00ebtu<\/a><\/noindex>.<\/p>\n

D\u00ebshtimi<\/h2>\n

\nN\u00eb t\u00eb premten, 19 korrik, nj\u00eb nga inxhinier\u00ebt aktivizoi nj\u00eb klaster t\u00eb ri t\u00eb dedikuar Cortex p\u00ebr nj\u00eb klient t\u00eb madh. Konfigurimi p\u00ebr k\u00ebt\u00eb klaster nuk p\u00ebrfshinte prioritetet e reja p\u00ebr pod-\u00ebt, k\u00ebshtu q\u00eb t\u00eb gjith\u00eb pod-\u00ebt e rinj merrnin prioritetin e parazgjedhur \u2014 mesatar<\/b>.<\/p>\n

N\u00eb klasterin Kubernetes nuk kishte burime t\u00eb mjaftueshme p\u00ebr klasterin e ri Cortex, dhe klasteri aktual i prodhimit Cortex nuk ishte p\u00ebrdit\u00ebsuar (Ingester-\u00ebt mbet\u00ebn pa t\u00eb lart\u00eb<\/b> prioritet). Pasi Ingester-\u00ebt e klasterit t\u00eb ri n\u00eb m\u00ebnyr\u00eb t\u00eb parazgjedhur kishin mesatar<\/b> prioritet, nd\u00ebrsa pod-\u00ebt ekzistues n\u00eb prodhim funksiononin pa prioritet fare, Ingester-\u00ebt e klasterit t\u00eb ri p\u00ebrzgjodh\u00ebn Ingester-\u00ebt nga klasteri ekzistues Cortex t\u00eb prodhimit.<\/p>\n

ReplicaSet p\u00ebr Ingester-in e p\u00ebrz\u00ebn\u00eb n\u00eb klasterin e prodhimit zbuloi pod-in e p\u00ebrz\u00ebn\u00eb dhe krijoi nj\u00eb t\u00eb ri p\u00ebr t\u00eb ruajtur numrin e copave t\u00eb p\u00ebrcaktuar. Pod-i t\u00eb ri automatikisht mori mesatar<\/b> prioritet, dhe nj\u00eb 'i vjet\u00ebr' Ingester n\u00eb produksion humbi burimet. Rezultati ishte nj\u00eb proces kaskad\u00eb<\/b>, q\u00eb \u00e7oi n\u00eb p\u00ebrzjehjen e t\u00eb gjith\u00eb pod-\u00ebve me Ingester p\u00ebr klasteret e prodhimit Cortex.<\/p>\n

Ingester-\u00ebt ruajn\u00eb gjendjen (stateful) dhe ruajn\u00eb t\u00eb dh\u00ebnat p\u00ebr 12 or\u00ebt e m\u00ebparshme. Kjo na lejon t\u00eb kompresojm\u00eb ato m\u00eb efikasht para se t'i ruajm\u00eb n\u00eb ruajtjen afatgjat\u00eb. P\u00ebr k\u00ebt\u00eb, Cortex b\u00ebn sharding t\u00eb t\u00eb dh\u00ebnave sipas serive, duke p\u00ebrdorur nj\u00eb tabel\u00eb hash t\u00eb shp\u00ebrndar\u00eb (Distributed Hash Table, DHT), dhe riprodhon \u00e7do seri te tre Ingester-\u00eb me an\u00eb t\u00eb konsensusit t\u00eb shumic\u00ebs n\u00eb stilin Dynamo. Cortex nuk shkruan t\u00eb dh\u00ebna n\u00eb Ingester-\u00ebt q\u00eb ndalojn\u00eb. Pra, kur nj\u00eb num\u00ebr i madh Ingester-\u00ebsh largohet nga DHT, Cortex nuk mund t\u00eb ofroj\u00eb mjaftuesh\u00ebm replikim t\u00eb regjistrimeve, dhe ato \"bien\".<\/p>\n

Zbulimi dhe zgjidhja<\/h2>\n

\nNjoftimet e reja t\u00eb Prometheus bazuar n\u00eb 'buxhetin e gabimeve' (error-budget-based<\/i> \u2014 detajet do t\u00eb shfaqen n\u00eb nj\u00eb artikull t\u00eb ardhsh\u00ebm) filluan t\u00eb alarmojn\u00eb pas 4 minutash nga fillimi i \u00e7aktivizimit. Gjat\u00eb pes\u00eb minutave t\u00eb ardhshme, ne b\u00ebm\u00eb diagnostikim dhe rrit\u00ebm klasterin e Kubernetes p\u00ebr t\u00eb akomoduar si klaster\u00ebt e ri, ashtu edhe ata ekzistues t\u00eb production-it.<\/p>\n

Disa minuta pas k\u00ebsaj, Ingester-\u00ebt e vjet\u00ebr shkruan me sukses t\u00eb dh\u00ebnat e tyre, nd\u00ebrsa ato t\u00eb reja jan\u00eb aktivizuar, dhe klasteret Cortex b\u00ebhen s\u00ebrish t\u00eb disponueshme.<\/p>\n

P\u00ebrs\u00ebri 10 minuta kaluan p\u00ebr diagnostikimin dhe zgjidhjen e gabimeve out-of-memory (OOM) nga server\u00ebt e prapavij\u00ebs s\u00eb autentifikimit, t\u00eb vendosur para Cortex. Gabimet OOM u shkaktuan nga rritja dhjet\u00ebfish t\u00eb QPS (si\u00e7 mendojm\u00eb, p\u00ebr shkak t\u00eb k\u00ebrkesave tep\u00ebr agresive nga server\u00ebt Prometheus t\u00eb klient\u00ebve).<\/p>\n

Pasojat<\/h2>\n

\nKoha totale e nd\u00ebrprerjes ishte 26 minuta. T\u00eb dh\u00ebnat nuk u humb\u00ebn. Ingester-\u00ebt shkarkuan me sukses t\u00eb gjitha t\u00eb dh\u00ebnat n\u00eb kujtes\u00eb n\u00eb ruajtjen afatgjat\u00eb. Gjat\u00eb nd\u00ebrprerjes, server\u00ebt Prometheus t\u00eb klient\u00ebve ruajt\u00ebn n\u00eb tampon t\u00eb dh\u00ebnat e larguara (remote)<\/i> me an\u00eb t\u00eb API-t t\u00eb ri remote_write<\/a><\/noindex> t\u00eb bazuar n\u00eb WAL (t\u00eb autorit Callum Styan<\/a><\/noindex> nga Grafana Labs) dhe rip\u00ebrs\u00ebrit\u00ebn regjistrimet e d\u00ebshtuara pas d\u00ebshtimit.<\/p>\n

\"Si
\nOperacionet e regjistrimit t\u00eb klasterit t\u00eb production-it<\/i><\/p>\n

P\u00ebrfundimet<\/h2>\n

\n\u00cbsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb nxjerrim m\u00ebsime nga ky incident dhe t\u00eb marrim masat e nevojshme p\u00ebr t\u00eb shmangur p\u00ebrs\u00ebritjen e tij.<\/p>\n

Duke u kthyer mbrapa, duhet t\u00eb pranojm\u00eb se nuk duhej t\u00eb vendosnim me default mesatar<\/b> prioritet, derisa t\u00eb gjith\u00eb Ingester-\u00ebt n\u00eb prodhim mor\u00ebn niveli<\/b> prioritet. P\u00ebr m\u00eb tep\u00ebr, duhej kujdesur paraprakisht p\u00ebr prioritetin e tyre t\u00eb lart\u00eb.<\/b> Tani gjith\u00e7ka \u00ebsht\u00eb rregulluar. Shpresojm\u00eb q\u00eb p\u00ebrvoja jon\u00eb do t\u00eb ndihmoj\u00eb organizatat e tjera q\u00eb po shqyrtojn\u00eb p\u00ebrdorimin e prioriteteve t\u00eb pod'\u00ebve n\u00eb Kubernetes.<\/p>\n

Ne do t\u00eb shtojm\u00eb nj\u00eb nivel t\u00eb konsideruesh\u00ebm kontrolli mbi shp\u00ebrndarjen e \u00e7do objekti shtes\u00eb, t\u00eb cilat konfigurimet e tyre jan\u00eb globale p\u00ebr klasterin. N\u00eb t\u00eb ardhmen, ndryshime t\u00eb tilla do t\u00eb vler\u00ebsohen ngao<\/i>nj\u00eb num\u00ebr m\u00eb t\u00eb madh njer\u00ebzish. P\u00ebr m\u00eb tep\u00ebr, modifikimi q\u00eb \u00e7oi n\u00eb d\u00ebshtim u konsiderua tep\u00ebr i vog\u00ebl p\u00ebr nj\u00eb dokument projekti t\u00eb ve\u00e7ant\u00eb \u2014 u diskutua vet\u00ebm n\u00eb \u00e7\u00ebshtjen GitHub. Nga ky moment e tutje, t\u00eb gjitha ndryshimet e tilla t\u00eb konfigurimeve do t\u00eb shoq\u00ebrohen me dokumentacionin p\u00ebrkat\u00ebs t\u00eb projektit.<\/p>\n

M\u00eb n\u00eb fund, ne do t\u00eb automatizojm\u00eb rritjen e madh\u00ebsis\u00eb s\u00eb serverit t\u00eb prapavij\u00ebs s\u00eb autentifikimit p\u00ebr t\u00eb parandaluar OOM gjat\u00eb ngarkes\u00ebs, t\u00eb cil\u00ebt ishim d\u00ebshmitar\u00eb, dhe do t\u00eb analizojm\u00eb parametrat default t\u00eb Prometheus q\u00eb lidhen me r\u00ebnien dhe shumimin, p\u00ebr t\u00eb parandaluar probleme t\u00eb ngjashme n\u00eb t\u00eb ardhmen.<\/p>\n

D\u00ebshtimi i p\u00ebrjetuar kishte gjithashtu disa pasoja pozitive: duke marr\u00eb burimet e nevojshme, Cortex u rikuperua automatikisht pa ndihm\u00eb shtes\u00eb. Ne gjithashtu mor\u00ebm p\u00ebrvoj\u00eb t\u00eb \u00e7muar n\u00eb pun\u00ebn me Grafana Loki<\/a><\/noindex> \u2014 sistemi yn\u00eb i ri i agregimit t\u00eb log-\u00ebve, \u2014 i cili ndihmoi p\u00ebr t\u00eb siguruar q\u00eb t\u00eb gjith\u00eb Ingester-\u00ebt t\u00eb silleshin si\u00e7 ishte e nevojshme gjat\u00eb dhe pas \u00e7\u00ebshtjes.<\/p>\n

P.S. nga p\u00ebrkthyesi<\/h2>\n

\nLexoni gjithashtu n\u00eb blogun ton\u00eb:<\/p>\n