{"id":31139,"date":"2019-10-31T21:39:40","date_gmt":"2019-10-31T18:39:40","guid":{"rendered":"https:\/\/prohoster.info\/blog\/monitorim-resursy-klasterov-kubernetes\/"},"modified":"2019-10-31T21:39:40","modified_gmt":"2019-10-31T18:39:40","slug":"monitorim-resursy-klasterov-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","title":{"rendered":"Monitorujeme zdroje klastrov Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Monitorujeme<\/a><\/noindex><\/p>\n

<\/p>\n

Vytvoril som Kube Eagle \u2014 export\u00e9r Prometheus. Uk\u00e1zalo sa, \u017ee to je skvel\u00fd n\u00e1stroj, ktor\u00fd pom\u00e1ha lep\u0161ie sa zorientova\u0165 v zdrojoch mal\u00fdch a stredn\u00fdch klastrov. V kone\u010dnom d\u00f4sledku som u\u0161etril nie jednu stovku dol\u00e1rov, preto\u017ee som vyberal spr\u00e1vne typy strojov a nastavoval obmedzenia zdrojov aplik\u00e1ci\u00ed pod\u013ea pracovn\u00fdch z\u00e1\u0165a\u017e\u00ed.<\/p>\n

<\/p>\n

Povediem v\u00e1s k v\u00fdhod\u00e1m Kube Eagle<\/a><\/noindex>, ale najprv vysvetl\u00edm, \u010do sp\u00f4sobilo neporozumenie a na \u010do bol potrebn\u00fd kvalitn\u00fd monitoring.<\/p>\n

<\/a><\/noindex><\/p>\n

Spravoval som nieko\u013eko klastrov so 4\u201350 uzlami. V ka\u017edom klastri bolo a\u017e 200 mikroslu\u017eieb a aplik\u00e1ci\u00ed. Aby som \u00fa\u010dinnej\u0161ie vyu\u017eil dostupn\u00fd hardv\u00e9r, v\u00e4\u010d\u0161ina nasaden\u00ed bola nastaven\u00e1 so burstable opera\u010dnou pam\u00e4\u0165ou a CPU zdrojmi. Tak podov\u00e9 kontajnery m\u00f4\u017eu vyu\u017ei\u0165 dostupn\u00e9 zdroje, ak to potrebuj\u00fa, a pri tom neobmedzuj\u00fa ostatn\u00e9 aplik\u00e1cie na tomto uzle. No nie je to skvel\u00e9?<\/p>\n

<\/p>\n

A hoci klaster spotreboval relat\u00edvne m\u00e1lo CPU (8%) a pam\u00e4te (40%), neust\u00e1le sme sa stret\u00e1vali s probl\u00e9mami pri vytl\u00e1\u010dan\u00ed podov, ke\u010f sa sna\u017eili z\u00edska\u0165 viac pam\u00e4te, ne\u017e bolo dostupn\u00e9 na uzle. Vtedy sme mali len jedin\u00e9 rozhranie pre monitoring zdrojov Kubernetes. Tak\u00e9to:<\/p>\n

<\/p>\n

\"Monitorujeme<\/a><\/noindex>
\nGrafana panel iba s metrikami cAdvisor<\/em><\/p>\n

<\/p>\n

S tak\u00fdmto panelom je vidie\u0165 uzly, ktor\u00e9 spotreb\u00favaj\u00fa ve\u013ea pam\u00e4te a CPU, bez probl\u00e9mov. Probl\u00e9m je pochopi\u0165, v \u010dom je pr\u00ed\u010dina. Aby podov\u00e9 kontajnery zostali na mieste, bolo by mo\u017en\u00e9 samozrejme nastavi\u0165 garantovan\u00e9 zdroje na v\u0161etk\u00fdch podoch (po\u017eadovan\u00e9 zdroje s\u00fa rovn\u00e9 limitu). Ale to nie je najrozumnej\u0161ie vyu\u017eitie hardv\u00e9ru. V klastere bolo nieko\u013eko stoviek gigabajtov pam\u00e4te, pri\u010dom niektor\u00e9 uzly hladovali, a in\u00e9 mali v rezerv\u00e1ch po 4\u201310 GB.<\/p>\n

<\/p>\n

Vych\u00e1dza to tak, \u017ee Kubernetes pl\u00e1nova\u010d rozde\u013eoval pracovn\u00e9 z\u00e1\u0165a\u017ee po dostupn\u00fdch zdrojoch nerovnomerne. Kubernetes pl\u00e1nova\u010d berie do \u00favahy r\u00f4zne konfigur\u00e1cie: pravidl\u00e1 affinity, taint a tolerancie, selektory uzlov, ktor\u00e9 m\u00f4\u017eu obmedzi\u0165 dostupn\u00e9 uzly. Ale v mojom pr\u00edpade ni\u010d tak\u00e9 neexistovalo, a podov\u00e9 kontajnery sa pl\u00e1novali na z\u00e1klade po\u017eadovan\u00fdch zdrojov na ka\u017edom uzle.<\/p>\n

<\/p>\n

Pre podov\u00fd kontajner sa vybrala uzol s najviac vo\u013en\u00fdmi zdrojmi a ktor\u00fd sp\u013a\u0148al podmienky po\u017eiadavky. Uk\u00e1zalo sa, \u017ee po\u017eadovan\u00e9 zdroje na uzloch sa nezhodovali so skuto\u010dn\u00fdm vyu\u017eit\u00edm, a tu pri\u0161iel na pomoc Kube Eagle a jeho mo\u017enosti monitorovania zdrojov.<\/p>\n

<\/p>\n

Takmer v\u0161etky moje Kubernetes klastre boli sledovan\u00e9 iba pomocou Node exporter<\/a><\/noindex> a Kube State Metrics<\/a><\/noindex>. Node Exporter poskytuje \u0161tatistiky o vstupe a v\u00fdstupe, vyu\u017eit\u00ed disku, CPU a pam\u00e4te, zatia\u013e \u010do Kube State Metrics ukazuje metriky objektov Kubernetes, ako napr\u00edklad po\u017eiadavky a limity na zdroje CPU a pam\u00e4\u0165.<\/p>\n

<\/p>\n

Mus\u00edme skombinova\u0165 metriky o vyu\u017eit\u00ed so metrikami po\u017eiadaviek a limitov v Grafane, \u010d\u00edm z\u00edskame v\u0161etky inform\u00e1cie o probl\u00e9me. Znie to jednoducho, ale v t\u00fdchto dvoch n\u00e1strojoch s\u00fa ozna\u010denia nazvan\u00e9 r\u00f4zne a niektor\u00e9 metriky nemaj\u00fa v\u00f4bec \u017eiadne metad\u00e1tov\u00e9 ozna\u010denia. Kube Eagle to rob\u00ed v\u0161etko automaticky a panel vyzer\u00e1 takto:<\/p>\n

<\/p>\n

\"Monitorujeme<\/a><\/noindex><\/p>\n

<\/p>\n

\"Monitorujeme<\/a><\/noindex>
\nMonitorovac\u00ed panel Kube Eagle<\/a><\/noindex><\/em><\/p>\n

<\/p>\n

Podarilo sa n\u00e1m vyrie\u0161i\u0165 ve\u013ea probl\u00e9mov so zdrojmi a u\u0161etri\u0165 vybavenie:<\/p>\n

<\/p>\n

    \n
  1. Niektor\u00ed v\u00fdvoj\u00e1ri nevedeli, ko\u013eko zdrojov potrebuj\u00fa mikroslu\u017eby (alebo sa jednoducho nezam\u00fd\u0161\u013eali). Nemali sme n\u00e1stroje na identifik\u00e1ciu nespr\u00e1vnych po\u017eiadaviek na zdroje - na to je potrebn\u00e9 pozna\u0165 spotrebu plus po\u017eiadavky a limity. Teraz vidia metriky Prometheus, sleduj\u00fa skuto\u010dn\u00e9 vyu\u017eitie a prisp\u00f4sobuj\u00fa po\u017eiadavky a limity.<\/li>\n
  2. JVM aplik\u00e1cie zaberaj\u00fa to\u013eko pam\u00e4te, ko\u013eko m\u00f4\u017eu. Zberate\u013e odpadu uvo\u013e\u0148uje pam\u00e4\u0165 iba vtedy, ak je vyu\u017eit\u00e9 viac ako 75%. A ke\u010f\u017ee v\u00e4\u010d\u0161ina slu\u017eieb m\u00e1 pam\u00e4\u0165 s burstable vlastnos\u0165ami, JVM ju v\u017edy zaberal. Preto tieto Java slu\u017eby spotrebov\u00e1vali ove\u013ea viac operat\u00edvnej pam\u00e4te, ne\u017e sa o\u010dak\u00e1valo.<\/li>\n
  3. Niektor\u00e9 aplik\u00e1cie po\u017eadovali pr\u00edli\u0161 ve\u013ea pam\u00e4te, a pl\u00e1nova\u010d Kubernetes nedovolil t\u00fdmto uzlom priradi\u0165 ostatn\u00e9 aplik\u00e1cie, hoci v skuto\u010dnosti boli vo\u013enej\u0161ie ako in\u00e9 uzly. Jeden v\u00fdvoj\u00e1r n\u00e1hodou pridal nadbyto\u010dn\u00fa \u010d\u00edslicu v po\u017eiadavke a zabral ve\u013ek\u00fd kus pam\u00e4te: 20 GB namiesto 2. Nikto si to nev\u0161imol. Aplik\u00e1cia mala 3 repliky, tak\u017ee trpeli aj 3 uzly.<\/li>\n
  4. Zaviedli sme obmedzenia na zdroje, prepl\u00e1novali pod a z\u00edskali ide\u00e1lny balans vyu\u017eitia hardv\u00e9ru na v\u0161etk\u00fdch uzloch. Niektor\u00e9 uzly sme v\u00f4bec mohli uzavrie\u0165. A potom sme zistili, \u017ee m\u00e1me nespr\u00e1vne ma\u0161iny (orientovan\u00e9 na CPU, a nie na pam\u00e4\u0165). Zmenili sme typ a odstr\u00e1nili \u010fal\u0161ie nieko\u013eko uzlov.<\/li>\n<\/ol>\n

    <\/p>\n

    Zhrnutie<\/h3>\n

    <\/p>\n

    S burstable zdrojmi v klastri efekt\u00edvnej\u0161ie vyu\u017e\u00edvate existuj\u00faci hardv\u00e9r, av\u0161ak pl\u00e1nova\u010d Kubernetes pl\u00e1nuje pod pod\u013ea po\u017eiadaviek na zdroje, \u010do m\u00f4\u017ee by\u0165 problematick\u00e9. Aby sme sa vyhli probl\u00e9mom a vyu\u017eili zdroje naplno, potrebujeme dobr\u00e9 monitorovanie. Na to sa aj hod\u00ed. Kube Eagle<\/a><\/noindex> (export\u00e9r Prometheus a panelu monitorovania Grafana).<\/p>\n

    Zdroj: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

    \u042f \u0441\u043e\u0437\u0434\u0430\u043b Kube Eagle\u00a0\u2014 \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 Prometheus. \u041e\u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c, \u043a\u0440\u0443\u0442\u0430\u044f \u0448\u0442\u0443\u043a\u0430, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u043b\u0443\u0447\u0448\u0435 \u0440\u0430\u0437\u0431\u0438\u0440\u0430\u0442\u044c\u0441\u044f \u0432 \u0440\u0435\u0441\u0443\u0440\u0441\u0430\u0445 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0445 \u0438 \u0441\u0440\u0435\u0434\u043d\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432. \u0412 \u0438\u0442\u043e\u0433\u0435 \u044f \u0441\u044d\u043a\u043e\u043d\u043e\u043c\u0438\u043b \u043d\u0435 \u043e\u0434\u043d\u0443 \u0441\u043e\u0442\u043d\u044e \u0434\u043e\u043b\u043b\u0430\u0440\u043e\u0432, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u043f\u043e\u0434\u0431\u0438\u0440\u0430\u043b \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0438\u043f\u044b \u043c\u0430\u0448\u0438\u043d \u0438 \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u043f\u043e\u0434 \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0438. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443 \u043e \u043f\u0440\u0435\u0438\u043c\u0443\u0449\u0435\u0441\u0442\u0432\u0430\u0445 Kube Eagle, \u043d\u043e \u0441\u043d\u0430\u0447\u0430\u043b\u0430 \u043e\u0431\u044a\u044f\u0441\u043d\u044e, \u0438\u0437-\u0437\u0430 \u0447\u0435\u0433\u043e \u0432\u044b\u0448\u0435\u043b \u0441\u044b\u0440-\u0431\u043e\u0440 \u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":23107,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-31139","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Sledujeme zdroje Kubernetes klastrov | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043c \u0440\u0435\u0441\u0443\u0440\u0441\u044b \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 Kubernetes | ProHoster","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:39:40+00:00","article:modified_time":"2019-10-31T18:39:40+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"31139","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 04:42:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:22:34","updated":"2026-01-21 04:42:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/31139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=31139"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/31139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/23107"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=31139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=31139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=31139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}