{"id":31139,"date":"2019-10-31T21:39:40","date_gmt":"2019-10-31T18:39:40","guid":{"rendered":"https:\/\/prohoster.info\/blog\/monitorim-resursy-klasterov-kubernetes\/"},"modified":"2019-10-31T21:39:40","modified_gmt":"2019-10-31T18:39:40","slug":"monitorim-resursy-klasterov-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","title":{"rendered":"Sledujeme zdroje klastr\u016f Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Sledujeme<\/a><\/noindex><\/p>\n

<\/p>\n

Vytvo\u0159il jsem Kube Eagle \u2014 exporter Prometheus. Uk\u00e1zalo se to jako skv\u011bl\u00fd n\u00e1stroj, kter\u00fd pom\u00e1h\u00e1 l\u00e9pe porozum\u011bt prost\u0159edk\u016fm mal\u00fdch a st\u0159edn\u00edch cluster\u016f. Nakonec jsem u\u0161et\u0159il n\u011bkolik stovek dolar\u016f, proto\u017ee jsem vyb\u00edral spr\u00e1vn\u00e9 typy stroj\u016f a nastavoval limity prost\u0159edk\u016f aplikac\u00ed dle pracovn\u00edch z\u00e1t\u011b\u017e\u00ed.<\/p>\n

<\/p>\n

Pov\u00edm o v\u00fdhod\u00e1ch Kube Eagle<\/a><\/noindex>, ale nejprve vysv\u011btl\u00edm, pro\u010d do\u0161lo ke konfliktu a k \u010demu bylo pot\u0159eba kvalitn\u00ed monitorov\u00e1n\u00ed.<\/p>\n

<\/a><\/noindex><\/p>\n

Spravoval jsem n\u011bkolik cluster\u016f s 4\u201350 uzly. V ka\u017ed\u00e9m clusteru bylo a\u017e 200 mikroslu\u017eeb a aplikac\u00ed. Abychom efektivn\u011b vyu\u017eili dostupn\u00fd hardware, v\u011bt\u0161ina nasazen\u00ed byla nastavena s burstable RAM a CPU prost\u0159edky. Tak mohly pody vz\u00edt dostupn\u00e9 prost\u0159edky, kdy\u017e bylo pot\u0159eba, a p\u0159itom nebr\u00e1nit ostatn\u00edm aplikac\u00edm na tomto uzlu. No, nen\u00ed to skv\u011bl\u00e9?<\/p>\n

<\/p>\n

I kdy\u017e cluster spot\u0159ebov\u00e1val relativn\u011b m\u00e1lo CPU (8 %) a RAM (40 %), neust\u00e1le jsme m\u011bli probl\u00e9my s odstra\u0148ov\u00e1n\u00edm pod\u016f, kdy\u017e se sna\u017eily alokovat v\u00edce pam\u011bti, ne\u017e bylo na uzlu k dispozici. M\u011bli jsme tehdy pouze jeden panel pro monitorov\u00e1n\u00ed zdroj\u016f Kubernetes. Takov\u00fdto:<\/p>\n

<\/p>\n

\"Sledujeme<\/a><\/noindex>
\nPanel Grafana pouze s metrikami cAdvisor<\/em><\/p>\n

<\/p>\n

S takov\u00fdm panelem nen\u00ed probl\u00e9m vid\u011bt uzly, kter\u00e9 spot\u0159ebov\u00e1vaj\u00ed hodn\u011b pam\u011bti a CPU. Probl\u00e9m je pochopit, co je p\u0159\u00ed\u010dinou. Aby pody z\u016fstaly na sv\u00e9m m\u00edst\u011b, mohly by se samoz\u0159ejm\u011b nastavit garantovan\u00e9 prost\u0159edky na v\u0161ech podech (po\u017eadovan\u00e9 prost\u0159edky se rovnaj\u00ed limitu). Ale to nen\u00ed nejchyt\u0159ej\u0161\u00ed zp\u016fsob vyu\u017eit\u00ed hardwaru. V clusteru bylo n\u011bkolik stovek gigabajt\u016f pam\u011bti, p\u0159i\u010dem\u017e n\u011bkter\u00e9 uzly hladov\u011bly a jin\u00e9 m\u011bly rezervu 4\u201310 GB.<\/p>\n

<\/p>\n

V\u00fdsledkem bylo, \u017ee pl\u00e1nova\u010d Kubernetes rozd\u011bloval pracovn\u00ed z\u00e1t\u011b\u017ee podle dostupn\u00fdch prost\u0159edk\u016f nerovnom\u011brn\u011b. Pl\u00e1nova\u010d Kubernetes zohled\u0148uje r\u016fzn\u00e9 konfigurace: pravidla affinity, taint a tolerace, selektory uzl\u016f, kter\u00e9 mohou omezovat dostupn\u00e9 uzly. Ale v m\u00e9m p\u0159\u00edpad\u011b nic takov\u00e9ho nebylo a pody byly pl\u00e1nov\u00e1ny podle po\u017eadovan\u00fdch prost\u0159edk\u016f na ka\u017ed\u00e9m uzlu.<\/p>\n

<\/p>\n

Pro pod se vyb\u00edrala uzlov\u00e1, kter\u00e1 m\u011bla nejv\u00edce dostupn\u00fdch prost\u0159edk\u016f a spl\u0148ovala po\u017eadavky. M\u011bli jsme situaci, \u017ee po\u017eadovan\u00e9 prost\u0159edky na uzlech se neshodovaly s skute\u010dn\u00fdm vyu\u017eit\u00edm, a zde p\u0159i\u0161el na pomoc Kube Eagle a jeho schopnosti monitorov\u00e1n\u00ed prost\u0159edk\u016f.<\/p>\n

<\/p>\n

Skoro v\u0161echny moje Kubernetes clustery byly sledov\u00e1ny pouze s Node exporter<\/a><\/noindex> a Kube State Metrics<\/a><\/noindex>. Node Exporter poskytuje statistiky o vstupu a v\u00fdstupu, vyu\u017eit\u00ed disku, CPU a RAM, zat\u00edmco Kube State Metrics zobrazuje metriky objekt\u016f Kubernetes, jako jsou po\u017eadavky a limity na zdroje CPU a pam\u011bti.<\/p>\n

<\/p>\n

Mus\u00edme zkombinovat metriky o vyu\u017eit\u00ed se metrikami po\u017eadavk\u016f a limit\u016f v Grafan\u011b, a pak z\u00edsk\u00e1me v\u0161echny informace o probl\u00e9mu. Zn\u00ed to jednodu\u0161e, ale ve skute\u010dnosti se v t\u011bchto dvou n\u00e1stroj\u00edch ozna\u010den\u00ed li\u0161\u00ed a n\u011bkter\u00e9 metriky nemaj\u00ed v\u016fbec \u017e\u00e1dn\u00e9 metadata. Kube Eagle to d\u011bl\u00e1 za v\u00e1s a panel vypad\u00e1 takto:<\/p>\n

<\/p>\n

\"Sledujeme<\/a><\/noindex><\/p>\n

<\/p>\n

\"Sledujeme<\/a><\/noindex>
\nMonitorovac\u00ed panel Kube Eagle<\/a><\/noindex><\/em><\/p>\n

<\/p>\n

Roz \u0159e\u0161ili jsme mnoh\u00e9 probl\u00e9my se zdroji a u\u0161et\u0159ili jsme hardware:<\/p>\n

<\/p>\n

    \n
  1. N\u011bkte\u0159\u00ed v\u00fdvoj\u00e1\u0159i nev\u011bd\u011bli, kolik zdroj\u016f pot\u0159ebuj\u00ed mikroservisy (nebo se prost\u011b nezam\u00fd\u0161leli). Nem\u011bli jsme co pou\u017e\u00edt k hled\u00e1n\u00ed \u0161patn\u00fdch po\u017eadavk\u016f na zdroje \u2014 k tomu je pot\u0159eba zn\u00e1t spot\u0159ebu plus po\u017eadavky a limity. Te\u010f vid\u00ed metriky Prometheus, monitoruj\u00ed skute\u010dn\u00e9 vyu\u017eit\u00ed a p\u0159izp\u016fsobuj\u00ed po\u017eadavky a limity.<\/li>\n
  2. JVM aplikace berou tolik RAM, kolik vlezou. Garbage collector uvol\u0148uje pam\u011b\u0165, pouze pokud je vyu\u017eito v\u00edce ne\u017e 75%. A jeliko\u017e v\u011bt\u0161ina slu\u017eeb m\u00e1 pam\u011b\u0165 burstable, JVM ji v\u017edy zab\u00edral. Proto v\u0161echny tyto Java slu\u017eby spot\u0159ebovaly mnohem v\u00edce RAM, ne\u017e se o\u010dek\u00e1valo.<\/li>\n
  3. N\u011bkter\u00e9 aplikace po\u017eadovaly p\u0159\u00edli\u0161 mnoho pam\u011bti, a pl\u00e1nova\u010d Kubernetes nedovolil tyto uzly ostatn\u00edm aplikac\u00edm, i kdy\u017e byly ve skute\u010dnosti svobodn\u011bj\u0161\u00ed ne\u017e jin\u00e9 uzly. Jeden v\u00fdvoj\u00e1\u0159 omylem p\u0159idal nav\u00edc \u010d\u00edslo do po\u017eadavku a zabral velk\u00fd kus pam\u011bti: 20 GB m\u00edsto 2. Nikdo si toho nev\u0161iml. Aplikace m\u011bla 3 repliky, tak\u017ee po\u0161kodilo to hned 3 uzly.<\/li>\n
  4. Zavedli jsme omezen\u00ed na zdroje, p\u0159epl\u00e1novali jsme pody s spr\u00e1vn\u00fdmi po\u017eadavky a z\u00edskali ide\u00e1ln\u00ed rovnov\u00e1hu ve vyu\u017eit\u00ed hardwaru na v\u0161ech uzlech. N\u011bkter\u00e9 uzly jsme mohli dokonce vypnout. A pak jsme zjistili, \u017ee m\u00e1me nespr\u00e1vn\u00fd typ stroj\u016f (zam\u011b\u0159en\u00e9 na CPU, nikoli na pam\u011b\u0165). Zm\u011bnili jsme typ a odstranili dal\u0161\u00ed uzly.<\/li>\n<\/ol>\n

    <\/p>\n

    Z\u00e1v\u011bry<\/h3>\n

    <\/p>\n

    S burstable zdroji v clustru efektivn\u011bji vyu\u017e\u00edv\u00e1te dostupn\u00fd hardware, ale pl\u00e1nova\u010d Kubernetes napl\u00e1nuje pody podle po\u017eadavk\u016f na zdroje, co\u017e m\u016f\u017ee b\u00fdt problematick\u00e9. Abychom zm\u00e1tli dv\u011b mouchy jednou ranou: vyhnuli se probl\u00e9m\u016fm a pln\u011b vyu\u017eili zdroje, pot\u0159ebujeme dobr\u00e9 monitorov\u00e1n\u00ed. K tomu se hod\u00ed. Kube Eagle<\/a><\/noindex> (export\u00e9r Prometheus a panel pro sledov\u00e1n\u00ed Grafana).<\/p>\n

    Zdroj: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

    \u042f \u0441\u043e\u0437\u0434\u0430\u043b Kube Eagle\u00a0\u2014 \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 Prometheus. \u041e\u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c, \u043a\u0440\u0443\u0442\u0430\u044f \u0448\u0442\u0443\u043a\u0430, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u043b\u0443\u0447\u0448\u0435 \u0440\u0430\u0437\u0431\u0438\u0440\u0430\u0442\u044c\u0441\u044f \u0432 \u0440\u0435\u0441\u0443\u0440\u0441\u0430\u0445 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0445 \u0438 \u0441\u0440\u0435\u0434\u043d\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432. \u0412 \u0438\u0442\u043e\u0433\u0435 \u044f \u0441\u044d\u043a\u043e\u043d\u043e\u043c\u0438\u043b \u043d\u0435 \u043e\u0434\u043d\u0443 \u0441\u043e\u0442\u043d\u044e \u0434\u043e\u043b\u043b\u0430\u0440\u043e\u0432, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u043f\u043e\u0434\u0431\u0438\u0440\u0430\u043b \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0438\u043f\u044b \u043c\u0430\u0448\u0438\u043d \u0438 \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u043f\u043e\u0434 \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0438. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443 \u043e \u043f\u0440\u0435\u0438\u043c\u0443\u0449\u0435\u0441\u0442\u0432\u0430\u0445 Kube Eagle, \u043d\u043e \u0441\u043d\u0430\u0447\u0430\u043b\u0430 \u043e\u0431\u044a\u044f\u0441\u043d\u044e, \u0438\u0437-\u0437\u0430 \u0447\u0435\u0433\u043e \u0432\u044b\u0448\u0435\u043b \u0441\u044b\u0440-\u0431\u043e\u0440 \u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":23107,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-31139","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Sledujeme zdroje Kubernetes cluster\u016f | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043c \u0440\u0435\u0441\u0443\u0440\u0441\u044b \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 Kubernetes | ProHoster","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:39:40+00:00","article:modified_time":"2019-10-31T18:39:40+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"31139","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 04:42:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:22:34","updated":"2026-01-21 04:42:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/31139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=31139"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/31139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/23107"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=31139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=31139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=31139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}