{"id":31139,"date":"2019-10-31T21:39:40","date_gmt":"2019-10-31T18:39:40","guid":{"rendered":"https:\/\/prohoster.info\/blog\/monitorim-resursy-klasterov-kubernetes\/"},"modified":"2019-10-31T21:39:40","modified_gmt":"2019-10-31T18:39:40","slug":"monitorim-resursy-klasterov-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/bs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","title":{"rendered":"Pratimo resurse Kubernetes klastera","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/bn\/g6\/jg\/bng6jgpkvlzy8-2zahywzsnz8du.png\"><img decoding=\"async\" alt=\"Pratimo resurse Kubernetes klastera\" src=\"\/wp-content\/uploads\/2019\/04\/e3e07ee5a2271263948b3d3b39f3b66d.png\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p><\/p>\n<p>Stvorio sam Kube Eagle \u2014 Prometheus izvoznik. Ispostavilo se da je to odli\u010dan alat koji poma\u017ee bolje razumjeti resurse malih i srednjih klastera. Na kraju sam u\u0161tedio vi\u0161e stotina dolara, jer sam biraju\u0107i prave tipove ma\u0161ina i postavljaju\u0107i ograni\u010denja resursa aplikacija prema radnim optere\u0107enjima.<\/p>\n<p><\/p>\n<p>Pri\u010da\u0107u o prednostima <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/google-cloud-tools\/kube-eagle\">Kube Eagle<\/a><\/noindex>, ali prvo \u0107u objasniti zbog \u010dega je do\u0161lo do problema i za\u0161to je potrebna kvalitetna analitika.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Upravljao sam nekoliko klastera sa 4\u201350 nodova. U svakom klasteru \u2014 do 200 mikroservisa i aplikacija. Da bismo efikasnije koristili dostupni hardver, ve\u0107ina deploy-a je bila postavljena sa burstable RAM-om i CPU resursima. Tako podovi mogu uzeti dostupne resurse kad je potrebno i ne ometaju druge aplikacije na toj nodi. Pa, zar to nije sjajno?<\/p>\n<p><\/p>\n<p>Iako je klaster relativno malo tro\u0161io CPU (8%) i RAM (40%), stalno smo imali problema sa izbacivanjem podova kada su poku\u0161avali da alociraju vi\u0161e memorije nego \u0161to je dostupno na nodi. Tada smo imali samo jedan panel za pra\u0107enje resursa Kubernetes-a. Ovako je izgledao:<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/qr\/cp\/x2\/qrcpx2avguhqnbxxg7trlxfnh4u.png\"><img decoding=\"async\" alt=\"Pratimo resurse Kubernetes klastera\" src=\"\/wp-content\/uploads\/2019\/04\/4ff784375fd4d01cc420d0d5b6389524.png\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<em>Grafana panel samo sa cAdvisor metrikama<\/em><\/p>\n<p><\/p>\n<p>Sa ovakvim panelom, nodovi koji tro\u0161e puno memorije i CPU-a nisu problem za uo\u010diti. Problem je shvatiti uzrok. Da bi podovi ostali na mestu, moglo bi se, naravno, postaviti garantovane resurse za sve podove (zatra\u017eeni resursi su jednaki limitu). No, to nije najsmislenije kori\u0161\u0107enje hardvera. U klasteru je bilo nekoliko stotina gigabajta memorije, a neki nodovi su bili u oskudici, dok su drugi imali po 4\u201310 GB vi\u0161ka.<\/p>\n<p><\/p>\n<p>Ispostavilo se da je Kubernetes planer nepravedno raspore\u0111ivao radna optere\u0107enja po dostupnim resursima. Kubernetes planer uzima u obzir razli\u010dite konfiguracije: pravila affinity, taints i tolerations, selektore nodova koji mogu ograni\u010diti dostupne nodove. No, u mom slu\u010daju nije bilo ni\u010dega takvog i podovi su se planirali na osnovu zatra\u017eenih resursa na svakoj nodi.<\/p>\n<p><\/p>\n<p>Za pod se birala noda koja ima najvi\u0161e slobodnih resursa i koja ispunjava uslove zahteva. Ispalo je da zatra\u017eeni resursi na nodama ne odgovaraju stvarnoj potro\u0161nji, i ovde je Kube Eagle do\u0161ao u pomo\u0107 sa svojim mogu\u0107nostima pra\u0107enja resursa.<\/p>\n<p><\/p>\n<p>U mojim klasterima Kubernetes skoro svi resursi se prate samo sa <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/prometheus\/node_exporter\">Node exporter<\/a><\/noindex> i <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kube-state-metrics\">Kube State Metrics<\/a><\/noindex>. Node Exporter pru\u017ea statistiku o ulazu-izlazu i kori\u0161tenju diska, CPU i RAM-a, dok Kube State Metrics prikazuje metrike objekata Kubernetes, kao \u0161to su zahtjevi i ograni\u010denja za resurse CPU i memoriju.<\/p>\n<p><\/p>\n<p>Moramo spojiti metrike kori\u0161tenja s metrima zahtjeva i ograni\u010denja u Grafana, i tada \u0107emo dobiti sve informacije o problemu. Izgleda jednostavno, ali u tim dvjema alatima oznake su razli\u010dito nazvane, a neke metrike uop\u0107e nemaju oznake metapodataka. Kube Eagle sve radi automatski, a panel izgleda ovako:<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/u3\/ce\/hk\/u3cehkycztlwc9m7lpip-4lw2xe.png\"><img decoding=\"async\" alt=\"Pratimo resurse Kubernetes klastera\" src=\"\/wp-content\/uploads\/2019\/04\/53898543679d337df5a151b54fb75f5d.png\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/po\/zi\/fm\/pozifm1dgtkf9nsasb42k1faqi4.png\"><img decoding=\"async\" alt=\"Pratimo resurse Kubernetes klastera\" src=\"\/wp-content\/uploads\/2019\/04\/a549c865b24f90fcc4375ffbb76ea270.png\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<em><noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/dashboards\/9871\">Kube Eagle nadzorna plo\u010da<\/a><\/noindex><\/em><\/p>\n<p><\/p>\n<p>Uspjeli smo rije\u0161iti mnoge probleme s resursima i u\u0161tedjeti opremu:<\/p>\n<p><\/p>\n<ol>\n<li>Neki programeri nisu znali koliko resursa su potrebni mikroservisima (ili se jednostavno nisu brinuli). Nismo imali na\u010dina da prona\u0111emo nepravilne zahtjeve za resursima \u2014 za to je potrebno znati potro\u0161nju uz zahtjeve i ograni\u010denja. Sada vide metrike Prometheus, prate stvarnu upotrebu i prilago\u0111avaju zahtjeve i ograni\u010denja.<\/li>\n<li>JVM aplikacije uzimaju onoliko RAM-a koliko mogu. Sakuplja\u010d sme\u0107a osloba\u0111a memoriju samo ako je iskori\u0161teno vi\u0161e od 75%. A s obzirom na to da je ve\u0107ina usluga burstable, JVM je uvijek zauzimao RAM. Stoga su sve te Java usluge tro\u0161ile mnogo vi\u0161e RAM-a nego \u0161to se o\u010dekivalo.<\/li>\n<li>Neke aplikacije su tra\u017eile previ\u0161e RAM-a, a Kubernetes scheduler nije davao te nodove ostalim aplikacijama, iako su zapravo bile slobodnije od drugih nodova. Jedan programer je slu\u010dajno dodao dodatnu cifru u zahtjevu i zauzeo veliki dio RAM-a: 20 GB umjesto 2. Nitko to nije primijetio. Aplikacija je imala 3 replikacije, tako da su i 3 nodova stradala.<\/li>\n<li>Postavili smo ograni\u010denja resursa, preplanirali podove s pravilnim zahtjevima i dobili savr\u0161enu ravnote\u017eu kori\u0161tenja opreme po svim nodovima. Par nodova se moglo potpuno zatvoriti. A onda smo primijetili da imamo pogre\u0161ne ma\u0161ine (orijentisane na CPU, a ne na memoriju). Promijenili smo tip i uklonili jo\u0161 nekoliko nodova.<\/li>\n<\/ol>\n<p><\/p>\n<h3 id=\"itogi\">Zaklju\u010dci<\/h3>\n<p><\/p>\n<p>S burstable resursima u klasteru u\u010dinkovitije koristite dostupne resurse, ali Kubernetes scheduler planira podove na temelju zahtjeva za resursima, \u0161to mo\u017ee biti problemati\u010dno. Da bismo rije\u0161ili dva problema: izbjegli neugodnosti i maksimalno iskoristili resurse, potrebna je dobra provjera. Zato \u0107e to biti korisno. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/google-cloud-tools\/kube-eagle\">Kube Eagle<\/a><\/noindex> (izvoznik Prometheus i kontrolna tabla Grafana).<\/p>\n<p>Izvor: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/447336\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u042f \u0441\u043e\u0437\u0434\u0430\u043b Kube Eagle\u00a0\u2014 \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0435\u0440 Prometheus. \u041e\u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c, \u043a\u0440\u0443\u0442\u0430\u044f \u0448\u0442\u0443\u043a\u0430, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u043b\u0443\u0447\u0448\u0435 \u0440\u0430\u0437\u0431\u0438\u0440\u0430\u0442\u044c\u0441\u044f \u0432 \u0440\u0435\u0441\u0443\u0440\u0441\u0430\u0445 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0445 \u0438 \u0441\u0440\u0435\u0434\u043d\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432. \u0412 \u0438\u0442\u043e\u0433\u0435 \u044f \u0441\u044d\u043a\u043e\u043d\u043e\u043c\u0438\u043b \u043d\u0435 \u043e\u0434\u043d\u0443 \u0441\u043e\u0442\u043d\u044e \u0434\u043e\u043b\u043b\u0430\u0440\u043e\u0432, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u043f\u043e\u0434\u0431\u0438\u0440\u0430\u043b \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0438\u043f\u044b \u043c\u0430\u0448\u0438\u043d \u0438 \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u043f\u043e\u0434 \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0438. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443 \u043e \u043f\u0440\u0435\u0438\u043c\u0443\u0449\u0435\u0441\u0442\u0432\u0430\u0445 Kube Eagle, \u043d\u043e \u0441\u043d\u0430\u0447\u0430\u043b\u0430 \u043e\u0431\u044a\u044f\u0441\u043d\u044e, \u0438\u0437-\u0437\u0430 \u0447\u0435\u0433\u043e \u0432\u044b\u0448\u0435\u043b \u0441\u044b\u0440-\u0431\u043e\u0440 \u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":23107,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-31139","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5.1 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/bs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"bs_BA\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043c \u0440\u0435\u0441\u0443\u0440\u0441\u044b \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/bs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:39:40+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:39:40+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Pratimo resurse Kubernetes klastera | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/bs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"bs_BA","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043c \u0440\u0435\u0441\u0443\u0440\u0441\u044b \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 Kubernetes | ProHoster","og:url":"https:\/\/prohoster.info\/bs\/blog\/administrirovanie\/monitorim-resursy-klasterov-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:39:40+00:00","article:modified_time":"2019-10-31T18:39:40+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"31139","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 04:42:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:22:34","updated":"2026-01-21 04:42:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/posts\/31139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/comments?post=31139"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/posts\/31139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/media\/23107"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/media?parent=31139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/categories?post=31139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/bs\/wp-json\/wp\/v2\/tags?post=31139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}