{"id":91199,"date":"2020-08-10T01:42:03","date_gmt":"2020-08-09T23:42:03","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov"},"modified":"2020-08-10T01:42:03","modified_gmt":"2020-08-09T23:42:03","slug":"kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","title":{"rendered":"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Jmenuji se Viktor Jagoofarov a pracuji na rozvoji Kubernetes platformy ve spole\u010dnosti DomClick jako technick\u00fd vedouc\u00ed v\u00fdvoje v t\u00fdmu Ops (provoz). R\u00e1d bych vypr\u00e1v\u011bl o struktu\u0159e na\u0161ich Dev &lt;-&gt; Ops proces\u016f, o zvl\u00e1\u0161tnostech provozu jednoho z nejv\u011bt\u0161\u00edch k8s cluster\u016f v Rusku a tak\u00e9 o DevOps\/SRE praktik\u00e1ch, kter\u00e9 n\u00e1\u0161 t\u00fdm pou\u017e\u00edv\u00e1.<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/6c908104c734b36cc7a960fbe4d65e3f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h4>T\u00fdm Ops<\/h4>\n<p>\nV t\u00fdmu Ops moment\u00e1ln\u011b pracuje 15 lid\u00ed. T\u0159i z nich maj\u00ed na starosti kancel\u00e1\u0159, dva pracuj\u00ed v jin\u00e9m \u010dasov\u00e9m p\u00e1smu a jsou dostupn\u00ed, v\u010detn\u011b noci. T\u00edm p\u00e1dem je v\u017edy n\u011bkdo z Ops u monitoru a p\u0159ipraven reagovat na incident jak\u00e9koliv slo\u017eitosti. Nem\u00e1me no\u010dn\u00ed slu\u017eby, co\u017e chr\u00e1n\u00ed na\u0161i psychiku a d\u00e1v\u00e1 v\u0161em mo\u017enost dostate\u010dn\u011b vyspat a tr\u00e1vit voln\u00fd \u010das nejen p\u0159ed po\u010d\u00edta\u010dem.<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/47051db273f116df376e64bad67c6f37.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u0161ichni m\u00e1me r\u016fzn\u00e9 kompetence: s\u00ed\u0165a\u0159i, DBA, specialist\u00e9 na ELK stack, Kubernetes administr\u00e1to\u0159i\/v\u00fdvoj\u00e1\u0159i, odborn\u00edci na monitoring, virtualizaci, hardware atd. Spojuje n\u00e1s to, \u017ee ka\u017ed\u00fd m\u016f\u017ee v ur\u010dit\u00e9 m\u00ed\u0159e nahradit kohokoliv z n\u00e1s: nap\u0159\u00edklad zav\u00e9st nov\u00e9 nody do k8s clusteru, aktualizovat PostgreSQL, napsat CI\/CD pipeline + Ansible, automatizovat n\u011bco v Pythonu\/Bash\/Go, p\u0159ipojit hardware do datov\u00e9ho centra. Siln\u00e9 kompetence v ur\u010dit\u00e9 oblasti nebr\u00e1n\u00ed zm\u011bnit sm\u011b\u65b9\u5411 a za\u010d\u00edt se rozv\u00edjet v jin\u00e9 oblasti. Nap\u0159\u00edklad jsem do spole\u010dnosti nastoupil jako specialista na PostgreSQL, ale nyn\u00ed je moj\u00ed hlavn\u00ed zodpov\u011bdnost\u00ed spr\u00e1va Kubernetes cluster\u016f. V t\u00fdmu je jak\u00fdkoliv r\u016fst v\u00edt\u00e1n a c\u00edt\u00edme silnou podporu jeden od druh\u00e9ho.<\/p>\n<p>Mimochodem, hled\u00e1me nov\u00e9 lidi. Po\u017eadavky na uchaze\u010de jsou pom\u011brn\u011b standardn\u00ed. Osobn\u011b je pro m\u011b d\u016fle\u017eit\u00e9, aby se \u010dlov\u011bk hodil do kolektivu, byl nekonfliktn\u00ed, p\u0159itom v\u0161ak dok\u00e1zal h\u00e1jit sv\u016fj n\u00e1zor, cht\u011bl se rozv\u00edjet a neb\u00e1l se zkou\u0161et n\u011bco nov\u00e9ho, navrhoval sv\u00e9 n\u00e1pady. D\u00e1le jsou nezbytn\u00e9 dovednosti programov\u00e1n\u00ed ve skriptovac\u00edch jazyc\u00edch, znalosti z\u00e1klad\u016f Linuxu a angli\u010dtiny. Angli\u010dtina je pot\u0159eba jen proto, aby si mohl \u010dlov\u011bk v p\u0159\u00edpad\u011b probl\u00e9mu rychle vyhledat \u0159e\u0161en\u00ed za 10 sekund, nikoli za 10 minut. Se specialisty s hlubok\u00fdmi znalostmi Linuxu je nyn\u00ed velmi obt\u00ed\u017en\u00e9: sm\u011b\u0161n\u00e9, ale dva z t\u0159\u00ed uchaze\u010d\u016f nedok\u00e1\u017eou odpov\u011bd\u011bt na ot\u00e1zku \u201eCo je to Load Average? Z \u010deho se skl\u00e1d\u00e1?\u201c a ot\u00e1zku \u201eJak zpracovat core dump ze C programu\u201c pova\u017euj\u00ed za n\u011bco, co pat\u0159\u00ed do sv\u011bta superlid\u00ed\u2026 nebo dinosaur\u016f. S t\u00edm se mus\u00edme sm\u00ed\u0159it, proto\u017ee obvykle maj\u00ed lid\u00e9 v\u00fdrazn\u011b rozvinut\u00e9 jin\u00e9 kompetence, a \u201eLinux\u201c je nau\u010d\u00edme. Odpov\u011b\u010f na ot\u00e1zku \u201epro\u010d to v\u0161echno pot\u0159ebuje DevOps in\u017een\u00fdr v modern\u00edm sv\u011bt\u011b cloud\u016f\u201c si nech\u00e1me stranou, ale zjednodu\u0161en\u011b \u0159e\u010deno: je to v\u0161echno pot\u0159eba.<\/p>\n<h4>T\u00fdm N\u00e1stroje<\/h4>\n<p>\nV\u00fdznamnou roli v automatizaci hraje t\u00fdm N\u00e1stroje. Jejich hlavn\u00edm \u00fakolem je vytv\u00e1\u0159en\u00ed u\u017eivatelsky p\u0159\u00edv\u011btiv\u00fdch grafick\u00fdch a CLI n\u00e1stroj\u016f pro v\u00fdvoj\u00e1\u0159e. Nap\u0159\u00edklad n\u00e1\u0161 intern\u011b vyvinut\u00fd n\u00e1stroj Confer umo\u017e\u0148uje doslova n\u011bkolika kliknut\u00edmi my\u0161i nasadit aplikaci do Kubernetes, nastavit ji zdroje, kl\u00ed\u010de z vault atd. D\u0159\u00edve jsme pou\u017e\u00edvali Jenkins + Helm 2, ale museli jsme vyvinout vlastn\u00ed n\u00e1stroj, abychom vylou\u010dili kop\u00edrov\u00e1n\u00ed a vnesli jednotnost do \u017eivotn\u00edho cyklu softwaru.<\/p>\n<p>T\u00fdm Ops nenap\u00ed\u0161e pipeline za v\u00fdvoj\u00e1\u0159e, ale m\u016f\u017ee poskytnout konzultace ohledn\u011b jak\u00fdchkoli ot\u00e1zek o jejich psan\u00ed (n\u011bkte\u0159\u00ed st\u00e1le pou\u017e\u00edvaj\u00ed Helm 3).<\/p>\n<h4>DevOps<\/h4>\n<p>\nPokud jde o DevOps, vid\u00edme ho takto:<\/p>\n<p>T\u00fdmy Dev p\u00ed\u0161\u00ed k\u00f3d a nasazuj\u00ed jej p\u0159es Confer do dev -&gt; qa\/stage -&gt; prod. Odpov\u011bdnost za to, aby k\u00f3d nebrzdil a nevytv\u00e1\u0159el chyby, le\u017e\u00ed na t\u00fdmech Dev a Ops. B\u011bhem dne by na incident se sv\u00fdm aplikac\u00ed m\u011bl prim\u00e1rn\u011b reagovat dozor\u010d\u00ed z t\u00fdmu Ops, a b\u011bhem ve\u010dera a noci by m\u011bl dozor\u010d\u00ed administr\u00e1tor (Ops) vzbudit dozor\u010d\u00edho v\u00fdvoj\u00e1\u0159e, pokud si je jist, \u017ee probl\u00e9m nen\u00ed v infrastruktu\u0159e. V\u0161echny metriky a alarmy v monitoringu se objevuj\u00ed automaticky nebo poloautomaticky.<\/p>\n<p>Oblast odpov\u011bdnosti Ops za\u010d\u00edn\u00e1 okam\u017eikem nasazen\u00ed aplikace do produkce, ale odpov\u011bdnost Dev t\u00edm nekon\u010d\u00ed \u2014 d\u011bl\u00e1me to stejn\u00e9 a jsme na jedn\u00e9 lodi.<\/p>\n<p>V\u00fdvoj\u00e1\u0159i konzultuj\u00ed spr\u00e1vce, pokud pot\u0159ebuj\u00ed pomoc s psan\u00edm administrativn\u00edho mikroslu\u017eby (nap\u0159\u00edklad Go backend + HTML5), a spr\u00e1vci konzultuj\u00ed v\u00fdvoj\u00e1\u0159e ohledn\u011b jak\u00fdchkoliv infrastrukturn\u00edch ot\u00e1zek nebo ot\u00e1zek souvisej\u00edc\u00edch s k8s.<\/p>\n<p>Mimochodem, v\u016fbec nem\u00e1me monolit, pouze mikroslu\u017eby. Jejich po\u010det se zat\u00edm pohybuje mezi 900 a 1000 v produk\u010dn\u00edm k8s clusteru, pokud m\u011b\u0159\u00edme podle po\u010dtu <i>nasazen\u00ed<\/i>. Po\u010det pod\u016f kol\u00eds\u00e1 mezi 1700 a 2000. V produk\u010dn\u00edm clusteru je nyn\u00ed kolem 2000 pod\u016f.<\/p>\n<p>P\u0159esn\u00e1 \u010d\u00edsla nemohu uv\u00e9st, proto\u017ee sledujeme nepot\u0159ebn\u00e9 mikroslu\u017eby a odstra\u0148ujeme je v poloautomatick\u00e9m re\u017eimu. Sledov\u00e1n\u00ed nepot\u0159ebn\u00fdch entit v k8s n\u00e1m pom\u00e1h\u00e1 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Nastradamus\/useless-operator\">useless-operator<\/a><\/noindex>, co\u017e skv\u011ble \u0161et\u0159\u00ed zdroje a pen\u00edze.<\/p>\n<h2> Spr\u00e1va zdroj\u016f <\/h2>\n<p><\/p>\n<h4> Monitoring <\/h4>\n<p>\nSt\u011b\u017eejn\u00edm kamenem provozov\u00e1n\u00ed velk\u00e9ho clusteru se st\u00e1v\u00e1 spr\u00e1vn\u011b nastaven\u00fd a informativn\u00ed monitoring. Zat\u00edm jsme nena\u0161li univerz\u00e1ln\u00ed \u0159e\u0161en\u00ed, kter\u00e9 by pokrylo 100 % v\u0161ech \"p\u0159\u00e1n\u00ed\" ohledn\u011b monitorov\u00e1n\u00ed, proto periodicky vytv\u00e1\u0159\u00edme r\u016fzn\u00e1 vlastn\u00ed \u0159e\u0161en\u00ed v tomto prost\u0159ed\u00ed.<\/p>\n<ul>\n<li><b>Zabbix<\/b>. Star\u00fd dobr\u00fd monitoring, kter\u00fd je ur\u010den p\u0159edev\u0161\u00edm k sledov\u00e1n\u00ed celkov\u00e9ho stavu infrastruktury. \u0158\u00edk\u00e1 n\u00e1m, kdy\u017e node um\u00edr\u00e1 kv\u016fli vyu\u017eit\u00ed CPU, pam\u011bti, disk\u016f, s\u00edt\u011b a tak d\u00e1le. Nic nadp\u0159irozen\u00e9ho, ale m\u00e1me tak\u00e9 samostatn\u00fd DaemonSet agent\u016f, s nimi\u017e nap\u0159\u00edklad monitorujeme stav DNS v clusteru: hled\u00e1me zpomaluj\u00edc\u00ed pody coredns, kontrolujeme dostupnost extern\u00edch hostitel\u016f. Mohlo by se zd\u00e1t, pro\u010d se t\u00edmto zab\u00fdvat, ale p\u0159i velk\u00fdch objemech provozu je tento komponent v\u00e1\u017en\u00fdm bodem selh\u00e1n\u00ed. U\u017e d\u0159\u00edve jsem <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/domclick\/blog\/495450\/\">popsal<\/a><\/noindex>, jak jsem bojoval s v\u00fdkonem DNS v clusteru.<\/li>\n<li><b>Prometheus Operator<\/b>. Sada r\u016fzn\u00fdch exporter\u016f poskytuje velk\u00fd p\u0159ehled o v\u0161ech komponent\u00e1ch clusteru. D\u00e1le to vizualizujeme na velk\u00fdch dashboards v Grafana a pro upozorn\u011bn\u00ed pou\u017e\u00edv\u00e1me alertmanager.\n<\/li>\n<\/ul>\n<p>\nDal\u0161\u00edm u\u017eite\u010dn\u00fdm n\u00e1strojem pro n\u00e1s byl <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Nastradamus\/list-ingress\">list-ingress<\/a><\/noindex>. Napsali jsme ho pot\u00e9, co jsme n\u011bkolikr\u00e1t \u010delili situaci, kdy jeden t\u00fdm zakr\u00fdv\u00e1 sv\u00e9 cesty Ingress jin\u00e9ho t\u00fdmu, co\u017e zp\u016fsobuje chyby 50x. Nyn\u00ed p\u0159ed nasazen\u00edm na produkci v\u00fdvoj\u00e1\u0159i kontroluj\u00ed, zda nikoho nezran\u00ed, a pro m\u016fj t\u00fdm je to dobr\u00fd n\u00e1stroj pro prvotn\u00ed diagnostiku probl\u00e9m\u016f s Ingress. Je vtipn\u00e9, \u017ee byl nejprve naps\u00e1n pro administr\u00e1tory a vypadal docela \"hrub\u011b\", ale pot\u00e9, co si ho v\u00fdvoj\u00e1\u0159sk\u00e9 t\u00fdmy obl\u00edbily, se zna\u010dn\u011b zm\u011bnil a p\u0159estal vypadat jako \"administr\u00e1tor vyrobil webovou tv\u00e1\u0159 pro administr\u00e1tory\". Brzy se tohoto n\u00e1stroje vzd\u00e1me a podobn\u00e9 situace budou validov\u00e1ny je\u0161t\u011b p\u0159ed vyd\u00e1n\u00edm pipeline.<\/p>\n<h4>Zdroje t\u00fdm\u016f v \"Kube\"<\/h4>\n<p>\nNe\u017e se pust\u00edme do p\u0159\u00edklad\u016f, stoj\u00ed za to vysv\u011btlit, jak u n\u00e1s funguje p\u0159id\u011blov\u00e1n\u00ed zdroj\u016f pro <i>mikroslu\u017eby<\/i>.<\/p>\n<p>Abychom pochopili, jak\u00e9 t\u00fdmy a v jak\u00fdch mno\u017estv\u00edch pou\u017e\u00edvaj\u00ed sv\u00e9 <i>zdroje<\/i> (procesor, pam\u011b\u0165, m\u00edstn\u00ed SSD), p\u0159id\u011blujeme ka\u017ed\u00e9mu t\u00fdmu jeho vlastn\u00ed <i>namespace<\/i> v \"Kontejneru\" a omezujeme jeho maxim\u00e1ln\u00ed mo\u017enosti podle procesoru, pam\u011bti a disku, p\u0159i\u010dem\u017e p\u0159edem projedn\u00e1v\u00e1me pot\u0159eby t\u00fdm\u016f. T\u00edm p\u00e1dem jeden t\u00fdm, obecn\u011b vzato, nezablokuje nasazen\u00ed cel\u00e9ho clusteru t\u00edm, \u017ee si vyhrad\u00ed tis\u00edce jader a terabajty pam\u011bti. P\u0159\u00edstupy do namespace jsou ud\u011blov\u00e1ny prost\u0159ednictv\u00edm AD (pou\u017e\u00edv\u00e1me RBAC). Namespace a jejich limity se p\u0159id\u00e1vaj\u00ed prost\u0159ednictv\u00edm pull requestu do GIT repozit\u00e1\u0159e, a pot\u00e9 se prost\u0159ednictv\u00edm Ansible pipeline v\u0161echno automaticky rozbal\u00ed.<\/p>\n<p>P\u0159\u00edklad p\u0159id\u011blen\u00ed zdroj\u016f na t\u00fdm:<\/p>\n<pre><code class=\"go\">namespaces:\n\n  chat-team:\n    pods: 23\n    limits:\n      cpu: 11\n      memory: 20Gi\n    requests:\n      cpu: 11\n      memory: 20Gi\n<\/code><\/pre>\n<p><\/p>\n<h4>Requesty a limity<\/h4>\n<p>\nV \"Kube\" <i>Request<\/i> je mno\u017estv\u00ed garantovan\u00fdch rezervovan\u00fdch zdroj\u016f pro <i>pod<\/i> (jeden nebo v\u00edce dockerov\u00fdch kontejner\u016f) v clusteru. Limit je negarantovan\u00e9 maximum. \u010casto lze vid\u011bt na grafech, jak n\u011bjak\u00fd t\u00fdm nastavil p\u0159\u00edli\u0161 mnoho po\u017eadavk\u016f pro v\u0161echny sv\u00e9 aplikace a nem\u016f\u017ee nasadit aplikaci do \"Kontejneru\", proto\u017ee pod jejich namespace jsou ji\u017e v\u0161echny po\u017eadavky \"vy\u010derp\u00e1ny\".<\/p>\n<p>Spr\u00e1vn\u00fd zp\u016fsob, jak se z t\u00e9to situace dostat: sledovat skute\u010dnou spot\u0159ebu zdroj\u016f a srovn\u00e1vat ji s po\u017eadovan\u00fdm mno\u017estv\u00edm (Request).<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/417373d4421064f28d57a03b6bb5bf94.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/3168ad6dc8b159059b21e1177e145376.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa sn\u00edmk\u00e1ch obrazovky v\u00fd\u0161e je vid\u011bt, \u017ee \"po\u017eadovan\u00e9\" (Requested) CPU se p\u0159izp\u016fsobuj\u00ed skute\u010dn\u00e9mu po\u010dtu vl\u00e1ken, zat\u00edmco Limity mohou p\u0159ekra\u010dovat skute\u010dn\u00fd po\u010det vl\u00e1ken procesor\u016f =)<\/p>\n<p>Nyn\u00ed podrobn\u011b prozkoum\u00e1me n\u011bjak\u00fd namespace (vybral jsem namespace kube-system \u2014 syst\u00e9mov\u00fd namespace pro komponenty samotn\u00e9ho \u201eKuba\u201c) a pod\u00edv\u00e1me se na pom\u011br skute\u010dn\u011b vyu\u017eit\u00e9ho procesorov\u00e9ho \u010dasu a pam\u011bti k po\u017eadovan\u00e9mu:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/ed6160de5397b8cd39cbae172d4bb872.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe z\u0159ejm\u00e9, \u017ee pro syst\u00e9mov\u00e9 slu\u017eby je vyhrazeno mnohem v\u00edce pam\u011bti a CPU, ne\u017e kolik se skute\u010dn\u011b pou\u017e\u00edv\u00e1. V p\u0159\u00edpad\u011b kube-system to m\u00e1 opodstatn\u011bn\u00ed: doch\u00e1zelo k situac\u00edm, kdy nginx ingress controller nebo nodelocaldns b\u011bhem \u0161pi\u010dky dosahovaly maxim\u00e1ln\u00edho CPU a zab\u00edraly velmi mnoho RAM, proto je zde takov\u00e1 rezerva opr\u00e1vn\u011bn\u00e1. Krom\u011b toho se nem\u016f\u017eeme spolehnout na grafy za posledn\u00ed 3 hodiny: je dobr\u00e9 vid\u011bt historick\u00e9 metriky za del\u0161\u00ed \u010dasov\u00e9 obdob\u00ed.<\/p>\n<p>Byl vyvinut syst\u00e9m \u201edoporu\u010den\u00ed\u201c. Nap\u0159\u00edklad zde lze vid\u011bt, jak\u00e9 zdroje by bylo lep\u0161\u00ed zv\u00fd\u0161it \u201elimity\u201c (horn\u00ed povolen\u00e1 \u00farove\u0148), aby nedoch\u00e1zelo k \u201etrotlinu\u201c (throttling): momentu, kdy se ji\u017e vyu\u017eilo CPU nebo pam\u011b\u0165 b\u011bhem p\u0159id\u011blen\u00e9ho \u010dasov\u00e9ho \u00faseku a \u010dek\u00e1 na \u201erozmrazen\u00ed\u201c:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/f23aece10a585e6847bcacfe947031d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA tady jsou pody, kter\u00e9 by m\u011bly um\u00edrnit sv\u00e9 chut\u011b:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/1ec9996ce6355aeba603008d51aac209.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nO <i>trotlinu<\/i> + monitorov\u00e1n\u00ed zdroj\u016f lze napsat nejen jeden \u010dl\u00e1nek, proto se ptejte na ot\u00e1zky v koment\u00e1\u0159\u00edch. Ve zkratce mohu \u0159\u00edci, \u017ee \u00fakol automatizace podobn\u00fdch metrik je velmi obt\u00ed\u017en\u00fd a vy\u017eaduje mnoho \u010dasu a akrobatiky s \u201eokenn\u00edmi\u201c funkcemi a \u201eCTE\u201c Prometheus \/ VictoriaMetrics (tyto term\u00edny jsou vzaty do uvozovek, proto\u017ee v PromQL t\u00e9m\u011b\u0159 neexistuje nic podobn\u00e9ho a mus\u00ed se vytv\u00e1\u0159et stra\u0161ideln\u00e9 dotazy na n\u011bkolik obrazovek textu a zab\u00fdvat se jejich optimalizac\u00ed).<\/p>\n<p>V\u00fdsledkem je, \u017ee v\u00fdvoj\u00e1\u0159i maj\u00ed n\u00e1stroje pro monitorov\u00e1n\u00ed sv\u00fdch namespaces v \u201eKub\u011b\u201c, a jsou schopni sami rozhodnout, kde a kdy lze n\u011bkter\u00fdm aplikac\u00edm \u201ese\u0159\u00edznout\u201c zdroje, a kter\u00fdm pod\u016fm lze na celou noc p\u0159id\u011blit cel\u00e9 CPU.<\/p>\n<h4>Metodologie<\/h4>\n<p>\nVe firm\u011b, jak je nyn\u00ed <i>m\u00f3dn\u00ed<\/i>, se dr\u017e\u00edme DevOps- a <i>SRE<\/i>-praktik. Kdy\u017e m\u00e1 firma 1000 mikroslu\u017eeb, p\u0159ibli\u017en\u011b 350 v\u00fdvoj\u00e1\u0159\u016f a 15 administr\u00e1tor\u016f pro celou infrastrukturu, je pot\u0159eba \u201eb\u00fdt m\u00f3dn\u00ed\u201c: za v\u0161emi t\u011bmito \u201ebuzzwordy\u201c se skr\u00fdv\u00e1 nal\u00e9hav\u00e1 pot\u0159eba automatizace v\u0161eho a administr\u00e1to\u0159i nesm\u00ed b\u00fdt z\u00fa\u017een\u00fdm hrdlem v procesech.<\/p>\n<p>Jako Ops poskytujeme r\u016fzn\u00e9 metriky a dashboardy pro v\u00fdvoj\u00e1\u0159e, souvisej\u00edc\u00ed s rychlost\u00ed odpov\u011bd\u00ed slu\u017eeb a jejich chybami.<\/p>\n<p>Pou\u017e\u00edv\u00e1me metodologie jako: <noindex><a rel=\"nofollow\" href=\"https:\/\/thenewstack.io\/monitoring-microservices-red-method\/\">RED<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"http:\/\/www.brendangregg.com\/usemethod.html\">USE<\/a><\/noindex> a <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/forepaas\/distributed-monitoring-101-the-four-golden-signals-305bbbc33d35\">Zlat\u00e9 sign\u00e1ly<\/a><\/noindex>, kombinov\u00e1n\u00edm dohromady. Sna\u017e\u00edme se minimalizovat po\u010det dashboard\u016f tak, aby z jednoho pohledu bylo jasn\u00e9, kter\u00fd servis pr\u00e1v\u011b degraduje (nap\u0159\u00edklad k\u00f3dy odpov\u011bd\u00ed za sekundu, \u010das odpov\u011bdi v 99. percentilu) a tak d\u00e1le. Jakmile jsou pot\u0159ebn\u00e9 n\u011bjak\u00e9 nov\u00e9 metriky pro spole\u010dn\u00e9 dashboardy, okam\u017eit\u011b je navrhneme a p\u0159id\u00e1me.<\/p>\n<p><i>Nep\u0159ipravoval jsem grafy u\u017e m\u011bs\u00edc. Asi to je dobr\u00e9 znamen\u00ed: to znamen\u00e1, \u017ee v\u011bt\u0161ina \"p\u0159\u00e1n\u00ed\" u\u017e byla realizov\u00e1na. Ob\u010das jsem za t\u00fdden kreslil alespo\u0148 jednou za den n\u011bjak\u00fd nov\u00fd graf.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/574f1b4d2be78fb779f1771bffac51dd.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/6ab2aa05f912839b14bdbcf84faaeb3a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdsledn\u00fd v\u00fdstup je cenn\u00fd t\u00edm, \u017ee te\u010f v\u00fdvoj\u00e1\u0159i celkem z\u0159\u00eddka chod\u00ed k administr\u00e1tor\u016fm s ot\u00e1zkami \"kde se pod\u00edvat na n\u011bjakou metriku\".<\/p>\n<p>Zaveden\u00ed <i>Service Mesh<\/i> n\u00e1m u\u017e klepe na dve\u0159e a m\u011blo by v\u0161em hodn\u011b usnadnit \u017eivot, kolegov\u00e9 z Tools jsou u\u017e bl\u00edzko zaveden\u00ed abstraktn\u00edho \u201eIstio zdrav\u00e9ho \u010dlov\u011bka\u201c: \u017eivotn\u00ed cyklus ka\u017ed\u00e9ho HTTP(s) po\u017eadavku bude viditeln\u00fd v monitorov\u00e1n\u00ed a v\u017edy bude mo\u017en\u00e9 pochopit \u201ev jak\u00e9 f\u00e1zi se to v\u0161echno rozbilo\u201c b\u011bhem meziservisn\u00ed (a nejen) interakce. P\u0159ihlaste se k odb\u011bru novinek hubu spole\u010dnosti DomClick. =)<\/p>\n<h2>Podpora infrastruktury Kubernetes<\/h2>\n<p>\nHistoricky jsme pou\u017e\u00edvali upravenou verzi <i>Kubespray<\/i> \u2014 Ansible role pro nasazen\u00ed, roz\u0161i\u0159ov\u00e1n\u00ed a aktualizaci Kubernetes. V ur\u010dit\u00e9m okam\u017eiku byla z hlavn\u00ed v\u011btve odstran\u011bna podpora non-kubeadm instalac\u00ed, a proces p\u0159echodu na kubeadm nebyl navr\u017een. Nakonec spole\u010dnost Southbridge vytvo\u0159ila sv\u016fj fork (s podporou kubeadm a rychl\u00fdm opravou kritick\u00fdch probl\u00e9m\u016f). <\/p>\n<p>Proces aktualizace v\u0161ech cluster\u016f k8s vypad\u00e1 takto:<\/p>\n<ul>\n<li>Bereme <i>Kubespray<\/i> od Southbridge, porovn\u00e1me na\u0161i v\u011btev, slou\u010d\u00edme.<\/li>\n<li>Vyd\u00e1v\u00e1me aktualizaci do <i>Stress<\/i>- \u201eKuba\u201c.<\/li>\n<li>Vyd\u00e1v\u00e1me aktualizaci po jedn\u00e9 n\u00f3d\u011b (v Ansible to je \u201eserial: 1\u201c) do <i>Dev<\/i>- \u201eKuba\u201c.<\/li>\n<li>Aktualizujeme <i>Prod<\/i> v sobotu ve\u010der po jedn\u00e9 n\u00f3d\u011b.<\/li>\n<\/ul>\n<p>\nV budoucnu m\u00e1me pl\u00e1ny nahradit <i>Kubespray<\/i> n\u011b\u010d\u00edm rychlej\u0161\u00edm a p\u0159ej\u00edt na <i>kubeadm<\/i>.<\/p>\n<p>Celkem m\u00e1me t\u0159i \u201eKuby\u201c: Stress, Dev a Prod. Pl\u00e1nujeme spustit dal\u0161\u00ed (<i>hot standby<\/i>) Prod-\u201eKubu\u201c ve druh\u00e9m datov\u00e9m centru. <i>Stress<\/i> a <i>Dev<\/i> \u017eij\u00ed ve \u201evirtu\u00e1lech\u201c (oVirt pro Stress a VMWare cloud pro Dev). <i>Prod<\/i>- \u201eKuba\u201c b\u011b\u017e\u00ed na \u201ehol\u00e9m \u017eeleze\u201c (bare metal): jedn\u00e1 se o stejn\u00e9 nody s 32 CPU vl\u00e1kny, 64-128 GB pam\u011bti a 300 GB SSD RAID 10 \u2014 celkem jich je 50. T\u0159i \u201etenk\u00e9\u201c nody jsou vyhrazeny pro \u201emaster\u201c <i>Prod<\/i>- \u201eKuby\u201c: 16 GB pam\u011bti, 12 CPU vl\u00e1ken.<\/p>\n<p>Pro produkci preferujeme pou\u017e\u00edvat \u201ehol\u00e9 \u017eelezo\u201c a vyh\u00fdb\u00e1me se zbyte\u010dn\u00fdm vrstven\u00edm jako je <i>OpenStack<\/i>: nepot\u0159ebujeme \u201ehlukov\u00e9 sousedy\u201c a CPU <i>steal time<\/i>. A slo\u017eitost spr\u00e1vy se zvy\u0161uje p\u0159ibli\u017en\u011b dvakr\u00e1t v p\u0159\u00edpad\u011b in-house OpenStack.<\/p>\n<p>Pro CI\/CD \u201eKubernetes\u201c a dal\u0161\u00ed infrastrukturn\u00ed komponenty pou\u017e\u00edv\u00e1me samostatn\u00fd GIT server, Helm 3 (p\u0159echod z Helm 2 byl docela bolestiv\u00fd, ale jsme velmi r\u00e1di za tuto mo\u017enost, <i>atomic<\/i>), Jenkins, Ansible a Docker. Milujeme feature branch a nasazen\u00ed do r\u016fzn\u00fdch prost\u0159ed\u00ed z jednoho repozit\u00e1\u0159e.<\/p>\n<h3>Z\u00e1v\u011br<\/h3>\n<p><img decoding=\"async\" alt=\"Kubernetes na DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster o 1000 mikroservisech\" src=\"\/wp-content\/uploads\/2020\/08\/34ddcd1f275c63c1ad821218d16b5abb.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nTakto, v obecn\u00fdch rysech, v spole\u010dnosti DomClick vypad\u00e1 proces DevOps z pohledu provozn\u00edho technika. \u010cl\u00e1nek se uk\u00e1zal b\u00fdt m\u00e9n\u011b technick\u00fd, ne\u017e jsem o\u010dek\u00e1val: sledujte novinky DomClick na Habr\u00e9, budou tam v\u00edce \u201ehardcore\u201c \u010dl\u00e1nky o Kubernetes a nejen o n\u011bm.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/domclick\/blog\/501122\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f). \u042f \u0445\u043e\u0442\u0435\u043b \u0431\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e\u0431 \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0435 \u043d\u0430\u0448\u0438\u0445 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0432 Dev &lt;-&gt; Ops, \u043e\u0431 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u044f\u0445 \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 \u043e\u0434\u043d\u043e\u0433\u043e \u0438\u0437 \u0441\u0430\u043c\u044b\u0445 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 k8s-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 \u0432 \u0420\u043e\u0441\u0441\u0438\u0438, \u0430 \u0442\u0430\u043a\u0436\u0435 \u043e DevOps\/SRE-\u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0430\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u044f\u0435\u0442 \u043d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430. \u041a\u043e\u043c\u0430\u043d\u0434\u0430 Ops \u0412 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91200,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91199","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Kubernetes \u0432 \u0414\u043e\u043c\u041a\u043b\u0438\u043a: \u043a\u0430\u043a \u0441\u043f\u0430\u0442\u044c \u0441\u043f\u043e\u043a\u043e\u0439\u043d\u043e, \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c \u043d\u0430 1000 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-09T23:42:03+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-09T23:42:03+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kubernetes v DomClick: jak klidn\u011b sp\u00e1t, kdy\u017e spravujete cluster se 1000 mikroservisami | ProHoster","description":"Jmenuji se Viktor Jagof\u00e1rov a zab\u00fdv\u00e1m se rozvojem platformy Kubernetes ve spole\u010dnosti DomClick v roli technick\u00e9ho vedouc\u00edho v\u00fdvoje v t\u00fdmu Ops (provoz).","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Kubernetes \u0432 \u0414\u043e\u043c\u041a\u043b\u0438\u043a: \u043a\u0430\u043a \u0441\u043f\u0430\u0442\u044c \u0441\u043f\u043e\u043a\u043e\u0439\u043d\u043e, \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c \u043d\u0430 1000 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-09T23:42:03+00:00","article:modified_time":"2020-08-09T23:42:03+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91199","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:32:23","updated":"2022-09-28 06:43:59","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91199","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=91199"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91199\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/91200"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=91199"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=91199"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=91199"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}