{"id":91199,"date":"2020-08-10T01:42:03","date_gmt":"2020-08-09T23:42:03","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov"},"modified":"2020-08-10T01:42:03","modified_gmt":"2020-08-09T23:42:03","slug":"kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","title":{"rendered":"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Vol\u00e1m sa Viktor Jagofarov a zameriavam sa na rozvoj platformy Kubernetes v spolo\u010dnosti DomKlik ako technick\u00fd ved\u00faci v\u00fdvoja v t\u00edme Ops (prev\u00e1dzka). Chcel by som sa podeli\u0165 o to, ako s\u00fa na\u0161e procesy Dev  Ops zorganizovan\u00e9, o \u0161pecifik\u00e1ch prev\u00e1dzky jedn\u00e9ho z najv\u00e4\u010d\u0161\u00edch k8s klastrov v Rusku, ako aj o DevOps\/SRE praktik\u00e1ch, ktor\u00e9 na\u0161a t\u00edm aplikuje.<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/6c908104c734b36cc7a960fbe4d65e3f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h4>T\u00edm Ops<\/h4>\n<p>\nV t\u00edme Ops aktu\u00e1lne pracuje 15 \u013eud\u00ed. Traja z nich s\u00fa zodpovedn\u00ed za kancel\u00e1riu, dvaja pracuj\u00fa v inom \u010dasovom p\u00e1sme a s\u00fa k dispoz\u00edcii aj v noci. T\u00fdmto sp\u00f4sobom je v\u017edy niekto z Ops pri monitore a pripraven\u00fd reagova\u0165 na incident akejko\u013evek zlo\u017eitosti. No\u010dn\u00e9 slu\u017eby nem\u00e1me, \u010do chr\u00e1ni na\u0161u psychiku a umo\u017e\u0148uje v\u0161etk\u00fdm sa vyspa\u0165 a tr\u00e1vi\u0165 vo\u013en\u00fd \u010das nielen za po\u010d\u00edta\u010dom.<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/47051db273f116df376e64bad67c6f37.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKompetencie s\u00fa r\u00f4zne: sie\u0165ov\u00ed \u0161pecialisti, DBA, odborn\u00edci na ELK stack, Kubernetes administr\u00e1tori\/v\u00fdvoj\u00e1ri, odborn\u00edci na monitorovanie, virtualiz\u00e1ciu, hardv\u00e9r a pod. Spojuje n\u00e1s jedno \u2014 ka\u017ed\u00fd z n\u00e1s m\u00f4\u017ee v ur\u010ditej miere nahradi\u0165 ak\u00e9hoko\u013evek in\u00e9ho \u010dlena t\u00edmu: napr\u00edklad prida\u0165 nov\u00e9 uzly do k8s klastru, aktualizova\u0165 PostgreSQL, nap\u00edsa\u0165 CI\/CD pipeline + Ansible, automatizova\u0165 nie\u010do v Pythone\/Bashi\/Go, pripoji\u0165 hardv\u00e9r v datovom centre. Siln\u00e9 kompetencie v ur\u010ditej oblasti nebr\u00e1nia zmeni\u0165 smerovanie a za\u010da\u0165 sa rozv\u00edja\u0165 v inej oblasti. Napr\u00edklad, do spolo\u010dnosti som nastupoval ako \u0161pecialista na PostgreSQL, ale teraz je mojou hlavnou zodpovednos\u0165ou spr\u00e1va klastrov Kubernetes. V t\u00edme je ka\u017ed\u00fd rast v\u00edtan\u00fd a ve\u013emi je rozvinut\u00fd pocit spolo\u010dnej podpory.<\/p>\n<p>Mimochodom, h\u013ead\u00e1me. Po\u017eiadavky na kandid\u00e1tov s\u00fa celkom \u0161tandardn\u00e9. Osobne je pre m\u0148a d\u00f4le\u017eit\u00e9, aby sa \u010dlovek zapadol do kolekt\u00edvu, bol nekonfliktn\u00fd, ale tie\u017e vedel obh\u00e1ji\u0165 svoj n\u00e1zor, mal t\u00fa\u017ebu rozv\u00edja\u0165 sa a neb\u00e1l sa sk\u00fa\u0161a\u0165 nie\u010do nov\u00e9, navrhoval svoje n\u00e1pady. Rovnako s\u00fa potrebn\u00e9 zru\u010dnosti programovania v skriptovac\u00edch jazykoch, znalos\u0165 z\u00e1sad Linuxu a anglick\u00e9ho jazyka. Angli\u010dtina je potrebn\u00e1 len na to, aby \u010dlovek v pr\u00edpade probl\u00e9mu mohol r\u00fdchlo n\u00e1js\u0165 jeho rie\u0161enie za 10 sek\u00fand, nie za 10 min\u00fat. So \u0161pecialistami s hlbok\u00fdmi znalos\u0165ami Linuxu je teraz ve\u013emi \u0165a\u017ek\u00e9: je to smie\u0161ne, ale dvaja z troch kandid\u00e1tov nevedia odpoveda\u0165 na ot\u00e1zku \u201e\u010co je to Load Average? Z \u010doho sa sklad\u00e1?\u201c, a ot\u00e1zku \u201eAko z\u00edska\u0165 core dump z C-programu\u201c pova\u017euj\u00fa za nie\u010do z r\u00ed\u0161e superhrdinov\u2026 alebo dinosaurov. S t\u00fdm sa mus\u00edme zmieri\u0165, preto\u017ee v\u00e4\u010d\u0161ina \u013eud\u00ed m\u00e1 zvy\u010dajne silne vyvinut\u00e9 in\u00e9 kompetencie, a \u201elinuxu\u201c sa nau\u010d\u00edme. Odpove\u010f na ot\u00e1zku \u201epre\u010do to v\u0161etko potrebujeme vedie\u0165 DevOps-in\u017einierovi v s\u00fa\u010dasnom svete cloudov\u201c mus\u00edme necha\u0165 mimo \u010dl\u00e1nku, ale ak by sme mali poveda\u0165 to tromi slovami: v\u0161etko toto je potrebn\u00e9.<\/p>\n<h4>T\u00edm Tools<\/h4>\n<p>\nV automatiz\u00e1cii m\u00e1 ve\u013ek\u00fa \u00falohu t\u00edm Tools. Ich hlavnou \u00falohou je vytv\u00e1ranie pohodln\u00fdch grafick\u00fdch a CLI n\u00e1strojov pre v\u00fdvoj\u00e1rov. Napr\u00edklad na\u0161a intern\u00e1 aplik\u00e1cia Confer umo\u017e\u0148uje doslova dvoma kliknutiami my\u0161i nasadi\u0165 aplik\u00e1ciu do Kubernetes, nastavi\u0165 jej zdroje, k\u013e\u00fa\u010de z trezoru at\u010f. Predt\u00fdm sme pou\u017e\u00edvali Jenkins + Helm 2, ale museli sme vyvin\u00fa\u0165 vlastn\u00fd n\u00e1stroj, aby sme eliminovali kop\u00edrovanie a priniesli jednotnos\u0165 do \u017eivotn\u00e9ho cyklu softv\u00e9ru.<\/p>\n<p>T\u00edm Ops nep\u00ed\u0161e pipeline za v\u00fdvoj\u00e1rov, ale m\u00f4\u017ee poskytn\u00fa\u0165 poradenstvo o ak\u00fdchko\u013evek ot\u00e1zkach pri ich p\u00edsan\u00ed (niektor\u00ed e\u0161te st\u00e1le pou\u017e\u00edvaj\u00fa Helm 3).<\/p>\n<h4>DevOps<\/h4>\n<p>\n\u010co sa t\u00fdka DevOps, vid\u00edme to takto:<\/p>\n<p>T\u00edmy Dev p\u00ed\u0161u k\u00f3d, nasadzuj\u00fa ho cez Confer do dev -&gt; qa\/stage -&gt; prod. Zodpovednos\u0165 za to, aby k\u00f3d nezastavoval a nevyhadzoval chyby, le\u017e\u00ed na t\u00edmok Dev a Ops. Po\u010das d\u0148a by na incident so svojou aplik\u00e1ciou mal prim\u00e1rne reagova\u0165 dispe\u010der z t\u00edmu Ops, a v noci a ve\u010der by mal dispe\u010der administr\u00e1tor (Ops) prebudi\u0165 dispe\u010dera v\u00fdvoj\u00e1ra, ak si je ist\u00fd, \u017ee probl\u00e9m nie je v infra\u0161trukt\u00fare. V\u0161etky metriky a alarmy v monitorovan\u00ed sa objavuj\u00fa automaticky alebo poloautomatically.<\/p>\n<p>Z\u00f3na zodpovednosti Ops za\u010d\u00edna okamihom nasadenia aplik\u00e1cie do produkcie, no zodpovednos\u0165 Dev sa t\u00fdm nekon\u010d\u00ed \u2014 rob\u00edme jednu vec a sme na jednej lodi.<\/p>\n<p>V\u00fdvoj\u00e1ri radia administr\u00e1torom, ak potrebuj\u00fa pomoc pri p\u00edsan\u00ed administr\u00e1torsk\u00e9ho mikroslu\u017eby (napr\u00edklad, Go backend + HTML5), a administr\u00e1tori radia v\u00fdvoj\u00e1rom oh\u013eadom ak\u00fdchko\u013evek ot\u00e1zok t\u00fdkaj\u00facich sa infra\u0161trukt\u00fary alebo ot\u00e1zok spojen\u00fdch s k8s.<\/p>\n<p>Mimochodom, nem\u00e1me \u017eiadny monolit, len mikroslu\u017eby. Ich po\u010det sa zatia\u013e pohybuje medzi 900 a 1000 v produk\u010dnom k8s klastre, ak mer\u00e1me pod\u013ea mno\u017estva. <i>nasadenia<\/i>Po\u010det podov sa pohybuje medzi 1700 a 2000. V produk\u010dnom klastre je teraz okolo 2000 podov.<\/p>\n<p>Presn\u00e9 \u010d\u00edsla nem\u00f4\u017eem uvies\u0165, preto\u017ee sledujeme zbyto\u010dn\u00e9 mikroslu\u017eby a odstra\u0148ujeme ich v polautomatickom re\u017eime. Sledovanie zbyto\u010dn\u00fdch ent\u00edt v k8s n\u00e1m pom\u00e1ha <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Nastradamus\/useless-operator\">useless-operator<\/a><\/noindex>, \u010do n\u00e1m v\u00fdrazne \u0161etr\u00ed zdroje a peniaze.<\/p>\n<h2> Spr\u00e1va resursov <\/h2>\n<p><\/p>\n<h4> Monitorovanie <\/h4>\n<p>\nKame\u0148om \u00farazu pri prev\u00e1dzke ve\u013ek\u00e9ho klastra sa st\u00e1va spr\u00e1vne vybudovan\u00fd a informat\u00edvny monitoring. Zatia\u013e sme nena\u0161li univerz\u00e1lne rie\u0161enie, ktor\u00e9 by pokrylo 100 % v\u0161etk\u00fdch po\u017eiadaviek na monitoring, a preto ob\u010das vytv\u00e1rame r\u00f4zne prisp\u00f4soben\u00e9 rie\u0161enia v tejto oblasti.<\/p>\n<ul>\n<li><b>Zabbix<\/b>Star\u00fd dobr\u00fd monitoring, ktor\u00fd je prim\u00e1rne ur\u010den\u00fd na sledovanie celkov\u00e9ho stavu infra\u0161trukt\u00fary. Hovor\u00ed n\u00e1m, ke\u010f node umiera z h\u013eadiska CPU, pam\u00e4te, diskov, siete a podobne. Ni\u010d nadprirodzen\u00e9, ale m\u00e1me aj samostatn\u00fd DaemonSet agentov, pomocou ktor\u00fdch napr\u00edklad sledujeme stav DNS v klastri: h\u013ead\u00e1me pomal\u00e9 pody coredns, overujeme dostupnos\u0165 extern\u00fdch hostite\u013eov. Mohlo by sa zda\u0165, pre\u010do si s t\u00fdm robi\u0165 starosti, ale pri ve\u013ek\u00fdch objemoch prev\u00e1dzky je tento komponent v\u00e1\u017enym bodom zlyhania. U\u017e som predt\u00fdm <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/domclick\/blog\/495450\/\">op\u00edsal<\/a><\/noindex>, ako som bojoval s v\u00fdkonom DNS v klastri.<\/li>\n<li><b>Prometheus Operator<\/b>. Sada r\u00f4znych export\u00e9rov poskytuje dobr\u00fd preh\u013ead o v\u0161etk\u00fdch komponentoch klastra. Potom to v\u0161etko vizualizujeme na ve\u013ek\u00fdch dashboardoch v Grafana a na upozornenia pou\u017e\u00edvame alertmanager.\n<\/li>\n<\/ul>\n<p>\n\u010eal\u0161\u00edm u\u017eito\u010dn\u00fdm n\u00e1strojom pre n\u00e1s sa stal <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Nastradamus\/list-ingress\">list-ingress<\/a><\/noindex>. Nap\u00edsali sme ho po tom, \u010do sme sa nieko\u013ekokr\u00e1t stretli so situ\u00e1ciou, ke\u010f jedna t\u00edm zablokoval svojimi cestami Ingress druh\u00e9ho t\u00edmu, \u010do sp\u00f4sobilo chyby 50x. Teraz pred nasaden\u00edm na produkciu v\u00fdvoj\u00e1ri kontroluj\u00fa, \u017ee nikoho nezasiahnu, a pre m\u00f4j t\u00edm je to dobr\u00fd n\u00e1stroj na prim\u00e1rnu diagnostiku probl\u00e9mov s Ingressami. Z\u00e1bavn\u00e9 je, \u017ee bol p\u00f4vodne nap\u00edsan\u00fd pre spr\u00e1vcov a vyzeral dos\u0165 \u201eneobratne\u201c, ale po tom, ako si n\u00e1stroj ob\u013e\u00fabili v\u00fdvojov\u00e9 t\u00edmy, sa v\u00fdrazne transformoval a prestal vyzera\u0165 ako \u201eadmin urobil webov\u00e9 rozhranie pre adminov\u201c. \u010coskoro sa od tohto n\u00e1stroja vzd\u00e1me a podobn\u00e9 situ\u00e1cie bud\u00fa validovan\u00e9 e\u0161te pred vyrolovan\u00edm pipeline.<\/p>\n<h4>Zdroje t\u00edmov v \u201eKube\u201c<\/h4>\n<p>\nPredt\u00fdm, ne\u017e sa pust\u00edme do pr\u00edkladov, stoj\u00ed za to vysvetli\u0165, ako u n\u00e1s funguje pride\u013eovanie zdrojov pre <i>mikroslu\u017eby<\/i>.<\/p>\n<p>Aby bolo jasn\u00e9, ak\u00e9 t\u00edmy a v ak\u00fdch mno\u017estv\u00e1ch vyu\u017e\u00edvaj\u00fa svoje <i>zdroje<\/i> (procesor, pam\u00e4\u0165, lok\u00e1lny SSD), pride\u013eujeme na ka\u017ed\u00fd t\u00edm svoj <i>namespace<\/i> v \u201eKube\u201c a obmedzujeme jeho maxim\u00e1lne mo\u017enosti pod\u013ea procesora, pam\u00e4te a disku, predt\u00fdm si preberieme potreby t\u00edmov. Zodpovedaj\u00facim sp\u00f4sobom jedna t\u00edm, vo v\u0161eobecnosti, nezablokuje nasadenie cel\u00e9ho klastru, ke\u010f si vyhrad\u00ed tis\u00edce jadier a terabajty pam\u00e4te. Pr\u00edstupy do namespace sa poskytuj\u00fa cez AD (pou\u017e\u00edvame RBAC). Namespace a ich limity sa prid\u00e1vaj\u00fa cez pull request v GIT repozit\u00e1ri, a potom sa automaticky distribuj\u00fa cez Ansible pipeline.<\/p>\n<p>Pr\u00edklad pride\u013eovania zdrojov pre t\u00edm:<\/p>\n<pre><code class=\"go\">namespaces:\n\n  chat-team:\n    pods: 23\n    limits:\n      cpu: 11\n      memory: 20Gi\n    requests:\n      cpu: 11\n      memory: 20Gi\n<\/code><\/pre>\n<p><\/p>\n<h4>Requesty a limity<\/h4>\n<p>\nV \u201eKube\u201c <i>Po\u017eiadavka<\/i> \u2014 to je mno\u017estvo garantovane vyhraden\u00fdch zdrojov pre <i>pod<\/i> (jeden alebo viac doker kontejnerov) v klastru. Limit je negarantovan\u00fd maximum. \u010casto m\u00f4\u017eeme vidie\u0165 na grafoch, ako si nejak\u00fd t\u00edm nastavil pr\u00edli\u0161 ve\u013ea po\u017eiadaviek pre v\u0161etky svoje aplik\u00e1cie a nem\u00f4\u017ee nasadi\u0165 aplik\u00e1ciu do \u201eKubu\u201c, preto\u017ee pod ich namespace s\u00fa v\u0161etky po\u017eiadavky u\u017e \u201eminut\u00e9\u201c.<\/p>\n<p>Spr\u00e1vny sp\u00f4sob, ako z takej situ\u00e1cie vyjs\u0165: sledova\u0165 re\u00e1lnu spotrebu zdrojov a porovna\u0165 ju s po\u017eadovan\u00fdm mno\u017estvom (Request).<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/417373d4421064f28d57a03b6bb5bf94.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/3168ad6dc8b159059b21e1177e145376.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa sn\u00edmkach obrazovky vy\u0161\u0161ie je vidie\u0165, \u017ee \u201epo\u017eadovan\u00e9\u201c (Requested) CPU sa prisp\u00f4sobuj\u00fa re\u00e1lnemu po\u010dtu vl\u00e1kien, a Limity m\u00f4\u017eu presahova\u0165 re\u00e1lny po\u010det procesorov\u00fdch vl\u00e1kien =)<\/p>\n<p>Teraz si podrobne rozoberieme nejak\u00fd namespace (vybral som namespace kube-system \u2014 syst\u00e9mov\u00fd namespace pre komponenty samotn\u00e9ho \u00abKuba\u00bb) a pozrieme sa na pomer skuto\u010dne pou\u017eit\u00e9ho procesorov\u00e9ho \u010dasu a pam\u00e4te k po\u017eadovan\u00e9mu:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/ed6160de5397b8cd39cbae172d4bb872.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe zrejm\u00e9, \u017ee pam\u00e4\u0165 a CPU rezervovan\u00e9 pre syst\u00e9mov\u00e9 slu\u017eby s\u00fa ove\u013ea vy\u0161\u0161ie, ne\u017e je skuto\u010dne pou\u017eit\u00e9. V pr\u00edpade kube-system je to ospravedlnite\u013en\u00e9: ob\u010das sa stalo, \u017ee nginx ingress controller alebo nodelocaldns na vrchole spotrebov\u00e1vali CPU a zaberali ve\u013emi ve\u013ea RAM, tak\u017ee tu je tak\u00e1 rezerva ospravedlnite\u013en\u00e1. Okrem toho, nem\u00f4\u017eeme sa spolieha\u0165 na grafy za posledn\u00e9 3 hodiny: ide\u00e1lne je vidie\u0165 historick\u00e9 metriky za dlh\u0161ie \u010dasov\u00e9 obdobie.<\/p>\n<p>Bola vyvinut\u00e1 s\u00fastava \u201eodpor\u00fa\u010dan\u00ed\u201c. Napr\u00edklad tu m\u00f4\u017eeme vidie\u0165, ak\u00e9 zdroje by bolo lep\u0161ie zv\u00fd\u0161i\u0165 \u201elimity\u201c (horn\u00e1 povolen\u00e1 hranica), aby nedoch\u00e1dzalo k \u201ethrottlingu\u201c: momentu, ke\u010f u\u017e boli minul\u00e9 CPU alebo pam\u00e4\u0165 v pridelenom \u010dasovom kvante a \u010dak\u00e1, k\u00fdm ho \u201erozmrazia\u201c:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/f23aece10a585e6847bcacfe947031d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA tu s\u00fa pod, ktor\u00fdm by sa mali obmedzi\u0165 apetity:<\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/1ec9996ce6355aeba603008d51aac209.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nO <i>throttling<\/i> + monitoring zdrojov je mo\u017en\u00e9 nap\u00edsa\u0165 nieko\u013eko \u010dl\u00e1nkov, tak\u017ee sa p\u00fdtajte v koment\u00e1roch. V skratke m\u00f4\u017eem poveda\u0165, \u017ee \u00faloha automatiz\u00e1cie tak\u00fdchto metr\u00edk je dos\u0165 zlo\u017eit\u00e1 a vy\u017eaduje ve\u013ea \u010dasu a akrobatick\u00fdch k\u00faskov s \u201eokenn\u00fdmi\u201c funkciami a \u201eCTE\u201c Prometheus \/ VictoriaMetrics (tieto pojmy s\u00fa v \u00favodzovk\u00e1ch, preto\u017ee v PromQL skoro nie je ni\u010d podobn\u00e9ho a mus\u00edme zostavova\u0165 hrozn\u00e9 dotazy na nieko\u013eko obrazoviek textu a zaobera\u0165 sa ich optimaliz\u00e1ciou).<\/p>\n<p>Nakoniec maj\u00fa v\u00fdvoj\u00e1ri n\u00e1stroje na monitoring svojich namespaces v \u201eKube\u201c a s\u00fa schopn\u00ed si sami vybera\u0165, kde a kedy pre ak\u00e9 aplik\u00e1cie m\u00f4\u017eu \u201estrihn\u00fa\u0165\u201c zdroje, a ktor\u00fdm podom m\u00f4\u017eu pre cel\u00fa noc odovzda\u0165 cel\u00fd CPU.<\/p>\n<h4>Metodol\u00f3gie<\/h4>\n<p>\nVo firme, ako je teraz <i>m\u00f3dne<\/i>, dodr\u017eiavame DevOps- a <i>SRE<\/i>-praktiky. Ke\u010f m\u00e1 firma 1000 mikroslu\u017eieb, pribli\u017ene 350 v\u00fdvoj\u00e1rov a 15 administr\u00e1torov na cel\u00fa infra\u0161trukt\u00faru, mus\u00ed sa \u201eby\u0165 m\u00f3dnym\u201d: za v\u0161etk\u00fdmi t\u00fdmito \u201ebuzzwordmi\u201d sa skr\u00fdva naliehav\u00e1 potreba automatiz\u00e1cie v\u0161etk\u00e9ho a v\u0161etk\u00e9ho, a administr\u00e1tori by nemali by\u0165 \u00fazkym miestom v procesoch.<\/p>\n<p>Ako Ops poskytujeme r\u00f4zne metriky a dashboards pre v\u00fdvoj\u00e1rov, t\u00fdkaj\u00face sa r\u00fdchlosti odpovede slu\u017eieb a ich ch\u00fdb.<\/p>\n<p>Pou\u017e\u00edvame tak\u00e9 metodol\u00f3gie ako: <noindex><a rel=\"nofollow\" href=\"https:\/\/thenewstack.io\/monitoring-microservices-red-method\/\">RED<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"http:\/\/www.brendangregg.com\/usemethod.html\">USE<\/a><\/noindex> a <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/forepaas\/distributed-monitoring-101-the-four-golden-signals-305bbbc33d35\">Golden Signals<\/a><\/noindex>, kombinovan\u00edm ich spolu. Sna\u017e\u00edme sa minimalizova\u0165 mno\u017estvo dashboardov tak, aby z jedn\u00e9ho poh\u013eadu bolo jasn\u00e9, ktor\u00fd servis teraz degraduje (napr\u00edklad k\u00f3dy odpoved\u00ed za sekundu, \u010das odpoved\u00ed pod\u013ea 99. percentilu) a tak \u010falej. Ako n\u00e1hle sa objavia nejak\u00e9 nov\u00e9 metriky pre spolo\u010dn\u00e9 dashboardy, okam\u017eite ich vytvor\u00edme a prid\u00e1me.<\/p>\n<p><i>Nerobil som grafy u\u017e mesiac. Asi je to dobr\u00e9 znamenie: to znamen\u00e1, \u017ee v\u00e4\u010d\u0161ina \u201epo\u017eiadaviek\u201c u\u017e bola splnen\u00e1. Niekedy sa stalo, \u017ee som aspo\u0148 raz denne robil nejak\u00fd nov\u00fd graf.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/574f1b4d2be78fb779f1771bffac51dd.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/6ab2aa05f912839b14bdbcf84faaeb3a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdsledok je cenn\u00fd t\u00fdm, \u017ee teraz v\u00fdvoj\u00e1ri dos\u0165 zriedka chodia za administr\u00e1tormi s ot\u00e1zkami \u201ekde n\u00e1js\u0165 nejak\u00fa metriku\u201c.<\/p>\n<p>Implement\u00e1cia <i>Service Mesh<\/i> nie je \u010faleko a malo by to v\u00fdrazne u\u013eah\u010di\u0165 \u017eivot v\u0161etk\u00fdm; kolegovia z Tools u\u017e s\u00fa bl\u00edzko k implement\u00e1cii abstrahovan\u00e9ho \u201eIstio zdrav\u00e9ho \u010dloveka\u201c: \u017eivotn\u00fd cyklus ka\u017ed\u00e9ho HTTP(s) po\u017eiadavku bude vidite\u013en\u00fd v monitoringu a v\u017edy bude mo\u017en\u00e9 pochopi\u0165 \u201ev akom \u0161t\u00e1diu sa v\u0161etko pokazilo\u201c pri medzi-service (a nielen) interakci\u00e1ch. Prihl\u00e1ste sa na odber noviniek z hubu spolo\u010dnosti DomClick. =)<\/p>\n<h2>Podpora infra\u0161trukt\u00fary Kubernetes<\/h2>\n<p>\nHistoricky sa to takto vyvinulo, \u017ee pou\u017e\u00edvame upraven\u00fa verziu <i>Kubespray<\/i> \u2014 Ansible-\u00faloha na nasadenie, roz\u0161\u00edrenie a aktualiz\u00e1ciu Kubernetes. V nejakom momente bola z hlavnej vetvy odstr\u00e1nen\u00e1 podpora non-kubeadm in\u0161tal\u00e1ci\u00ed a proces prechodu na kubeadm nebol navrhnut\u00fd. Nakoniec spolo\u010dnos\u0165 Southbridge urobila svoj fork (s podporou kubeadm a r\u00fdchlym opraven\u00edm kritick\u00fdch probl\u00e9mov). <\/p>\n<p>Proces aktualiz\u00e1cie v\u0161etk\u00fdch klastrov k8s vyzer\u00e1 takto:<\/p>\n<ul>\n<li>Bereme <i>Kubespray<\/i> od Southbridge, porovn\u00e1me s na\u0161ou vetvou, a zl\u00fa\u010dime.<\/li>\n<li>Zverejn\u00edme aktualiz\u00e1ciu v <i>Stress<\/i>-\u201eKubo\u201c.<\/li>\n<li>Zverej\u0148ujeme aktualiz\u00e1ciu po jednej node (v Ansible je to \u201eserial: 1\u201c) v <i>Dev<\/i>-\u201eKubo\u201c.<\/li>\n<li>Aktualizujeme <i>Prod<\/i> v sobotu ve\u010der po jednej node.<\/li>\n<\/ul>\n<p>\nV bud\u00facnosti pl\u00e1nujeme nahradi\u0165 <i>Kubespray<\/i> nie\u010d\u00edm r\u00fdchlej\u0161\u00edm a prejs\u0165 na <i>kubeadm<\/i>.<\/p>\n<p>M\u00e1me tri \u201eKuba\u201c: Stress, Dev a Prod. Pl\u00e1nujeme spusti\u0165 \u010fal\u0161ie (<i>hot standby<\/i>) Prod-\u201eKubo\u201c v druhom DC. <i>Stress<\/i> a <i>Dev<\/i> \u017eij\u00fa vo \u201evirtu\u00e1lnych strojoch\u201c (oVirt pre Stress a VMWare cloud pre Dev). <i>Prod<\/i>-\u201eKubo\u201c \u017eije na \u201eholom \u017eeleze\u201c (bare metal): s\u00fa to rovnak\u00e9 nody s 32 CPU vl\u00e1knami, 64-128 GB pam\u00e4ti a 300 GB SSD RAID 10 \u2014 je ich celkom 50. Tri \u201etenk\u00e9\u201c nody s\u00fa vyhraden\u00e9 pre \u201emajstrou\u201c <i>Prod<\/i>-\u201eKuba\u201c: 16 GB pam\u00e4ti, 12 CPU vl\u00e1knami.<\/p>\n<p>Pre produkciu preferujeme pou\u017e\u00edva\u0165 \u201ehol\u00e9 \u017eelezo\u201c a vyh\u00fdbame sa zbyto\u010dn\u00fdm vrstv\u00e1m, ako s\u00fa <i>OpenStack<\/i>: nepotrebujeme \u201ehlu\u010dn\u00fdch susedov\u201c a CPU <i>steal time<\/i>A navy a zlo\u017eitost spr\u00e1vy se p\u0159ibli\u017en\u011b zdvojn\u00e1sob\u00ed v p\u0159\u00edpad\u011b in-house OpenStack.<\/p>\n<p>Pro CI\/CD kubov\u00fdch a dal\u0161\u00edch infrastrukturn\u00edch komponent pou\u017e\u00edv\u00e1me samostatn\u00fd GIT server, Helm 3 (p\u0159e\u0161li jsme pom\u011brn\u011b bolestiv\u011b z Helmu 2, ale jsme velmi r\u00e1di za tuto mo\u017enost <i>atomic<\/i>), Jenkins, Ansible a Docker. M\u00e1me r\u00e1di feature branches a nasazen\u00ed do r\u016fzn\u00fdch prost\u0159ed\u00ed z jednoho repozit\u00e1\u0159e.<\/p>\n<h3>Z\u00e1ver<\/h3>\n<p><img decoding=\"async\" alt=\"Kubernetes v DomKlik: ako pokojne sp\u00e1va\u0165 pri spr\u00e1ve klasteru s 1000 mikroslu\u017ebami\" src=\"\/wp-content\/uploads\/2020\/08\/34ddcd1f275c63c1ad821218d16b5abb.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nTakto, v obecn\u00fdch rysech, v spole\u010dnosti DomClick vypad\u00e1 proces DevOps z pohledu provozn\u00edho in\u017een\u00fdra. \u010cl\u00e1nek se uk\u00e1zal b\u00fdt m\u00e9n\u011b technick\u00fd, ne\u017e jsem o\u010dek\u00e1val: proto sledujte novinky DomClick na Habr\u00e9: budou drsn\u011bj\u0161\u00ed \u010dl\u00e1nky o Kubernetes a nejen o n\u011bm.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/domclick\/blog\/501122\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f). \u042f \u0445\u043e\u0442\u0435\u043b \u0431\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e\u0431 \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0435 \u043d\u0430\u0448\u0438\u0445 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0432 Dev &lt;-&gt; Ops, \u043e\u0431 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u044f\u0445 \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 \u043e\u0434\u043d\u043e\u0433\u043e \u0438\u0437 \u0441\u0430\u043c\u044b\u0445 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 k8s-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 \u0432 \u0420\u043e\u0441\u0441\u0438\u0438, \u0430 \u0442\u0430\u043a\u0436\u0435 \u043e DevOps\/SRE-\u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0430\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u044f\u0435\u0442 \u043d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430. \u041a\u043e\u043c\u0430\u043d\u0434\u0430 Ops \u0412 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91200,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91199","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Kubernetes \u0432 \u0414\u043e\u043c\u041a\u043b\u0438\u043a: \u043a\u0430\u043a \u0441\u043f\u0430\u0442\u044c \u0441\u043f\u043e\u043a\u043e\u0439\u043d\u043e, \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c \u043d\u0430 1000 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-09T23:42:03+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-09T23:42:03+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kubernetes v DomClick: jak klidn\u011b sp\u00e1t p\u0159i spr\u00e1v\u011b clustera s 1000 mikroservisy | ProHoster","description":"Jmenuji se Viktor Jagofarov a zab\u00fdv\u00e1m se rozvojem Kubernetes platformy ve spole\u010dnosti DomClick v pozici technick\u00e9ho vedouc\u00edho v\u00fdvoje v t\u00fdmu Ops (provoz).","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Kubernetes \u0432 \u0414\u043e\u043c\u041a\u043b\u0438\u043a: \u043a\u0430\u043a \u0441\u043f\u0430\u0442\u044c \u0441\u043f\u043e\u043a\u043e\u0439\u043d\u043e, \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c \u043d\u0430 1000 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0412\u0438\u043a\u0442\u043e\u0440 \u042f\u0433\u043e\u0444\u0430\u0440\u043e\u0432, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u0435\u043c Kubernetes-\u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0432 \u0434\u043e\u043b\u0436\u043d\u043e\u0441\u0442\u0438 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 Ops (\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f).","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-09T23:42:03+00:00","article:modified_time":"2020-08-09T23:42:03+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91199","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:32:23","updated":"2022-09-28 06:43:59","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/91199","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=91199"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/91199\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/91200"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=91199"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=91199"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=91199"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}