{"id":77693,"date":"2020-04-13T01:42:39","date_gmt":"2020-04-12T23:42:39","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes"},"modified":"2020-04-13T01:42:39","modified_gmt":"2020-04-12T23:42:39","slug":"cpu-limity-i-agressivnyj-trottling-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","title":{"rendered":"CPU limity a agresivn\u00ed throttling v Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Pozn. p\u0159ek.<\/b>: tento pou\u010dn\u00fd p\u0159\u00edb\u011bh Omio \u2014 evropsk\u00e9ho cestovn\u00edho agreg\u00e1toru \u2014 prov\u00e1d\u00ed \u010dten\u00e1\u0159e od z\u00e1kladn\u00ed teorie po fascinuj\u00edc\u00ed praktick\u00e9 detaily konfigurace Kubernetes. Sezn\u00e1men\u00ed s t\u011bmito p\u0159\u00edpady pom\u00e1h\u00e1 nejen roz\u0161i\u0159ovat obzory, ale tak\u00e9 p\u0159edch\u00e1zet netrivi\u00e1ln\u00edm probl\u00e9m\u016fm.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/1175c9df746e43b5a7d81476164929db.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSetkali jste se n\u011bkdy s t\u00edm, \u017ee aplikace \"zamrzla\", p\u0159estala reagovat na dotazy o stavu (health checky) a nedok\u00e1zali jste pochopit p\u0159\u00ed\u010dinu takov\u00e9ho chov\u00e1n\u00ed? Jedn\u00edm z mo\u017en\u00fdch vysv\u011btlen\u00ed jsou limity kv\u00f3t na zdroje CPU. O tom bude tento \u010dl\u00e1nek.<\/p>\n<p><b>TL;DR:<br \/>\nD\u016frazn\u011b doporu\u010dujeme vzd\u00e1t se limit\u016f CPU v Kubernetes (nebo deaktivovat kv\u00f3ty CFS v Kubeletu), pokud pou\u017e\u00edv\u00e1te verzi j\u00e1dra Linuxu s chybou CFS-kv\u00f3t. <noindex><a rel=\"nofollow\" href=\"https:\/\/bugzilla.kernel.org\/show_bug.cgi?id=198197\">je zde<\/a><\/noindex> je to v\u00e1\u017en\u00e1 a <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/67577\">dob\u0159e zn\u00e1m\u00e1<\/a><\/noindex> chyba, kter\u00e1 vede k nadm\u011brn\u00e9mu zpo\u017e\u010fov\u00e1n\u00ed a latenci.<\/b>.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>V Omio <b>je cel\u00e1 infrastruktura \u0159\u00edzena Kubernetes.<\/b>. V\u0161echny na\u0161e stateful a stateless z\u00e1t\u011b\u017ee b\u011b\u017e\u00ed v\u00fdhradn\u011b na Kubernetes (pou\u017e\u00edv\u00e1me Google Kubernetes Engine). V posledn\u00edch \u0161esti m\u011bs\u00edc\u00edch jsme za\u010dali pozorovat n\u00e1hodn\u00e9 zpo\u017e\u010fov\u00e1n\u00ed. Aplikace zamrzaj\u00ed nebo p\u0159est\u00e1vaj\u00ed reagovat na health checky, ztr\u00e1cej\u00ed spojen\u00ed se s\u00edt\u00ed atd. Takov\u00e9 chov\u00e1n\u00ed n\u00e1s dlouho tr\u00e1pilo a nakonec jsme se rozhodli probl\u00e9m \u0159e\u0161it zbl\u00edzka.<\/p>\n<p>Shrnut\u00ed \u010dl\u00e1nku:<\/p>\n<ul>\n<li> N\u011bco m\u00e1lo o kontejnerech a Kubernetes;<\/li>\n<li> Jak jsou nastaveny po\u017eadavky (requests) a limity (limits);<\/li>\n<li> Jak limit CPU funguje v prost\u0159ed\u00edch s v\u00edce j\u00e1dry;<\/li>\n<li> Jak sledovat throttling CPU;<\/li>\n<li> \u0158e\u0161en\u00ed probl\u00e9mu a nuance.<\/li>\n<\/ul>\n<p><\/p>\n<h2>N\u011bco m\u00e1lo o kontejnerech a Kubernetes<\/h2>\n<p>\nKubernetes je v podstat\u011b modern\u00ed standard ve sv\u011bt\u011b infrastruktury. Jeho hlavn\u00edm \u00fakolem je orchestraci kontejner\u016f.<\/p>\n<h3>Kontejnery<\/h3>\n<p>\nV minulosti jsme museli vytv\u00e1\u0159et artefakty jako Java JARy\/WARy, Python Eggy nebo spustiteln\u00e9 soubory, kter\u00e9 jsme mohli pot\u00e9 spou\u0161t\u011bt na serverech. Nicm\u00e9n\u011b, abychom je dostali do funk\u010dn\u00edho stavu, museli jsme vykonat dodate\u010dnou pr\u00e1ci: instalovat runtime (Java\/Python), um\u00edstit pot\u0159ebn\u00e9 soubory na spr\u00e1vn\u00e1 m\u00edsta, zajistit kompatibilitu s konkr\u00e9tn\u00ed verz\u00ed opera\u010dn\u00edho syst\u00e9mu atd. Jin\u00fdmi slovy, museli jsme v\u011bnovat velkou pozornost spr\u00e1v\u011b konfigurace (co\u017e bylo \u010dasto p\u0159\u00ed\u010dinou spor\u016f mezi v\u00fdvoj\u00e1\u0159i a syst\u00e9mov\u00fdmi administr\u00e1tory).<\/p>\n<p><b>Kontejnery v\u0161echno zm\u011bnily.<\/b> Nyn\u00ed artefaktem p\u0159edstavuje kontejnerov\u00fd obraz. M\u016f\u017eeme ho ch\u00e1pat jako roz\u0161\u00ed\u0159en\u00fd spustiteln\u00fd soubor, kter\u00fd obsahuje nejen program, ale i plnohodnotn\u00e9 b\u011bhov\u00e9 prost\u0159ed\u00ed (Java\/Python\/\u2026) a tak\u00e9 pot\u0159ebn\u00e9 soubory\/bal\u00ed\u010dky, p\u0159edinstalovan\u00e9 a p\u0159ipraven\u00e9 k spu\u0161t\u011bn\u00ed. Kontejnery lze nasazovat a spou\u0161t\u011bt na r\u016fzn\u00fdch serverech bez jak\u00fdchkoliv dal\u0161\u00edch akc\u00ed.<\/p>\n<p>Nav\u00edc kontejnery funguj\u00ed ve vlastn\u00edm p\u00edse\u010dkov\u00e9m prost\u0159ed\u00ed. Maj\u00ed sv\u016fj vlastn\u00ed virtu\u00e1ln\u00ed s\u00ed\u0165ov\u00fd adapt\u00e9r, vlastn\u00ed souborov\u00fd syst\u00e9m s omezen\u00fdm p\u0159\u00edstupem, svou hierarchii proces\u016f, sv\u00e1 omezen\u00ed na CPU a pam\u011b\u0165 atd. To v\u0161e je realizov\u00e1no d\u00edky zvl\u00e1\u0161tn\u00ed subsyst\u00e9mu j\u00e1dra Linux \u2013 namespaces (jmenn\u00e9 prostory).<\/p>\n<h3>Kubernetes<\/h3>\n<p>\nJak bylo zm\u00edn\u011bno d\u0159\u00edve, Kubernetes je orchestrace kontejner\u016f. Funguje takto: poskytnete mu skupinu stroj\u016f a pak \u0159\u00edk\u00e1te: \u201eHej, Kubernetes, spus\u0165 deset instanc\u00ed m\u00e9ho kontejneru s 2 procesory a 3 GB pam\u011bti na ka\u017edou a udr\u017euj je v chodu!\u201c. Kubernetes se postar\u00e1 o v\u0161echno ostatn\u00ed. Najde voln\u00e9 kapacity, spust\u00ed kontejnery a bude je restartovat podle pot\u0159eby, provede aktualizaci p\u0159i zm\u011bn\u011b verz\u00ed atd. V podstat\u011b Kubernetes umo\u017e\u0148uje abstrahovat se od hardwarov\u00e9 \u010d\u00e1sti a \u010din\u00ed v\u0161echny tyto r\u016fzn\u00e9 syst\u00e9my vhodn\u00fdmi pro nasazen\u00ed a pr\u00e1ci aplikac\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/6509bb1b66a4a0f5e9a479d0bd6ececb.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Kubernetes z pohledu oby\u010dejn\u00e9ho \u010dlov\u011bka<\/i><\/p>\n<h2>Co jsou po\u017eadavky (requests) a limity (limits) v Kubernetes<\/h2>\n<p>\nDob\u0159e, rozum\u00edme kontejner\u016fm a Kubernetes. Tak\u00e9 v\u00edme, \u017ee n\u011bkolik kontejner\u016f m\u016f\u017ee b\u00fdt na jednom stroji.<\/p>\n<p>M\u016f\u017eeme si to p\u0159irovnat k dru\u017estevn\u00edmu bytu. M\u00e1me prostornou m\u00edstnost (stroje\/uzly) a pronaj\u00edm\u00e1me ji n\u011bkolika n\u00e1jemn\u00edk\u016fm (kontejner\u016fm). Kubernetes hraje roli realitn\u00edho makl\u00e9\u0159e. Vznik\u00e1 ot\u00e1zka, jak udr\u017eet n\u00e1jemn\u00edky od vz\u00e1jemn\u00fdch konflikt\u016f? Co kdy\u017e se jeden z nich rozhodne zabrat koupelnu na p\u016fl dne?<\/p>\n<p>Tady vstupuj\u00ed do hry po\u017eadavky (requests) a limity (limits) CPU. <b>Request<\/b> je pot\u0159eba v\u00fdhradn\u011b pro pl\u00e1nov\u00e1n\u00ed. Je to n\u011bco jako \u201ep\u0159\u00e1n\u00ed seznam\u201c kontejneru a slou\u017e\u00ed k vyb\u00edr\u00e1n\u00ed nejvhodn\u011bj\u0161\u00edho uzlu. Z\u00e1rove\u0148 CPU <b>Limit<\/b> se d\u00e1 p\u0159irovnat k n\u00e1jemn\u00ed smlouv\u011b \u2013 jakmile vybereme uzel pro kontejner, ten <b>nebude moci<\/b> p\u0159ekro\u010dit stanoven\u00e9 hranice. A tady vznik\u00e1 probl\u00e9m\u2026<\/p>\n<h3>Jak jsou realizov\u00e1ny po\u017eadavky (requests) a limity (limits) v Kubernetes<\/h3>\n<p>\nKubernetes pou\u017e\u00edv\u00e1 vestav\u011bn\u00fd mechanizmus \u0161krt\u011bn\u00ed (throttling) pro implementaci limit\u016f CPU. Pokud aplikace p\u0159ekro\u010d\u00ed limit, dojde k \u0161krt\u011bn\u00ed (tj. dost\u00e1v\u00e1 m\u00e9n\u011b takt\u016f CPU). Po\u017eadavky (requests) a limity (limits) pro pam\u011b\u0165 jsou organizov\u00e1ny jinak, a proto je snaz\u0161\u00ed je odhalit. Sta\u010d\u00ed zkontrolovat posledn\u00ed stav restartu podu: nebyl \"OOMKilled\". Se \u0161krt\u011bn\u00edm CPU to nen\u00ed tak jednoduch\u00e9, proto\u017ee K8s poskytuje pouze metriky o pou\u017eit\u00ed, a ne o cgroups.<\/p>\n<h4>CPU Request<\/h4>\n<p>\n<img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/fbc8bf3f775ffd9513374ce94b20535d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Jak je implementov\u00e1n CPU request<\/i><\/p>\n<p>Pro jednoduchost si uve\u010fme proces na p\u0159\u00edkladu stroje se 4j\u00e1drov\u00fdm CPU.<\/p>\n<p>K8s pou\u017e\u00edv\u00e1 mechanismus kontroln\u00edch skupin (cgroups) pro spr\u00e1vu rozd\u011blen\u00ed zdroj\u016f (pam\u011bti a procesoru). Je k dispozici hierarchick\u00fd model: potomek d\u011bd\u00ed limity rodi\u010dovsk\u00e9 skupiny. Podrobnosti o rozd\u011blen\u00ed jsou ulo\u017eeny ve virtu\u00e1ln\u00edm souborov\u00e9m syst\u00e9mu (<code>\/sys\/fs\/cgroup<\/code>). V p\u0159\u00edpad\u011b procesoru je to <code>\/sys\/fs\/cgroup\/cpu,cpuacct\/*<\/code>.<\/p>\n<p>K8s pou\u017e\u00edv\u00e1 soubor <code>cpu.share<\/code> k rozd\u011blov\u00e1n\u00ed procesorov\u00fdch zdroj\u016f. V na\u0161em p\u0159\u00edpad\u011b z\u00edsk\u00e1v\u00e1 ko\u0159enov\u00e1 kontroln\u00ed skupina 4096 pod\u00edl\u016f zdroj\u016f CPU \u2014 100% dostupn\u00e9ho v\u00fdkonu procesoru (1 j\u00e1dro = 1024; jedn\u00e1 se o fixn\u00ed hodnotu). Ko\u0159enov\u00e1 skupina distribuuje zdroje proporcion\u00e1ln\u011b podle pod\u00edl\u016f potomk\u016f uveden\u00fdch v <code>cpu.share<\/code>, a ti na opl\u00e1tku zach\u00e1zej\u00ed stejn\u00fdm zp\u016fsobem se sv\u00fdmi potomky, a tak d\u00e1le. V typick\u00e9m uzlu Kubernetes m\u00e1 ko\u0159enov\u00e1 kontroln\u00ed skupina t\u0159i potomky: <code>system.slice<\/code>, <code>user.slice<\/code> a <code>kubepods<\/code>. Dv\u011b prvn\u00ed podskupiny jsou pou\u017e\u00edv\u00e1ny k rozd\u011blen\u00ed zdroj\u016f mezi kritick\u00e9 syst\u00e9mov\u00e9 z\u00e1t\u011b\u017ee a u\u017eivatelsk\u00e9 programy mimo K8s. Posledn\u00ed \u2014 <code>kubepods<\/code> \u2014 vytv\u00e1\u0159\u00ed Kubernetes pro rozd\u011blen\u00ed zdroj\u016f mezi pod.<\/p>\n<p>Na v\u00fd\u0161e uveden\u00e9m sch\u00e9matu je vid\u011bt, \u017ee prvn\u00ed a druh\u00e1 podskupina dostaly po <b>1024<\/b> pod\u00edlu, p\u0159i\u010dem\u017e podskupin\u011b kubepod bylo p\u0159id\u011bleno <b>4096<\/b> pod\u00edl\u016f. Jak je to mo\u017en\u00e9: v\u017edy\u0165 ko\u0159enov\u00e9 skupin\u011b je k dispozici pouze <b>4096<\/b> pod\u00edl\u016f, a sou\u010det pod\u00edl\u016f jej\u00edch potomk\u016f podstatn\u011b tento po\u010det p\u0159esahuje (<b>6144<\/b>)? Skute\u010dnost je takov\u00e1, \u017ee hodnota m\u00e1 logick\u00fd v\u00fdznam, a proto pl\u00e1nova\u010d Linuxu (CFS) ji pou\u017e\u00edv\u00e1 k proporcion\u00e1ln\u00edmu rozd\u011blen\u00ed zdroj\u016f CPU. V na\u0161em p\u0159\u00edpad\u011b prvn\u00ed dv\u011b skupiny dost\u00e1vaj\u00ed po <b>680<\/b> skute\u010dn\u00fdch pod\u00edlech (16,6% z 4096), zat\u00edmco kubepod dost\u00e1v\u00e1 zb\u00fdvaj\u00edc\u00edch <b>2736<\/b> pod\u00edl\u016f. V p\u0159\u00edpad\u011b ne\u010dinnosti prvn\u00ed dv\u011b skupiny nebudou vyu\u017e\u00edvat p\u0159id\u011blen\u00e9 zdroje.<\/p>\n<p>Na\u0161t\u011bst\u00ed pl\u00e1nova\u010d disponuje mechanismem, kter\u00fd zabra\u0148uje ztr\u00e1t\u011b nevyu\u017eit\u00fdch CPU zdroj\u016f. P\u0159ev\u00e1d\u00ed \u201ene\u010dinn\u00e9\u201c v\u00fdkonov\u00e9 jednotky do glob\u00e1ln\u00edho fondu, z n\u011bho\u017e jsou rozd\u011blov\u00e1ny mezi skupiny vy\u017eaduj\u00edc\u00ed dodate\u010dn\u00e9 v\u00fdpo\u010detn\u00ed schopnosti (p\u0159en\u00e1\u0161en\u00ed prob\u00edh\u00e1 po d\u00e1vk\u00e1ch, aby se p\u0159ede\u0161lo ztr\u00e1t\u00e1m zp\u016fsoben\u00fdm zaokrouhlov\u00e1n\u00edm). Podobn\u00e1 metoda se aplikuje i na v\u0161echny potomstvo.<\/p>\n<p>Tento mechanismus zaji\u0161\u0165uje spravedliv\u00e9 rozd\u011blen\u00ed procesorov\u00fdch v\u00fdkon\u016f a db\u00e1 na to, aby \u017e\u00e1dn\u00fd proces \u201enez\u00edsk\u00e1val\u201c zdroje od ostatn\u00edch.<\/p>\n<h4>CPU Limit<\/h4>\n<p>\nI kdy\u017e konfigurace limit\u016f a po\u017eadavk\u016f v K8s vypadaj\u00ed podobn\u011b, jejich implementace se radik\u00e1ln\u011b li\u0161\u00ed: je to <b>nejv\u00edce zav\u00e1d\u011bj\u00edc\u00ed<\/b> a nejm\u00e9n\u011b zdokumentovan\u00e1 \u010d\u00e1st.<\/p>\n<p>K8s vyu\u017e\u00edv\u00e1 <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-design-CFS.txt\">mechanismus CFS kvot<\/a><\/noindex> pro implementaci limit\u016f. Jejich nastaven\u00ed je ur\u010deno v souborech <code>cfs_period_us<\/code> a <code>cfs_quota_us<\/code> ve slo\u017ece cgroup (tam se tak\u00e9 nach\u00e1z\u00ed soubor <code>cpu.share<\/code>).<\/p>\n<p>Na rozd\u00edl od <code>cpu.share<\/code>kvoty vych\u00e1z\u00ed z <b>\u010dasov\u00e9ho obdob\u00ed<\/b>, nikoli z dostupn\u00e9 procesorov\u00e9 kapacity. <code>cfs_period_us<\/code> ur\u010duje d\u00e9lku obdob\u00ed (epochy) \u2014 to je v\u017edy 100000 mikrosekund (100 ms). V K8s je mo\u017en\u00e9 tuto hodnotu zm\u011bnit, av\u0161ak zat\u00edm je to mo\u017en\u00e9 pouze v alpha verzi. Pl\u00e1nova\u010d vyu\u017e\u00edv\u00e1 epochu k resetov\u00e1n\u00ed vy\u010derpan\u00fdch kvot. Druh\u00fd soubor, <code>cfs_quota_us<\/code>, ur\u010duje dostupn\u00fd \u010das (kvotu) v ka\u017ed\u00e9 epo\u0161e. V\u0161imn\u011bte si, \u017ee je tak\u00e9 uvedena v mikrosekund\u00e1ch. Kvota m\u016f\u017ee p\u0159ekra\u010dovat d\u00e9lku epochy; jin\u00fdmi slovy, m\u016f\u017ee b\u00fdt del\u0161\u00ed ne\u017e 100 ms.<\/p>\n<p>Pod\u00edvejme se na dva sc\u00e9n\u00e1\u0159e na 16j\u00e1drov\u00fdch stroj\u00edch (nejb\u011b\u017en\u011bj\u0161\u00ed typ po\u010d\u00edta\u010d\u016f u n\u00e1s v Omio):<\/p>\n<p><img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/61c6105320af3a92a91cec1007da0d45.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Sc\u00e9n\u00e1\u0159 1: 2 vl\u00e1kna a limit 200 ms. Bez throttle.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/c33993341d5395bb73dda1f4c8482a11.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Sc\u00e9n\u00e1\u0159 2: 10 vl\u00e1ken a limit 200 ms. Throttle za\u010d\u00edn\u00e1 po 20 ms, p\u0159\u00edstup k procesorov\u00fdm zdroj\u016fm se obnov\u00ed o dal\u0161\u00edch 80 ms.<\/i><\/p>\n<p>P\u0159edpokl\u00e1dejme, \u017ee jste nastavili limit CPU na <b>2<\/b> j\u00e1dra; Kubernetes p\u0159evede tuto hodnotu na 200 ms. To znamen\u00e1, \u017ee kontejner m\u016f\u017ee maxim\u00e1ln\u011b vyu\u017e\u00edvat 200 ms procesorov\u00e9ho \u010dasu bez throttlingu.<\/p>\n<p>A tady to za\u010d\u00edn\u00e1 b\u00fdt zaj\u00edmav\u00e9. Jak bylo uvedeno v\u00fd\u0161e, dostupn\u00e1 kvota \u010din\u00ed 200 ms. Pokud s v\u00e1mi paraleln\u011b b\u011b\u017e\u00ed <b>deset<\/b> vl\u00e1ken na 12j\u00e1drov\u00e9m stroji (viz ilustrace k sc\u00e9n\u00e1\u0159i 2), zat\u00edmco v\u0161echny ostatn\u00ed pody ne\u010dinn\u011b \u010dekaj\u00ed, kv\u00f3ta bude vy\u010derp\u00e1na b\u011bhem pouh\u00fdch 20 ms (proto\u017ee 10 * 20 ms = 200 ms), a v\u0161echna vl\u00e1kna dan\u00e9ho podu se \u201ezaseknou\u201c <i>(throttle)<\/i> na n\u00e1sleduj\u00edc\u00edch 80 ms. Situaci komplikuje ji\u017e zm\u00edn\u011bn\u00fd <noindex><a rel=\"nofollow\" href=\"https:\/\/bugzilla.kernel.org\/show_bug.cgi?id=198197\">bug pl\u00e1nova\u010de<\/a><\/noindex>, kv\u016fli kter\u00e9mu doch\u00e1z\u00ed k nadm\u011brn\u00e9mu \u0161krcen\u00ed a kontejner nem\u016f\u017ee vyprodukovat ani p\u0159id\u011blenou kv\u00f3tu.<\/p>\n<h2>Jak ocenit throttling v podech?<\/h2>\n<p>\nJednodu\u0161e se p\u0159ihlaste do podu a prove\u010fte <code>cat \/sys\/fs\/cgroup\/cpu\/cpu.stat<\/code>.<\/p>\n<ul>\n<li> <code>nr_periods<\/code> \u2014 celkov\u00fd po\u010det obdob\u00ed pl\u00e1nova\u010de;<\/li>\n<li> <code>nr_throttled<\/code> \u2014 po\u010det \u0161krcen\u00fdch obdob\u00ed v r\u00e1mci <code>nr_periods<\/code>;<\/li>\n<li> <code>throttled_time<\/code> \u2014 souhrnn\u00e9 \u0161krcen\u00e9 \u010dasov\u00e9 \u00faseky v nanosekund\u00e1ch.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/a1818614914a8c31f7f7f1331cc02e8b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Co se vlastn\u011b d\u011bje?<\/h3>\n<p>\nNakonec m\u00e1me vysok\u00e9 \u0161krcen\u00ed ve v\u0161ech aplikac\u00edch. N\u011bkdy je to <b>jeden a p\u016flkr\u00e1t<\/b> siln\u011bj\u0161\u00ed ne\u017e odhadovan\u00e9!<\/p>\n<p>To vede k r\u016fzn\u00fdm chyb\u00e1m \u2014 selh\u00e1n\u00ed kontrol p\u0159ipravenosti (readiness), zamrznut\u00ed kontejner\u016f, p\u0159eru\u0161en\u00ed s\u00ed\u0165ov\u00fdch p\u0159ipojen\u00ed, \u010dasov\u00fdm limit\u016fm v r\u00e1mci slu\u017eebn\u00edch vol\u00e1n\u00ed. Nakonec se to projevuje ve zv\u00fd\u0161en\u00e9 latenci a vy\u0161\u0161\u00edm po\u010dtu chyb.<\/p>\n<h2>\u0158e\u0161en\u00ed a d\u016fsledky<\/h2>\n<p>\nTady je to jednoduch\u00e9. Vzdali jsme se limit\u016f CPU a soust\u0159edili se na aktualizaci j\u00e1dra OS v clusterech na nejnov\u011bj\u0161\u00ed verzi, ve kter\u00e9 byla chyba opravena. Po\u010det chyb (HTTP 5xx) v na\u0161ich slu\u017eb\u00e1ch okam\u017eit\u011b v\u00fdrazn\u011b klesl:<\/p>\n<h3>Chyby HTTP 5xx<\/h3>\n<p>\n<img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/86ab1694b952b5451a4ae075cbda2788.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Chyby HTTP 5xx jedn\u00e9 kriticky d\u016fle\u017eit\u00e9 slu\u017eby<\/i><\/p>\n<h3>\u010cas odezvy p95<\/h3>\n<p>\n<img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/63922c8eb468366a7d05154a3adf40e6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Latence po\u017eadavk\u016f kriticky d\u016fle\u017eit\u00e9 slu\u017eby, 95. percentil<\/i><\/p>\n<h3>N\u00e1klady na provoz<\/h3>\n<p>\n<img decoding=\"async\" alt=\"CPU limity a agresivn\u00ed throttling v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/04\/8405c26fcbd82853e296fa7e3bcba131.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Po\u010det vynalo\u017een\u00fdch instance-hodin<\/i><\/p>\n<h2>Jak\u00fd je h\u00e1\u010dek?<\/h2>\n<p>\nJak bylo \u0159e\u010deno na za\u010d\u00e1tku \u010dl\u00e1nku:<\/p>\n<blockquote><p>M\u016f\u017eeme ud\u011blat analogii s dru\u017estevn\u00edm bytem\u2026 Kubernetes hraje roli realitn\u00edho makl\u00e9\u0159e. Ale jak udr\u017eet n\u00e1jemn\u00edky, aby nedoch\u00e1zelo ke konflikt\u016fm? Co kdy\u017e se jeden z nich nap\u0159\u00edklad rozhodne zabrat koupelnu na p\u016fl dne?<\/p><\/blockquote>\n<p>\nTady je h\u00e1\u010dek. Jeden nezodpov\u011bdn\u00fd kontejner m\u016f\u017ee pohlcovat ve\u0161ker\u00e9 dostupn\u00e9 procesorov\u00e9 zdroje na stroji. Pokud m\u00e1te solidn\u00ed technologick\u00fd stack (nap\u0159\u00edklad spr\u00e1vn\u011b nastaven\u00e9 JVM, Go, Node VM), pak to nen\u00ed probl\u00e9m: v takov\u00fdch podm\u00ednk\u00e1ch se d\u00e1 fungovat dlouhou dobu. Ale pokud jsou aplikace \u0161patn\u011b optimalizov\u00e1ny nebo v\u016fbec neoptimalizov\u00e1ny (<code>FROM java:latest<\/code>), situace se m\u016f\u017ee vymknout kontrole. U n\u00e1s v Omio m\u00e1me automatizovan\u00e9 z\u00e1kladn\u00ed Dockerfilese s adekv\u00e1tn\u00edmi v\u00fdchoz\u00edmi nastaven\u00edmi pro hlavn\u00ed jazykov\u00e9 stacky, tak\u017ee takov\u00fd probl\u00e9m neexistoval.<\/p>\n<p>Doporu\u010dujeme sledovat metriky <noindex><a rel=\"nofollow\" href=\"http:\/\/www.brendangregg.com\/usemethod.html\">USE<\/a><\/noindex> (vyu\u017eit\u00ed, saturace a chyby), zpo\u017ed\u011bn\u00ed API a frekvence v\u00fdskytu chyb. Sledujte, aby v\u00fdsledky odpov\u00eddaly o\u010dek\u00e1v\u00e1n\u00ed.<\/p>\n<h2>Odkazy<\/h2>\n<p>\nTakov\u00e1 je na\u0161e historie. N\u00e1sleduj\u00edc\u00ed materi\u00e1ly velmi pomohly objasnit, co se d\u011bje:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-design-CFS.txt\">kernel.org \u2192 CFS pl\u00e1nova\u010d<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-bwc.txt\">kernel.org \u2192 CFS kontrola \u0161\u00ed\u0159ky p\u00e1sma<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/engineering.squarespace.com\/blog\/2017\/understanding-linux-container-scheduling\">Pochopen\u00ed pl\u00e1nov\u00e1n\u00ed kontejner\u016f v Linuxu<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linuxjournal.com\/content\/everything-you-need-know-about-linux-containers-part-i-linux-control-groups-and-process\">V\u0161e, co pot\u0159ebujete v\u011bd\u011bt o kontejnerech v Linuxu, \u010d\u00e1st I: \u0158\u00eddic\u00ed skupiny Linuxu a izolace proces\u016f<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/k8s.af\/\">P\u0159\u00edb\u011bhy selh\u00e1n\u00ed Kubernetes<\/a><\/noindex> \u2014 hledejte \"cpu throttling\".<\/li>\n<\/ul>\n<p>\nZpr\u00e1vy o chyb\u00e1ch Kubernetes:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/51135#issuecomment-373454012\">#51135: Avoid setting CPU limits for Guaranteed pods<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/67577\">#67577: CFS quotas can lead to unnecessary throttling<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gist.github.com\/bobrik\/2030ff040fad360327a5fab7a09c4ff1\">P\u0159\u00edli\u0161 agresivn\u00ed CFS<\/a><\/noindex>.<\/li>\n<\/ul>\n<p>\nSetkali jste se s podobn\u00fdmi probl\u00e9my ve sv\u00e9 praxi nebo m\u00e1te zku\u0161enosti s throttlingem v kontejnerizovan\u00fdch produk\u010dn\u00edch prost\u0159ed\u00edch? Pod\u011blte se o sv\u016fj p\u0159\u00edb\u011bh v koment\u00e1\u0159\u00edch!<\/p>\n<h2>P.S. od p\u0159ekladatele<\/h2>\n<p>\nP\u0159e\u010dt\u011bte si tak\u00e9 n\u00e1\u0161 blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/459326\/\">Automatick\u00e9 \u0161k\u00e1lov\u00e1n\u00ed a spr\u00e1va zdroj\u016f v Kubernetes (p\u0159ehled a video prezentace)<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/418269\/\">Jak funguje CPU Manager v Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/342822\/\">Co se v Kubernetes d\u011bje p\u0159i spu\u0161t\u011bn\u00ed kubectl run? \u010c\u00e1st 2<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/489668\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes. \u0417\u043d\u0430\u043a\u043e\u043c\u0441\u0442\u0432\u043e \u0441 \u0442\u0430\u043a\u0438\u043c\u0438 \u0441\u043b\u0443\u0447\u0430\u044f\u043c\u0438 \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c \u043a\u0440\u0443\u0433\u043e\u0437\u043e\u0440, \u043d\u043e \u0438 \u043f\u0440\u0435\u0434\u043e\u0442\u0432\u0440\u0430\u0449\u0430\u0442\u044c \u043d\u0435\u0442\u0440\u0438\u0432\u0438\u0430\u043b\u044c\u043d\u044b\u0435 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b. \u0414\u043e\u0432\u043e\u0434\u0438\u043b\u043e\u0441\u044c \u043b\u0438 \u0432\u0430\u043c \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u0442\u044c\u0441\u044f \u0441 \u0442\u0435\u043c, \u0447\u0442\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435 \u00ab\u0437\u0430\u0441\u0442\u0440\u0435\u0432\u0430\u043b\u043e\u00bb \u043d\u0430 \u043c\u0435\u0441\u0442\u0435, \u043f\u0435\u0440\u0435\u0441\u0442\u0430\u0432\u0430\u043b\u043e \u043e\u0442\u0432\u0435\u0447\u0430\u0442\u044c \u043d\u0430 \u0437\u0430\u043f\u0440\u043e\u0441\u044b \u043e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0435 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":77694,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-77693","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47CPU-\u043b\u0438\u043c\u0438\u0442\u044b \u0438 \u0430\u0433\u0440\u0435\u0441\u0441\u0438\u0432\u043d\u044b\u0439 \u0442\u0440\u043e\u0442\u0442\u043b\u0438\u043d\u0433 \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-04-12T23:42:39+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-04-12T23:42:39+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47 CPU limity a agresivn\u00ed throttling v Kubernetes | ProHoster","description":"Pozn. p\u0159ekl.: Tento pou\u010dn\u00fd p\u0159\u00edb\u011bh Omio \u2014 evropsk\u00e9ho agreg\u00e1tora cestov\u00e1n\u00ed \u2014 prov\u00e1d\u00ed \u010dten\u00e1\u0159e od z\u00e1kladn\u00ed teorie k fascinuj\u00edc\u00edm praktick\u00fdm nuanc\u00edm v konfiguraci Kubernetes.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47CPU-\u043b\u0438\u043c\u0438\u0442\u044b \u0438 \u0430\u0433\u0440\u0435\u0441\u0441\u0438\u0432\u043d\u044b\u0439 \u0442\u0440\u043e\u0442\u0442\u043b\u0438\u043d\u0433 \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-04-12T23:42:39+00:00","article:modified_time":"2020-04-12T23:42:39+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"77693","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:13:23","updated":"2022-09-29 12:06:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/77693","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=77693"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/77693\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/77694"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=77693"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=77693"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=77693"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}