{"id":77693,"date":"2020-04-13T01:42:39","date_gmt":"2020-04-12T23:42:39","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes"},"modified":"2020-04-13T01:42:39","modified_gmt":"2020-04-12T23:42:39","slug":"cpu-limity-i-agressivnyj-trottling-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","title":{"rendered":"Omejitve CPU in agresivno omejevanje v Kubernetesu","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Op. prev.<\/b>: Ta zgodba Omio \u2013 evropskega agregatorja potovanj \u2013 vodi bralce od osnovne teorije do zanimivih prakti\u010dnih podrobnosti pri konfiguraciji Kubernetes. Seznanitev s tak\u0161nimi primeri ne le \u0161iri obzorja, temve\u010d tudi prepre\u010duje netrivijalne te\u017eave.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/1175c9df746e43b5a7d81476164929db.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nStekli ste se s tim da aplikacija \u201ezastane\u201c, prestane odgovarati na zahtjeve za provjeru stanja (health check) i niste mogli shvatiti uzrok takvog pona\u0161anja? Jedno od mogu\u0107ih obja\u0161njenja je povezano s limitom kvota na resurse CPU. O tome \u0107e biti rije\u010di u ovom \u010dlanku.<\/p>\n<p><b>TL;DR:<br \/>\nSna\u017eno preporu\u010dujemo da se odreknete CPU limit\u00e2 u Kubernetesu (ili isklju\u010dite CFS kvote u Kubeletu), ako koristite verziju Linux kernela s gre\u0161kom CFS kvota. <noindex><a rel=\"nofollow\" href=\"https:\/\/bugzilla.kernel.org\/show_bug.cgi?id=198197\">je na voljo<\/a><\/noindex> je resna in <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/67577\">dobro znana<\/a><\/noindex> napaka, ki vodi v pretirano omejevanje in zakasnitve.<\/b>.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>V Omio <b>vsa infrastruktura deluje na Kubernetes.<\/b>Sve na\u0161e stateful i stateless optere\u0107enja rade isklju\u010divo na Kubernetesu (koristimo Google Kubernetes Engine). U posljednjih \u0161est mjeseci primijetili smo slu\u010dajna usporavanja. Aplikacije se zamrzavaju ili prestaju odgovarati na health check, gube vezu s mre\u017eom itd. Takvo pona\u0161anje dugo nas je zbunjivalo, a na kraju smo se odlu\u010dili ozbiljno pozabaviti tim problemom.<\/p>\n<p>Povzetek \u010dlanka:<\/p>\n<ul>\n<li> Nekaj besed o vsebniki in Kubernetes;<\/li>\n<li> Kako su implementirani CPU zahtjevi i limiti;<\/li>\n<li> Kako omejitev CPU deluje v ve\u010djedrnih okoljih;<\/li>\n<li> Kako spremljati omejevanje CPU;<\/li>\n<li> Re\u0161evanje te\u017eave in podrobnosti.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Nekaj besed o vsebniki in Kubernetes<\/h2>\n<p>\nKubernetes je v bistvu sodoben standard v svetu infrastrukture. Njegova glavna naloga je orkestracija vsebnikov.<\/p>\n<h3>Kontejnerji<\/h3>\n<p>\nU pro\u0161losti smo morali stvarati artefakte poput Java JAR-ova\/WAR-ova, Python Egg-ova ili izvr\u0161nih datoteka za kasnije pokretanje na poslu\u017eiteljima. Me\u0111utim, da bismo ih natjerali da funkcioniraju, morali smo obaviti dodatni posao: postaviti izvo\u0111a\u010dku sredinu (Java\/Python), poslo\u017eiti potrebne datoteke na odgovaraju\u0107a mjesta, osigurati kompatibilnost s odre\u0111enom verzijom operativnog sustava itd. Drugim rije\u010dima, morali smo pridavati veliku pozornost upravljanju konfiguracijama (\u0161to je \u010desto bilo izvor nesuglasica izme\u0111u programera i sistemskih administratora).<\/p>\n<p><b>Vse posodobe so to spremenile.<\/b> Zdaj je artefakt kontejnerska slika. Lahko si jo predstavljamo kot raz\u0161irjeno izvr\u0161no datoteko, ki ne vsebuje le programa, temve\u010d tudi celovito okolje za izvajanje (Java\/Python\/\u2026) ter potrebne datoteke\/pakete, predname\u0161\u010dene in pripravljene za zagon. Kontejnerje lahko uvajamo in za\u017eenemo na razli\u010dnih stre\u017enikih brez dodatnih korakov.<\/p>\n<p>Poleg tega kontejnerji delujejo v svojem lastnem okolju peskovnika. Imajo svoj lasten virtualni omre\u017eni adapter, lasten datote\u010dni sistem z omejenim dostopom, svojo hierarhijo procesov, svoje omejitve glede CPU in pomnilnika itd. Vse to je omogo\u010deno zaradi posebne podsklade jedra Linux \u2014 namespaces (imeninska prostora).<\/p>\n<h3>Kubernetes<\/h3>\n<p>\nKot smo \u017ee omenili, je Kubernetes orkestrator kontejnerjev. Deluje na naslednji na\u010din: zagotovite mu skupino ra\u010dunalnikov in nato re\u010dete: \"Hej, Kubernetes, za\u017eeni deset primerkov mojega kontejnerja z 2 procesorjema in 3 GB pomnilnika za vsakogar ter jih ohrani v delovnem stanju!\" Kubernetes se bo poskrbel za vse ostalo. Poiskal bo razpolo\u017eljive zmogljivosti, zagnal kontejnerje in jih po potrebi ponovno zagnal, izdal posodobitve ob spremembi razli\u010dic itd. V bistvu Kubernetes omogo\u010da, da se abstrahiramo od strojne opreme in naredi vse raznolikosti sistemov primerne za uvajanje in delovanje aplikacij.<\/p>\n<p><img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/6509bb1b66a4a0f5e9a479d0bd6ececb.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Kubernetes z vidika navadnega uporabnika<\/i><\/p>\n<h2>\u0160to su zahtjevi i limiti u Kubernetesu<\/h2>\n<p>\nV redu, spoznali smo kontejnerje in Kubernetes. Prav tako vemo, da lahko ve\u010d kontejnerjev deluje na istem ra\u010dunalniku.<\/p>\n<p>Lahko naredimo analogijo s skupnim stanovanjem. Vzame se prostoren prostor (ra\u010dunalniki\/vozli) in odda ve\u010d najemnikom (kontejnerjem). Kubernetes deluje kot nepremi\u010dninski agent. Pojavi se vpra\u0161anje, kako zadr\u017eati najemnike, da ne pride do konfliktov med seboj? Kaj, \u010de eden od njih, recimo, sklene zasedeti kopalnico za pol dneva?<\/p>\n<p>Ovdje ulaze u igru zahtjevi i limiti. CPU <b>Request<\/b> je namenjen izklju\u010dno za na\u010drtovanje. To je nekaj, kar lahko primerjamo z \"seznamom \u017eelja\" kontejnerja in uporablja se za iskanje najprimernej\u0161ega vozla. Hkrati se CPU <b>Limit<\/b> lahko primerjamo s pogodbo o najemu \u2014 takoj ko izberemo vozel za kontejner, <b>ne bo mogel<\/b> \u0432\u044b\u0445\u043e\u0434\u0438\u0442\u044c \u0437\u0430 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u0435 \u043f\u0440\u0435\u0434\u0435\u043b\u044b. \u0418 \u0432\u043e\u0442 \u0437\u0434\u0435\u0441\u044c \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u2026<\/p>\n<h3>Kako su implementirani zahtjevi i limiti u Kubernetesu<\/h3>\n<p>\nKubernetes koristi ugra\u0111eni mehanizam throttlinga (uobi\u010dajenog propu\u0161tanja ciklusa) za implementaciju CPU limit\u00e2. Ako aplikacija prema\u0161i limit, uklju\u010duje se throttling (tj. dobiva manje CPU ciklusa). Zahtjevi i limiti za memoriju organizirani su druga\u010dije, pa ih je lak\u0161e uo\u010diti. Dovoljno je provjeriti posljednji status ponovnog pokretanja pod-a: nije li \u201eOOMKilled\u201c. S throttlingom CPU nije tako jednostavno, budu\u0107i da K8s pru\u017ea samo metrike kori\u0161tenja, a ne o cgroups.<\/p>\n<h4>CPU Request<\/h4>\n<p>\n<img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/fbc8bf3f775ffd9513374ce94b20535d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>\u041a\u0430\u043a \u0440\u0435\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d CPU request<\/i><\/p>\n<p>\u0414\u043b\u044f \u043f\u0440\u043e\u0441\u0442\u043e\u0442\u044b \u0434\u0430\u0432\u0430\u0439\u0442\u0435 \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u043d\u0430 \u043f\u0440\u0438\u043c\u0435\u0440\u0435 \u043c\u0430\u0448\u0438\u043d\u044b \u0441 4-\u044f\u0434\u0435\u0440\u043d\u044b\u043c CPU.<\/p>\n<p>K8s uporablja mehanizem kontrolnih skupin (cgroups) za upravljanje razporeditve virov (pomnilnika in procesorja). Na voljo je hierarhi\u010dni model: podedovana enota prevzema omejitve star\u0161evske skupine. Podrobnosti razporeditve se shranjujejo v virtualni datote\u010dni sistem (<code>\/sys\/fs\/cgroup<\/code>). \u0412 \u0441\u043b\u0443\u0447\u0430\u0435 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0440\u0430 \u044d\u0442\u043e <code>\/sys\/fs\/cgroup\/cpu,cpuacct\/*<\/code>.<\/p>\n<p>K8s \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u0444\u0430\u0439\u043b <code>cpu.share<\/code> \u0434\u043b\u044f \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0440\u0430. \u0412 \u043d\u0430\u0448\u0435\u043c \u0441\u043b\u0443\u0447\u0430\u0435 \u043a\u043e\u0440\u043d\u0435\u0432\u0430\u044f \u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u0430\u044f \u0433\u0440\u0443\u043f\u043f\u0430 \u043f\u043e\u043b\u0443\u0447\u0430\u0435\u0442 4096 \u0434\u043e\u043b\u0435\u0439 \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 CPU \u2014 100% \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0439 \u043c\u043e\u0449\u043d\u043e\u0441\u0442\u0438 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0440\u0430 (1 \u044f\u0434\u0440\u043e = 1024; \u044d\u0442\u043e \u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u043e\u0435 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435). \u041a\u043e\u0440\u043d\u0435\u0432\u0430\u044f \u0433\u0440\u0443\u043f\u043f\u0430 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u0442 \u0440\u0435\u0441\u0443\u0440\u0441\u044b \u043f\u0440\u043e\u043f\u043e\u0440\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e \u0432 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0438 \u043e\u0442 \u0434\u043e\u043b\u0435\u0439 \u043f\u043e\u0442\u043e\u043c\u043a\u043e\u0432, \u043f\u0440\u043e\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0445 \u0432 <code>cpu.share<\/code>, \u0430 \u0442\u0435, \u0432 \u0441\u0432\u043e\u044e \u043e\u0447\u0435\u0440\u0435\u0434\u044c, \u0430\u043d\u0430\u043b\u043e\u0433\u0438\u0447\u043d\u043e \u0440\u0430\u0441\u043f\u043e\u0440\u044f\u0436\u0430\u044e\u0442\u0441\u044f \u0441\u043e \u0441\u0432\u043e\u0438\u043c\u0438 \u043f\u043e\u0442\u043e\u043c\u043a\u0430\u043c\u0438, \u0438 \u0442.\u0434. \u0412 \u0442\u0438\u043f\u0438\u0447\u043d\u043e\u043c \u0443\u0437\u043b\u0435 Kubernetes \u043a\u043e\u0440\u043d\u0435\u0432\u0430\u044f \u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u0430\u044f \u0433\u0440\u0443\u043f\u043f\u0430 \u0438\u043c\u0435\u0435\u0442 \u0442\u0440\u0438 \u043f\u043e\u0442\u043e\u043c\u043a\u0430: <code>system.slice<\/code>, <code>user.slice<\/code> in <code>kubepods<\/code>. \u0414\u0432\u0435 \u043f\u0435\u0440\u0432\u044b\u0445 \u043f\u043e\u0434\u0433\u0440\u0443\u043f\u043f\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044e\u0442\u0441\u044f \u0434\u043b\u044f \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u043c\u0435\u0436\u0434\u0443 \u043a\u0440\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u0432\u0430\u0436\u043d\u044b\u043c\u0438 \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c\u0438 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0430\u043c\u0438 \u0438 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044c\u0441\u043a\u0438\u043c\u0438 \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u0430\u043c\u0438 \u0432\u043d\u0435 K8s. \u041f\u043e\u0441\u043b\u0435\u0434\u043d\u044f\u044f \u2014 <code>kubepods<\/code> \u2014 ustvarja ga Kubernetes za razporeditev virov med podi.<\/p>\n<p>\u041d\u0430 \u0441\u0445\u0435\u043c\u0435 \u0432\u044b\u0448\u0435 \u0432\u0438\u0434\u043d\u043e, \u0447\u0442\u043e \u043f\u0435\u0440\u0432\u0430\u044f \u0438 \u0432\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u0434\u0433\u0440\u0443\u043f\u043f\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u043f\u043e <b>1024<\/b> \u0434\u043e\u043b\u0438, \u043f\u0440\u0438 \u044d\u0442\u043e\u043c \u043f\u043e\u0434\u0433\u0440\u0443\u043f\u043f\u0435 kubepod \u0432\u044b\u0434\u0435\u043b\u0435\u043d\u043e <b>4096<\/b> \u0434\u043e\u043b\u0435\u0439. \u041a\u0430\u043a \u0442\u0430\u043a\u043e\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e: \u0432\u0435\u0434\u044c \u043a\u043e\u0440\u043d\u0435\u0432\u043e\u0439 \u0433\u0440\u0443\u043f\u043f\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b \u0432\u0441\u0435\u0433\u043e <b>4096<\/b> \u0434\u043e\u043b\u0435\u0439, \u0430 \u0441\u0443\u043c\u043c\u0430 \u0434\u043e\u043b\u0435\u0439 \u0435\u0435 \u043f\u043e\u0442\u043e\u043c\u043a\u043e\u0432 \u0437\u043d\u0430\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043f\u0440\u0435\u0432\u044b\u0448\u0430\u0435\u0442 \u044d\u0442\u043e \u0447\u0438\u0441\u043b\u043e (<b>6144<\/b>)? \u0414\u0435\u043b\u043e \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u0438\u043c\u0435\u0435\u0442 \u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u0441\u043c\u044b\u0441\u043b, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u043b\u0430\u043d\u0438\u0440\u043e\u0432\u0449\u0438\u043a Linux (CFS) \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u0435\u0433\u043e \u0434\u043b\u044f \u043f\u0440\u043e\u043f\u043e\u0440\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 CPU. \u0412 \u043d\u0430\u0448\u0435\u043c \u0441\u043b\u0443\u0447\u0430\u0435 \u043f\u0435\u0440\u0432\u044b\u0435 \u0434\u0432\u0435 \u0433\u0440\u0443\u043f\u043f\u044b \u043f\u043e\u043b\u0443\u0447\u0430\u044e\u0442 \u043f\u043e <b>680<\/b> \u0440\u0435\u0430\u043b\u044c\u043d\u044b\u0445 \u0434\u043e\u043b\u0435\u0439 (16,6% \u043e\u0442 4096), \u0430 kubepod \u043f\u043e\u043b\u0443\u0447\u0430\u0435\u0442 \u043e\u0441\u0442\u0430\u0432\u0448\u0438\u0435\u0441\u044f <b>2736<\/b> delite. V primeru nedelovanja prvi dve skupini ne bosta uporabljali dodeljenih virov.<\/p>\n<p>Na sre\u010do ima na\u010drtovalnik mehanizem, ki prepre\u010duje izgubo neizkori\u0161\u010denih virov CPU. Prenese \"neaktivne\" zmogljivosti v globalni sklad, iz katerega so razporejene skupinam, ki potrebujejo dodatne zmogljivosti procesorja (prenosi se izvajajo po paketih, da se izognejo izgubam zaradi zaokro\u017eevanja). Podoben na\u010din se uporablja tudi za vse potomce.<\/p>\n<p>Ta mehanizem zagotavlja po\u0161teno porazdelitev zmogljivosti procesorja in skrbi, da noben proces ne \"krade\" virov od drugih.<\/p>\n<h4>Omejitev CPU<\/h4>\n<p>\n\u010ceprav se konfiguracije omejitev in zahtev v K8s zdijo podobne, je njihova implementacija povsem druga\u010dna: to je <b>najbolj zavajajo\u010d<\/b> in najmanj dokumentiran del.<\/p>\n<p>K8s uporablja <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-design-CFS.txt\">mehanizem CFS kvot<\/a><\/noindex> za izvajanje omejitev. Njihove nastavitve so dolo\u010dene v datotekah <code>cfs_period_us<\/code> in <code>cfs_quota_us<\/code> v direktoriju cgroup (tukaj se nahaja tudi datoteka <code>cpu.share<\/code>).<\/p>\n<p>V nasprotju z <code>cpu.share<\/code>, kvota temelji na <b>\u010dasovnem obdobju<\/b>, ne pa na razpolo\u017eljivih zmogljivostih procesorja. <code>cfs_period_us<\/code> dolo\u010da trajanje obdobja (epoh) \u2014 to je vedno 100000 \u03bcs (100 ms). V K8s je mo\u017eno spremeniti to vrednost, vendar je trenutno na voljo le v alfa razli\u010dici. Na\u010drtovalnik uporablja epohe za ponovni zagon uporabljenih kvot. Druga datoteka, <code>cfs_quota_us<\/code>, dolo\u010da razpolo\u017eljiv \u010das (kvoto) v vsaki epohi. Upo\u0161tevajte, da je ta prav tako izra\u017eena v mikroseundah. Kvota lahko presega trajanje epoh; z drugimi besedami, lahko je ve\u010dja od 100 ms.<\/p>\n<p>Oglejmo si dva scenarija na 16-jedrnih strojih (najpogostej\u0161a vrsta ra\u010dunalnikov pri nas v Omio):<\/p>\n<p><img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/61c6105320af3a92a91cec1007da0d45.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Scenarij 1: 2 niti in omejitev 200 ms. Brez omejevanja.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/c33993341d5395bb73dda1f4c8482a11.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Scenarij 2: 10 niti in omejitev 200 ms. Omejevanje se za\u010dne po 20 ms, dostop do virov procesorja se obnovi \u0161e po 80 ms.<\/i><\/p>\n<p>Recimo, da ste nastavili omejitev CPU na <b>2<\/b> jedra; Kubernetes to vrednost pretvori v 200 ms. To pomeni, da lahko kontejner uporabi najve\u010d 200 ms procesorskega \u010dasa brez omejevanja.<\/p>\n<p>In tukaj se za\u010dne najbolj zanimivo. Kot je bilo \u017ee omenjeno, razpolo\u017eljiva kvota zna\u0161a 200 ms. \u010ce delate vzporedno <b>deset<\/b> tokov na 12-jedrnem stroju (glej ilustracijo k scenariju 2), medtem ko vsi ostali podi mirujejo, bo kvota iz\u010drpana \u017ee \u010dez 20 ms (ker 10 * 20 ms = 200 ms), in vsi tokovi tega poda bodo \"zataknjeni\" <i>(throttle)<\/i> u narednih 80 ms. Situacija se dodatno pogor\u0161ava ve\u0107 pomenutim <noindex><a rel=\"nofollow\" href=\"https:\/\/bugzilla.kernel.org\/show_bug.cgi?id=198197\">bagom planira\u010da<\/a><\/noindex>, zbog kojeg dolazi do prekomernog throttlinga i kontejner ne mo\u017ee da iskoristi ni dostupnu kvotu.<\/p>\n<h2>Kako oceniti trotliranje v podih?<\/h2>\n<p>\nJednostavno se pove\u017eite sa podom i izvr\u0161ite <code>cat \/sys\/fs\/cgroup\/cpu\/cpu.stat<\/code>.<\/p>\n<ul>\n<li> <code>nr_periods<\/code> \u2014 ukupni broj perioda planera;<\/li>\n<li> <code>nr_throttled<\/code> \u2014 broj throttled perioda u sklopu <code>nr_periods<\/code>;<\/li>\n<li> <code>throttled_time<\/code> \u2014 ukupno throttled vreme u nanosekundama.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/a1818614914a8c31f7f7f1331cc02e8b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>\u0160ta se zapravo de\u0161ava?<\/h3>\n<p>\nNa kraju, dobijamo visok throttling u svim aplikacijama. Ponekad je u <b>pola vi\u0161e<\/b> od o\u010dekivanog!<\/p>\n<p>To dovodi do raznih gre\u0161aka \u2014 gre\u0161ke u proverama spremnosti (readiness), zaglavljivanja kontejnera, prekida mre\u017enih veza, vremenskih ograni\u010denja unutar servisnih poziva. Na kraju, to se manifestuje kao pove\u0107ano ka\u0161njenje i pove\u0107anje broja gre\u0161aka.<\/p>\n<h2>Re\u0161enje i posledice<\/h2>\n<p>\nTukaj je vse preprosto. Odrekli smo se omejitvam CPU in se lotili posodobitve jedra OS v grozdih na najnovej\u0161o razli\u010dico, v kateri je bila napaka odpravljena. \u0160tevilo napak (HTTP 5xx) v na\u0161ih storitvah je takoj znatno padlo:<\/p>\n<h3>Gre\u0161ke HTTP 5xx<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/86ab1694b952b5451a4ae075cbda2788.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Gre\u0161ke HTTP 5xx jednog kriti\u010dnog servisa<\/i><\/p>\n<h3>Vreme odziva p95<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/63922c8eb468366a7d05154a3adf40e6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Ka\u0161njenje zahteva kriti\u010dnog servisa, 95. percentil<\/i><\/p>\n<h3>Tro\u0161kovi eksploatacije<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Omejitve CPU in agresivno omejevanje v Kubernetesu\" src=\"\/wp-content\/uploads\/2020\/04\/8405c26fcbd82853e296fa7e3bcba131.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Broj potro\u0161enih primeru-sati<\/i><\/p>\n<h2>U \u010demu je zamka?<\/h2>\n<p>\nKao \u0161to je re\u010deno na po\u010detku \u010dlanka:<\/p>\n<blockquote><p>Mo\u017ee se napraviti paralela sa stambenom zgradom\u2026 Kubernetes igra ulogu agenta za nekretnine. Ali kako zadr\u017eati stanare da ne upadaju u me\u0111usobne sukobe? \u0160ta ako jedan od njih, recimo, odlu\u010di da zauzme kupatilo na pola dana?<\/p><\/blockquote>\n<p>\nTo je zamka. Jedan nemaran kontejner mo\u017ee da upije sve dostupne resurse procesora na ma\u0161ini. Ako imate dobar skup aplikacija (na primer, pravilno pode\u0161ene JVM, Go, Node VM), to nije problem: mo\u017ee se raditi u takvim uslovima du\u017ee vreme. Ali ako su aplikacije slabo optimizovane ili potpuno neoptimizovane (<code>FROM java:latest<\/code>), situacija mo\u017ee izma\u0107i kontroli. U Omio imamo automatizovane osnovne Dockerfile-ove sa odgovaraju\u0107im podrazumevanim pode\u0161avanjima za staklenu glavnih jezika, tako da ovaj problem nije postojao.<\/p>\n<p>Preporu\u010dujemo da pratite metrike <noindex><a rel=\"nofollow\" href=\"http:\/\/www.brendangregg.com\/usemethod.html\">KORISTITE<\/a><\/noindex> (kori\u0161\u0107enje, saturacija i gre\u0161ke), ka\u0161njenja API-ja i u\u010destalost gre\u0161aka. Pazite da rezultati budu u skladu sa o\u010dekivanjima.<\/p>\n<h2>Povezave<\/h2>\n<p>\nTo je na\u0161a pri\u010da. Slede\u0107i materijali su u velikoj meri pomogli da se razjasni \u0161ta se de\u0161ava:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-design-CFS.txt\">kernel.org \u2192 CFS Scheduler<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.kernel.org\/doc\/Documentation\/scheduler\/sched-bwc.txt\">kernel.org \u2192 CFS Kontrola propusnosti<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/engineering.squarespace.com\/blog\/2017\/understanding-linux-container-scheduling\">Razumevanje zakazivanja Linux kontejnera<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linuxjournal.com\/content\/everything-you-need-know-about-linux-containers-part-i-linux-control-groups-and-process\">Sve \u0161to treba da znate o Linux kontejnerima, Deo I: Linux kontrolne grupe i izolacija procesa<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/k8s.af\/\">Kubernetes pri\u010de o neuspesima<\/a><\/noindex> \u2014 tra\u017eite \u201ecpu throttling\u201c.<\/li>\n<\/ul>\n<p>\nIzve\u0161taji o gre\u0161kama Kubernetes-a:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/51135#issuecomment-373454012\">#51135: Avoid setting CPU limits for Guaranteed pods<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/67577\">#67577: CFS quotas can lead to unnecessary throttling<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gist.github.com\/bobrik\/2030ff040fad360327a5fab7a09c4ff1\">Previ\u0161e agresivan CFS<\/a><\/noindex>.<\/li>\n<\/ul>\n<p>\nDa li ste se susretali sa sli\u010dnim problemima u va\u0161oj praksi ili imate iskustva vezana za throttling u kontejnerizovanim produkcionim okru\u017eenjima? Podelite svoju pri\u010du u komentarima!<\/p>\n<h2>P.S. od prevajalca<\/h2>\n<p>\nPreberite tudi na na\u0161em blogu:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/459326\/\">\u0410\u0432\u0442\u043e\u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0438 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0440\u0435\u0441\u0443\u0440\u0441\u0430\u043c\u0438 \u0432 Kubernetes (\u043e\u0431\u0437\u043e\u0440 \u0438 \u0432\u0438\u0434\u0435\u043e \u0434\u043e\u043a\u043b\u0430\u0434\u0430)<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/418269\/\">Kako funkcioni\u0161e CPU Manager u Kubernetes-u<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/342822\/\">\u0160ta se de\u0161ava u Kubernetes-u kada pokrenete kubectl run? Deo 2<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Izvor: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/489668\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes. \u0417\u043d\u0430\u043a\u043e\u043c\u0441\u0442\u0432\u043e \u0441 \u0442\u0430\u043a\u0438\u043c\u0438 \u0441\u043b\u0443\u0447\u0430\u044f\u043c\u0438 \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c \u043a\u0440\u0443\u0433\u043e\u0437\u043e\u0440, \u043d\u043e \u0438 \u043f\u0440\u0435\u0434\u043e\u0442\u0432\u0440\u0430\u0449\u0430\u0442\u044c \u043d\u0435\u0442\u0440\u0438\u0432\u0438\u0430\u043b\u044c\u043d\u044b\u0435 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b. \u0414\u043e\u0432\u043e\u0434\u0438\u043b\u043e\u0441\u044c \u043b\u0438 \u0432\u0430\u043c \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u0442\u044c\u0441\u044f \u0441 \u0442\u0435\u043c, \u0447\u0442\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435 \u00ab\u0437\u0430\u0441\u0442\u0440\u0435\u0432\u0430\u043b\u043e\u00bb \u043d\u0430 \u043c\u0435\u0441\u0442\u0435, \u043f\u0435\u0440\u0435\u0441\u0442\u0430\u0432\u0430\u043b\u043e \u043e\u0442\u0432\u0435\u0447\u0430\u0442\u044c \u043d\u0430 \u0437\u0430\u043f\u0440\u043e\u0441\u044b \u043e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0435 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":77694,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-77693","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sl_SI\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47CPU-\u043b\u0438\u043c\u0438\u0442\u044b \u0438 \u0430\u0433\u0440\u0435\u0441\u0441\u0438\u0432\u043d\u044b\u0439 \u0442\u0440\u043e\u0442\u0442\u043b\u0438\u043d\u0433 \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-04-12T23:42:39+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-04-12T23:42:39+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47CPU-limiti i agresivan throttling u Kubernetes-u | ProHoster","description":"Napomena prev.: ova pou\u010dna pri\u010da Omio - evropskog agregatora putovanja - vodi \u010ditaoce od osnovne teorije do uzbudljivih prakti\u010dnih nijansi u konfiguraciji Kubernetes-a.","canonical_url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sl_SI","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47CPU-\u043b\u0438\u043c\u0438\u0442\u044b \u0438 \u0430\u0433\u0440\u0435\u0441\u0441\u0438\u0432\u043d\u044b\u0439 \u0442\u0440\u043e\u0442\u0442\u043b\u0438\u043d\u0433 \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u044d\u0442\u0430 \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f Omio \u2014 \u0435\u0432\u0440\u043e\u043f\u0435\u0439\u0441\u043a\u043e\u0433\u043e \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440\u0430 \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0438\u0439 \u2014 \u043f\u0440\u043e\u0432\u043e\u0434\u0438\u0442 \u0447\u0438\u0442\u0430\u0442\u0435\u043b\u0435\u0439 \u043e\u0442 \u0431\u0430\u0437\u043e\u0432\u043e\u0439 \u0442\u0435\u043e\u0440\u0438\u0438 \u0434\u043e \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0445 \u0442\u043e\u043d\u043a\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 Kubernetes.","og:url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/cpu-limity-i-agressivnyj-trottling-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-04-12T23:42:39+00:00","article:modified_time":"2020-04-12T23:42:39+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"77693","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:13:23","updated":"2022-09-29 12:06:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/77693","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/comments?post=77693"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/77693\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media\/77694"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media?parent=77693"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/categories?post=77693"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/tags?post=77693"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}