{"id":94977,"date":"2020-09-24T07:43:00","date_gmt":"2020-09-24T05:43:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes"},"modified":"2020-09-24T07:43:00","modified_gmt":"2020-09-24T05:43:00","slug":"nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","title":{"rendered":"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Pozn. prekl.<\/b>: adapt\u00e1cia Kubernetes v GitLab je pova\u017eovan\u00e1 za jeden z dvoch hlavn\u00fdch faktorov prispievaj\u00facich k rastu spolo\u010dnosti. Napriek tomu bola do ned\u00e1vna infra\u0161trukt\u00fara online slu\u017eby GitLab.com postaven\u00e1 na virtu\u00e1lnych strojoch a len pred pribli\u017ene rokom sa za\u010dala jej migr\u00e1cia do K8s, ktor\u00e1 st\u00e1le nie je dokon\u010den\u00e1. S rados\u0165ou predstavujeme preklad ned\u00e1vneho \u010dl\u00e1nku SRE in\u017einiera GitLab o tom, ako sa to deje a ak\u00e9 z\u00e1very robia in\u017einieri zapojen\u00ed do projektu.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/58429abf60cd19a49c5ce593051c2fca.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nU\u017e asi rok sa na\u0161e infra\u0161trukt\u00farne oddelenie zaober\u00e1 migr\u00e1ciou v\u0161etk\u00fdch slu\u017eieb, ktor\u00e9 funguj\u00fa na GitLab.com, do Kubernetes. Po\u010das tejto doby sme \u010delili probl\u00e9mom nielen pri pres\u00favan\u00ed slu\u017eieb do Kubernetes, ale aj pri spr\u00e1ve hybridn\u00e9ho deploymentu po\u010das prechodu. O cenn\u00fdch lekci\u00e1ch, ktor\u00e9 sme z\u00edskali, bude t\u00e1to \u010dl\u00e1nok.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Od sam\u00e9ho za\u010diatku GitLab.com jeho servery fungovali v cloude na virtu\u00e1lnych strojoch. Tieto virtu\u00e1lne stroje spravuje Chef a ich in\u0161tal\u00e1cia prebieha pomocou n\u00e1\u0161ho <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/install\/#ubuntu\">ofici\u00e1lneho Linux bal\u00edka<\/a><\/noindex>. <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/release\/docs\/-\/blob\/master\/general\/deploy\/gitlab-com-deployer.md\">Strat\u00e9gia nasadenia<\/a><\/noindex> v pr\u00edpade, \u017ee je potrebn\u00e9 aktualizova\u0165 aplik\u00e1ciu, spo\u010d\u00edva v jednoduchom aktualizovan\u00ed parku serverov koordinovan\u00fdm sekven\u010dn\u00fdm sp\u00f4sobom prostredn\u00edctvom CI pipeline. T\u00e1to met\u00f3da \u2014 aj ke\u010f je pomal\u00e1 a trochu <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/values\/#boring-solutions\">nudn\u00e1<\/a><\/noindex> \u2014 zaru\u010duje, \u017ee GitLab.com aplikuje tie ist\u00e9 met\u00f3dy in\u0161tal\u00e1cie a konfigur\u00e1cie, ak\u00e9 pou\u017e\u00edvaj\u00fa u\u017e\u00edvatelia samostatn\u00fdch <i>(self-managed)<\/i> in\u0161tal\u00e1ci\u00ed GitLab, ktor\u00ed na to vyu\u017e\u00edvaj\u00fa na\u0161e Linux bal\u00edky.<\/p>\n<p>T\u00fato met\u00f3du pou\u017e\u00edvame, preto\u017ee je nesmierne d\u00f4le\u017eit\u00e9 za\u017ei\u0165 na vlastnej ko\u017ei v\u0161etky tr\u00e1penia a radosti, ktor\u00e9 be\u017en\u00ed \u010dlenovia komunity za\u017e\u00edvaj\u00fa pri in\u0161tal\u00e1cii a nastaven\u00ed svojich k\u00f3pi\u00ed GitLab. Tento pr\u00edstup fungoval dobre po\u010das ur\u010dit\u00e9ho \u010dasu, av\u0161ak ke\u010f po\u010det projektov na GitLab prekro\u010dil 10 mili\u00f3nov, uvedomili sme si, \u017ee u\u017e nesp\u013a\u0148a na\u0161e potreby v oblasti \u0161k\u00e1lovania a nasadenia.<\/p>\n<h2>Prv\u00e9 kroky k Kubernetes a cloud-native GitLab<\/h2>\n<p>\nV roku 2017 bol zalo\u017een\u00fd projekt <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\">GitLab Charts<\/a><\/noindex> na pr\u00edpravu GitLab na nasadenie v cloude a tie\u017e na to, aby sme pou\u017e\u00edvate\u013eom umo\u017enili in\u0161talova\u0165 GitLab do klastrov Kubernetes. Vtedy sme vedeli, \u017ee presun GitLab do Kubernetes zv\u00fd\u0161i mo\u017enosti \u0161k\u00e1lovania SaaS platforiem, zjednodu\u0161\u00ed nasadenia a zv\u00fd\u0161i efektivitu vyu\u017e\u00edvania v\u00fdpo\u010dtov\u00fdch zdrojov. Z\u00e1rove\u0148 mnoh\u00e9 funkcie na\u0161ej aplik\u00e1cie z\u00e1viseli od pripojen\u00fdch NFS oddielov, \u010do spoma\u013eovalo prechod z virtu\u00e1lnych strojov.<\/p>\n<p>T\u00fa\u017eba po cloud native a Kubernetes umo\u017enila na\u0161im in\u017einierom pl\u00e1nova\u0165 postupn\u00fd prechod, po\u010das ktor\u00e9ho sme sa vzdali niektor\u00fdch z\u00e1vislost\u00ed aplik\u00e1cie od sie\u0165ov\u00fdch \u00falo\u017e\u00edsk, pri\u010dom sme pokra\u010dovali v v\u00fdvoji nov\u00fdch funkci\u00ed. Od za\u010diatku pl\u00e1novania migr\u00e1cie v lete 2019 mnoh\u00e9 z t\u00fdchto obmedzen\u00ed boli odstr\u00e1nen\u00e9 a proces prevodu GitLab.com na Kubernetes teraz prebieha naplno!<\/p>\n<h2>Charakteristiky fungovania GitLab.com v Kubernetes<\/h2>\n<p>\nPre GitLab.com pou\u017e\u00edvame jednotn\u00fd region\u00e1lny klaster GKE, ktor\u00fd spracov\u00e1va cel\u00fd prenos aplik\u00e1cie. Aby sme minimalizovali zlo\u017eitosti (u\u017e aj tak zlo\u017eit\u00fdch) migr\u00e1ci\u00ed, s\u00fastred\u00edme sa na slu\u017eby, ktor\u00e9 nez\u00e1visia od miestneho \u00falo\u017eiska alebo NFS. GitLab.com pou\u017e\u00edva preva\u017ene monolitn\u00fa k\u00f3dov\u00fa z\u00e1klad\u0148u na Rails a prenos smerujeme v z\u00e1vislosti od vlastnost\u00ed pracovnej z\u00e1\u0165a\u017ee na r\u00f4zne endpointy, izolovan\u00e9 vo vlastn\u00fdch uzlov\u00fdch pooloch.<\/p>\n<p>Pokia\u013e ide o frontend, tieto typy sa delia na po\u017eiadavky na web, API, Git SSH\/HTTPS a Registry. Pokia\u013e ide o backend, rozde\u013eujeme joby v porad\u00ed pod\u013ea r\u00f4znych vlastnost\u00ed v z\u00e1vislosti od <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/blog\/2020\/06\/24\/scaling-our-use-of-sidekiq\/\">preddefinovan\u00fdch hran\u00edc zdrojov<\/a><\/noindex>, ktor\u00e9 n\u00e1m umo\u017e\u0148uj\u00fa nastavi\u0165 ciele \u00farovne slu\u017eieb (Service-Level Objectives, SLOs) pre r\u00f4zne z\u00e1\u0165a\u017ee.<\/p>\n<p>V\u0161etky tieto slu\u017eby GitLab.com s\u00fa nastaven\u00e9 pomocou nemodifikovan\u00e9ho Helm chartu GitLab. Konfigur\u00e1cia sa vykon\u00e1va v subchirtoch, ktor\u00e9 m\u00f4\u017eu by\u0165 v\u00fdberovo zahrnut\u00e9, ke\u010f postupne pren\u00e1\u0161ame slu\u017eby do klastera. Aj ke\u010f bolo rozhodnut\u00e9 nezah\u0155\u0148a\u0165 do migr\u00e1cie niektor\u00e9 z na\u0161ich stateful slu\u017eieb, ako Redis, Postgres, GitLab Pages a Gitaly, pou\u017eitie Kubernetes umo\u017e\u0148uje radik\u00e1lne zn\u00ed\u017ei\u0165 po\u010det VM, ktor\u00e9 moment\u00e1lne spravuje Chef.<\/p>\n<h2>Prieh\u013eadnos\u0165 a spr\u00e1va konfigur\u00e1cie Kubernetes<\/h2>\n<p>\nV\u0161etky nastavenia spravuje samotn\u00fd GitLab. Na tento \u00fa\u010del pou\u017e\u00edvame tri konfigura\u010dn\u00e9 projekty zalo\u017een\u00e9 na Terraform a Helm. Sna\u017e\u00edme sa v\u0161ade, kde je to mo\u017en\u00e9, pou\u017e\u00edva\u0165 samotn\u00fd GitLab na spustenie GitLab, ale pre prev\u00e1dzkov\u00e9 \u00falohy m\u00e1me samostatn\u00fa in\u0161tal\u00e1ciu GitLab. T\u00e1 je potrebn\u00e1 na to, aby sme nez\u00e1viseli od dostupnosti GitLab.com pri vykon\u00e1van\u00ed nasaden\u00ed a aktualiz\u00e1ci\u00ed GitLab.com.<\/p>\n<p>Aj ke\u010f na\u0161e pipeline pre klaster Kubernetes funguj\u00fa na samostatnej in\u0161tal\u00e1cii GitLab, k\u00f3dov\u00e9 repozit\u00e1re maj\u00fa zrkadl\u00e1, ktor\u00e9 s\u00fa verejne dostupn\u00e9 na nasleduj\u00facich adres\u00e1ch:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-com\">k8s-workloads\/gitlab-com<\/a><\/noindex> \u2014 konfigura\u010dn\u00e1 v\u00e4zba GitLab.com pre Helm chart GitLab;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-helmfiles\/\">k8s-workloads\/gitlab-helmfiles<\/a><\/noindex> \u2014 obsahuje konfigur\u00e1cie pre slu\u017eby, ktor\u00e9 nie s\u00fa priamo spojen\u00e9 s aplik\u00e1ciou GitLab. Patr\u00ed sem konfigur\u00e1cia pre spr\u00e1vu logov a monitorovanie klastra, ako aj pre integrovan\u00e9 n\u00e1stroje ako PlantUML;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gitlab-com-infrastructure\">Gitlab-com-infrastructure<\/a><\/noindex> \u2014 konfigur\u00e1cia Terraform pre Kubernetes a star\u00fa (legacy) VM-infrastrukt\u00faru. Tu sa nastavuj\u00fa v\u0161etky zdroje potrebn\u00e9 na spustenie klastra, vr\u00e1tane samotn\u00e9ho klastra, poolov uzlov, slu\u017eobn\u00fdch \u00fa\u010dtov a rezerv\u00e1cie IP adries.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/612125403171d73106a081bf4244a52b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Pri vykon\u00e1van\u00ed zmien sa zobrazuje verejn\u00fd <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-com\/-\/merge_requests\/315#note_390180361\"><i>preh\u013ead<\/i><\/a><\/noindex><i> s odkazom na podrobn\u00fd diff, ktor\u00fd SRE analyzuje pred vykonan\u00edm zmien v klastri.<\/i><\/p>\n<p>Pre SRE odkaz vedie na podrobn\u00fd diff v in\u0161tal\u00e1cii GitLab, ktor\u00e1 sa pou\u017e\u00edva na prev\u00e1dzku a pr\u00edstup k nej je obmedzen\u00fd. To umo\u017e\u0148uje zamestnancom a komunite bez pr\u00edstupu k prev\u00e1dzkov\u00e9mu projektu (ten je otvoren\u00fd len pre SRE) prezera\u0165 navrhovan\u00e9 zmeny v konfigur\u00e1cii. Kombinovan\u00edm verejn\u00e9ho GitLab exempl\u00e1ru pre k\u00f3d s uzavret\u00fdm exempl\u00e1rom pre CI pipeline zachov\u00e1vame jednotn\u00fd pracovn\u00fd proces, pri\u010dom zaru\u010dujeme nez\u00e1vislos\u0165 od GitLab.com pri aktualiz\u00e1ci\u00e1ch konfigur\u00e1cie.<\/p>\n<h2>\u010co sme zistili po\u010das migr\u00e1cie<\/h2>\n<p>\nPo\u010das presunu sme z\u00edskali sk\u00fasenosti, ktor\u00e9 aplikujeme na nov\u00e9 migr\u00e1cie a nasadenia v Kubernetes.<\/p>\n<h3>1. Rast n\u00e1kladov kv\u00f4li prev\u00e1dzke medzi z\u00f3nami dostupnosti<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/3dce44b3f803ffcea13e0101e7343d91.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Denn\u00e1 egress \u0161tatistika (bytov za de\u0148) pre park Git \u00falo\u017e\u00edsk na GitLab.com<\/i><\/p>\n<p>Google rozde\u013euje svoju sie\u0165 na regi\u00f3ny. Tieto sa n\u00e1sledne delia na dostupnostn\u00e9 z\u00f3ny (AZ). Git hosting je spojen\u00fd s ve\u013ek\u00fdmi objemami d\u00e1t, a preto je d\u00f4le\u017eit\u00e9 kontrolova\u0165 sie\u0165ov\u00fd egress. V pr\u00edpade intern\u00e9ho prenosu je egress bezplatn\u00fd len ak zost\u00e1va v r\u00e1mci jednej dostupnostnej z\u00f3ny. V \u010dase p\u00edsania tohto \u010dl\u00e1nku odovzd\u00e1vame pribli\u017ene 100 TB d\u00e1t v be\u017en\u00fd pracovn\u00fd de\u0148 (a to len pre Git repozit\u00e1re). Slu\u017eby, ktor\u00e9 v na\u0161ej starej topol\u00f3gii, zalo\u017eenej na VM, boli na rovnak\u00fdch virtu\u00e1lnych strojoch, teraz funguj\u00fa v r\u00f4znych pod&#8217;och Kubernetes. To znamen\u00e1, \u017ee \u010das\u0165 prenosu, ktor\u00e1 bola predt\u00fdm lok\u00e1lna pre VM, m\u00f4\u017ee potenci\u00e1lne prekro\u010di\u0165 hranice dostupnostn\u00fdch z\u00f3n.<\/p>\n<p>Region\u00e1lne klastre GKE umo\u017e\u0148uj\u00fa pokrytie viacer\u00fdch z\u00f3n dostupnosti pre rezerv\u00e1cie. Zva\u017eujeme <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/delivery\/-\/issues\/1175\">rozdelenie region\u00e1lneho klastra GKE na jedzonov\u00e9 klastre<\/a><\/noindex> pre slu\u017eby, ktor\u00e9 generuj\u00fa ve\u013ek\u00e9 objemy prenosu. To pom\u00f4\u017ee zn\u00ed\u017ei\u0165 n\u00e1klady na egress pri zachovan\u00ed rezerv\u00e1cie na \u00farovni klastra.<\/p>\n<h3>2. Limity, po\u017eiadavky na zdroje a \u0161k\u00e1lovanie<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/6e2e1ca4d37666b49358d94cd8660c56.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Po\u010det replik spracov\u00e1vaj\u00facich produk\u010dn\u00fd prenos na registry.gitlab.com. Prenos dosahuje svoj vrchol okolo 15:00 UTC.<\/i><\/p>\n<p>Na\u0161a hist\u00f3ria migr\u00e1cie sa za\u010dala v auguste 2019, ke\u010f sme presunuli prv\u00fa slu\u017ebu \u2014 registry kontajnerov GitLab (GitLab Container Registry) \u2014 do Kubernetes. T\u00e1to kriticky d\u00f4le\u017eit\u00e1 slu\u017eba s vysok\u00fdm prenosom bola pre prv\u00fa migr\u00e1ciu ide\u00e1lna, preto\u017ee predstavuje stateless aplik\u00e1ciu s minim\u00e1lnymi extern\u00fdmi z\u00e1vislos\u0165ami. Prv\u00fdm probl\u00e9mom, s ktor\u00fdm sme sa stretli, bolo vysok\u00e9 mno\u017estvo vytla\u010den\u00fdch pod&#8217;ov kv\u00f4li nedostatku pam\u00e4te na uzloch. Z tohto d\u00f4vodu sme museli upravi\u0165 po\u017eiadavky a limity.<\/p>\n<p>Zistili sme, \u017ee v pr\u00edpade aplik\u00e1cie, ktorej spotreba pam\u00e4te rastie v priebehu \u010dasu, n\u00edzke hodnoty pre po\u017eiadavky (rezervuj\u00face pam\u00e4\u0165 pre ka\u017ed\u00fd pod&#8217;) spolu s \u201e\u0161tedr\u00fdm\u201c pr\u00edsnym limitom na pou\u017eitie viedli k satur\u00e1cii. <i>(saturation)<\/i> uzlov a k vysok\u00e9mu po\u010dtu vytla\u010den\u00ed. Aby sme sa s t\u00fdmto probl\u00e9mom vyrovnali, bolo <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/delivery\/-\/issues\/998#note_388983696\">rozhodli sme sa zv\u00fd\u0161i\u0165 po\u017eiadavky a zn\u00ed\u017ei\u0165 limity.<\/a><\/noindex>. To zmiernilo tlak na uzloch a zabezpe\u010dilo \u017eivotn\u00fd cyklus pod&#8217;ov, ktor\u00fd nekonal pr\u00edli\u0161 vysok\u00fd tlak na uzol. Teraz za\u010d\u00edname migr\u00e1cie so \u0161tedr\u00fdmi (a takmer rovnak\u00fdmi) hodnotami po\u017eiadaviek a limitov, ktor\u00e9 upravujeme pod\u013ea potreby.<\/p>\n<h3>3. Metrix a logy<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e zhrnutie za rok migr\u00e1cie GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/66d1fd47b57d5826f13defa6e6a7fe3c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Infrastrukt\u00farne oddelenie sa zameriava na oneskorenia, percento ch\u00fdb a satur\u00e1ciu s nastaven\u00fdmi <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Service-level_objective\"><i>cie\u013emi \u00farovne slu\u017eieb<\/i><\/a><\/noindex><i> (SLO), viazan\u00fdmi na <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/dashboards-gitlab-com\/-\/metrics\/sla-dashboard.yml?environment=1790496&amp;duration_seconds=86400\"><i>celkov\u00fa dostupnos\u0165 n\u00e1\u0161ho syst\u00e9mu<\/i><\/a><\/noindex><i>.<\/i><\/p>\n<p>V uplynulom roku sa jedn\u00fdm z k\u013e\u00fa\u010dov\u00fdch udalost\u00ed v infra\u0161trukt\u00farnom oddelen\u00ed stali vylep\u0161enia v monitorovan\u00ed a pr\u00e1ci so SLO. SLO n\u00e1m umo\u017enili nastavova\u0165 ciele pre jednotliv\u00e9 slu\u017eby, ktor\u00e9 sme starostlivo sledovali po\u010das migr\u00e1cie. Ale aj s takto vylep\u0161enou vidite\u013enos\u0165ou nie v\u017edy m\u00f4\u017eeme okam\u017eite odhali\u0165 probl\u00e9my, pou\u017e\u00edvaj\u00fac metriky a upozornenia. Napr\u00edklad, zameran\u00edm sa na oneskorenia a percento ch\u00fdb, \u00faplne nezachyt\u00e1vame v\u0161etky scen\u00e1re pou\u017e\u00edvania slu\u017eby, ktor\u00e1 prech\u00e1dza migr\u00e1ciou.<\/p>\n<p>Tento probl\u00e9m bol objaven\u00fd takmer okam\u017eite po presune \u010dasti pracovn\u00fdch z\u00e1\u0165a\u017e\u00ed do clusteru. Najviac sa prejavil, ke\u010f bolo potrebn\u00e9 kontrolova\u0165 funkcie, po\u010det po\u017eiadaviek na ktor\u00e9 nie je vysok\u00fd, ale maj\u00fa ve\u013emi \u0161pecifick\u00e9 konfigura\u010dn\u00e9 z\u00e1vislosti. Jedn\u00fdm z k\u013e\u00fa\u010dov\u00fdch ponau\u010den\u00ed z migr\u00e1cie bola potreba zoh\u013eadni\u0165 pri monitorovan\u00ed nielen metriky, ale aj logy a \u201edlh\u00fd chvost\u201c <i>(t\u00fdka sa to <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Long_tail\"><i>tak\u00e9hoto ich rozdelenia<\/i><\/a><\/noindex><i> na grafe \u2014 pozn. prekl.)<\/i> ch\u00fdb. Teraz pre ka\u017ed\u00fa migr\u00e1ciu zah\u0155\u0148ame podrobn\u00fd zoznam po\u017eiadaviek na logy <i>(log queries)<\/i> a pl\u00e1nujeme jasn\u00e9 postupy na vr\u00e1tenie, ktor\u00e9 m\u00f4\u017eeme v pr\u00edpade probl\u00e9mov pren\u00e1\u0161a\u0165 z jednej zmeny na druh\u00fa.<\/p>\n<p>Paraleln\u00e9 spracovanie rovnak\u00fdch po\u017eiadaviek na starej VM-infrastrukt\u00fare a novej, zalo\u017eenej na Kubernetes, predstavovalo jedine\u010dn\u00fa \u00falohu. Na rozdiel od migr\u00e1cie typu lift-and-shift <i>(r\u00fdchly presun aplik\u00e1ci\u00ed \u201etak, ako s\u00fa\u201c do novej infra\u0161trukt\u00fary; viac inform\u00e1ci\u00ed si m\u00f4\u017eete pre\u010d\u00edta\u0165 napr\u00edklad, <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/www.ibm.com\/cloud\/learn\/lift-and-shift\"><i>tu<\/i><\/a><\/noindex><i> \u2014 pozn. prekl.)<\/i>, paraleln\u00e9 fungovanie na \u201estar\u00fdch\u201c VM a Kubernetes vy\u017eaduje, aby n\u00e1stroje na monitorovanie boli kompatibiln\u00e9 s oboma prostrediami a vedeli spoji\u0165 metriky do jedn\u00e9ho zobrazenia. Je d\u00f4le\u017eit\u00e9, \u017ee pou\u017e\u00edvame tie ist\u00e9 dashboardy a dopyty na logy, aby sme dosiahli konzistentn\u00fa observabilitu po\u010das prechodn\u00e9ho obdobia.<\/p>\n<h3>4. Prepnutie prev\u00e1dzky na nov\u00fd klaster<\/h3>\n<p>\nPre GitLab.com je \u010das\u0165 serverov vy\u010dlenen\u00e1 na <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/#canary-testing\">kan\u00e1rkov\u00fa (canary) f\u00e1zu<\/a><\/noindex>. Kan\u00e1rkov\u00fd park obsluhuje na\u0161e vn\u00fatorn\u00e9 projekty a m\u00f4\u017ee sa <noindex><a rel=\"nofollow\" href=\"https:\/\/next.gitlab.com\/\">zap\u00edna\u0165 pou\u017e\u00edvate\u013emi<\/a><\/noindex>. Predov\u0161etk\u00fdm je ur\u010den\u00fd na testovanie zmien v infra\u0161trukt\u00fare a aplik\u00e1cii. Prv\u00e1 prenesen\u00e1 slu\u017eba za\u010dala s prij\u00edman\u00edm obmedzen\u00e9ho objemu vn\u00fatorn\u00e9ho prev\u00e1dzky a na\u010falej pou\u017e\u00edvame t\u00fato met\u00f3du, aby sme sa uistili, \u017ee sa dodr\u017eiava SLO, predt\u00fdm ne\u017e nasmerujeme cel\u00fa prev\u00e1dzku do klastru.<\/p>\n<p>V pr\u00edpade migr\u00e1cie to znamen\u00e1, \u017ee najprv s\u00fa do Kubernetes smerovan\u00e9 dopyty na vn\u00fatorn\u00e9 projekty a potom postupne prep\u00edname na klaster zvy\u0161ok prev\u00e1dzky zmenou v\u00e1hy pre backend cez HAProxy. Po\u010das prechodu z VM na Kubernetes sa uk\u00e1zalo, \u017ee je ve\u013emi v\u00fdhodn\u00e9 ma\u0165 po ruke jednoduch\u00fd sp\u00f4sob presmerovania prev\u00e1dzky medzi starou a novou infra\u0161trukt\u00farou a t\u00fdm p\u00e1dom udr\u017eiava\u0165 star\u00fa infra\u0161trukt\u00faru pripraven\u00fa na n\u00e1vrat nasp\u00e4\u0165 v priebehu prv\u00fdch nieko\u013ek\u00fdch dn\u00ed po migr\u00e1cii.<\/p>\n<h3>5. Rezervn\u00e9 kapacity pod&#8217;ov a ich vyu\u017eitie<\/h3>\n<p>\nTakmer okam\u017eite sa objavil \u010fal\u0161\u00ed probl\u00e9m: pod&#8217;y pre slu\u017ebu Registry sa \u0161tartovali r\u00fdchlo, av\u0161ak spustenie pod&#8217;ov pre Sidekiq trvalo a\u017e <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\/gitlab\/-\/issues\/1775\">dve min\u00faty<\/a><\/noindex>. Pred\u013a\u017een\u00e9 sp\u00fa\u0161\u0165anie pod&#8217;ov pre Sidekiq sa stalo probl\u00e9mom, ke\u010f sme za\u010dali migr\u00e1ciu pracovn\u00fdch z\u00e1\u0165a\u017e\u00ed worker&#8217;ov do Kubernetes, ktor\u00ed musia r\u00fdchlo spracov\u00e1va\u0165 job&#8217;y a r\u00fdchlo sa \u0161k\u00e1lova\u0165.<\/p>\n<p>V tomto pr\u00edpade sa pou\u010denie sklad\u00e1 v tom, \u017ee hoci Horizontal Pod Autoscaler (HPA) v Kubernetes dobre zvl\u00e1da n\u00e1rast prev\u00e1dzky, je d\u00f4le\u017eit\u00e9 zoh\u013eadni\u0165 charakteristiky pracovn\u00fdch z\u00e1\u0165a\u017e\u00ed a vy\u010dleni\u0165 rezervn\u00e9 kapacity podov (najm\u00e4 v podmienkach nerovnomern\u00e9ho rozdelenia dopytu). V na\u0161om pr\u00edpade sme zaznamenali n\u00e1hly n\u00e1rast \u00faloh, \u010do viedlo k r\u00fdchlemu \u0161k\u00e1lovaniu a satur\u00e1cii CPU zdrojov sk\u00f4r, ako sme stihli roz\u0161\u00edri\u0165 skupinu uzlov.<\/p>\n<p>V\u017edy tu je poku\u0161enie \u201evy\u017em\u00fdka\u0165\u201c z klastra \u010do najviac, av\u0161ak po po\u010diato\u010dn\u00fdch probl\u00e9moch s v\u00fdkonom za\u010d\u00edname so \u0161tedr\u00fdm rozpo\u010dtom podov a nesk\u00f4r ho zni\u017eujeme, starostlivo sleduj\u00fac SLO. Spustenie podov pre slu\u017ebu Sidekiq sa v\u00fdrazne ur\u00fdchlilo a teraz v priemere trv\u00e1 pribli\u017ene 40 sek\u00fand. <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\/gitlab\/-\/issues\/1775\">Zn\u00ed\u017eenie \u010dasu spustenia podov<\/a><\/noindex> t\u00fdm profitovali ako GitLab.com, tak aj na\u0161i pou\u017e\u00edvatelia in\u0161tal\u00e1ci\u00ed self-managed, ktor\u00ed pracuj\u00fa s ofici\u00e1lnym Helm chartom GitLab.<\/p>\n<h2>Z\u00e1ver<\/h2>\n<p>\nPo presune ka\u017edej slu\u017eby sme sa te\u0161ili z v\u00fdhod pou\u017e\u00edvania Kubernetes v produkcii: r\u00fdchlej\u0161iemu a bezpe\u010dnej\u0161iemu nasadeniu aplik\u00e1cie, \u0161k\u00e1lovaniu a efekt\u00edvnej\u0161iemu rozdeleniu zdrojov. Navy\u0161e s\u00fa v\u00fdhody migr\u00e1cie nad r\u00e1mec slu\u017eby GitLab.com. Z ka\u017ed\u00e9ho zlep\u0161enia ofici\u00e1lneho Helm chartu profituj\u00fa aj jeho pou\u017e\u00edvatelia.<\/p>\n<p>D\u00fafam, \u017ee sa v\u00e1m p\u00e1\u010dila na\u0161a hist\u00f3ria o na\u0161ich dobrodru\u017estv\u00e1ch s migr\u00e1ciou do Kubernetes. Pokra\u010dujeme v prenose nov\u00fdch slu\u017eieb do klastru. \u010eal\u0161ie inform\u00e1cie z\u00edskate z nasleduj\u00facich publik\u00e1ci\u00ed:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/infrastructure\/production\/kubernetes\/gitlab-com\/\">Pre\u010do migrujeme do Kubernetes?<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/infrastructure\/production\/architecture\/#gitlab-com-on-kubernetes\">GitLab.com na Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/groups\/gitlab-com\/gl-infra\/-\/epics\/112\">Epic migr\u00e1cia GitLab.com na Kubernetes<\/a><\/noindex>.<\/li>\n<\/ul>\n<p><\/p>\n<h2>P.S. od prekladate\u013ea<\/h2>\n<p>\nPre\u010d\u00edtajte si tie\u017e na na\u0161om blogu:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/519962\/\">3 roky s Kubernetes v produkcii: toto sme pochopili<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/504396\/\">10 typick\u00fdch ch\u00fdb pri pou\u017e\u00edvan\u00ed Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/335814\/\">Pr\u00edbehy \u00faspechu Kubernetes v produkcii. \u010cas\u0165 3: GitHub<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/440278\/\">Pr\u00edbehy \u00faspechu Kubernetes v produkcii. \u010cas\u0165 10: Reddit<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/520150\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438. \u0422\u0435\u043c \u043d\u0435 \u043c\u0435\u043d\u0435\u0435, \u0434\u043e \u043d\u0435\u0434\u0430\u0432\u043d\u0435\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0430 \u043e\u043d\u043b\u0430\u0439\u043d-\u0441\u0435\u0440\u0432\u0438\u0441\u0430 GitLab.com \u0431\u044b\u043b\u0430 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0430 \u043d\u0430 \u0432\u0438\u0440\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0445 \u043c\u0430\u0448\u0438\u043d\u0430\u0445, \u0438 \u0442\u043e\u043b\u044c\u043a\u043e \u043e\u043a\u043e\u043b\u043e \u0433\u043e\u0434\u0430 \u043d\u0430\u0437\u0430\u0434 \u043d\u0430\u0447\u0430\u043b\u0430\u0441\u044c \u0435\u0451 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u044f \u0432 K8s, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0434\u043e \u0441\u0438\u0445 \u043f\u043e\u0440 \u043d\u0435 \u0437\u0430\u0432\u0435\u0440\u0448\u0435\u043d\u0430. \u0420\u0430\u0434\u044b \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043d\u0435\u0434\u0430\u0432\u043d\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 SRE-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u0430 GitLab \u043e \u0442\u043e\u043c, \u043a\u0430\u043a [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":94978,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-94977","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0430\u0448\u0438 \u0432\u044b\u0432\u043e\u0434\u044b \u0437\u0430 \u0433\u043e\u0434 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u0438 GitLab.com \u043d\u0430 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-09-24T05:43:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-09-24T05:43:00+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Na\u0161e zistenia za rok migr\u00e1cie GitLab.com na Kubernetes | ProHoster","description":"Pozn. prekl.: adapt\u00e1ciu Kubernetes v GitLab pova\u017euj\u00fa za jeden z dvoch hlavn\u00fdch faktorov, ktor\u00e9 prispievaj\u00fa k rastu spolo\u010dnosti.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0430\u0448\u0438 \u0432\u044b\u0432\u043e\u0434\u044b \u0437\u0430 \u0433\u043e\u0434 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u0438 GitLab.com \u043d\u0430 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-24T05:43:00+00:00","article:modified_time":"2020-09-24T05:43:00+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"94977","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:14:45","updated":"2022-10-03 07:39:43","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/94977","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=94977"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/94977\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/94978"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=94977"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=94977"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=94977"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}