{"id":94977,"date":"2020-09-24T07:43:00","date_gmt":"2020-09-24T05:43:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes"},"modified":"2020-09-24T07:43:00","modified_gmt":"2020-09-24T05:43:00","slug":"nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","title":{"rendered":"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Pozn. p\u0159ek.<\/b>: adaptace Kubernetes v GitLab je pova\u017eov\u00e1na za jeden ze dvou hlavn\u00edch faktor\u016f p\u0159isp\u00edvaj\u00edc\u00edch k r\u016fstu spole\u010dnosti. Nicm\u00e9n\u011b, a\u017e do ned\u00e1vn\u00e9 doby byla infrastruktura online slu\u017eby GitLab.com postavena na virtu\u00e1ln\u00edch stroj\u00edch, a teprve p\u0159ed rokem za\u010dala jej\u00ed migrace do K8s, kter\u00e1 dosud nebyla dokon\u010dena. R\u00e1di bychom p\u0159edstavili p\u0159eklad ned\u00e1vn\u00e9ho \u010dl\u00e1nku SRE in\u017een\u00fdra GitLab o tom, jak to prob\u00edh\u00e1 a jak\u00e9 z\u00e1v\u011bry d\u011blaj\u00ed in\u017een\u00fd\u0159i \u00fa\u010dastn\u00edc\u00ed se projektu.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/58429abf60cd19a49c5ce593051c2fca.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa\u0161e infrastrukturn\u00ed odd\u011blen\u00ed se ji\u017e t\u00e9m\u011b\u0159 rok zab\u00fdv\u00e1 migrac\u00ed v\u0161ech slu\u017eeb, kter\u00e9 funguj\u00ed na GitLab.com, do Kubernetes. B\u011bhem t\u00e9to doby jsme \u010delili probl\u00e9m\u016fm souvisej\u00edc\u00edm nejen s p\u0159esunem slu\u017eeb do Kubernetes, ale tak\u00e9 s \u0159\u00edzen\u00edm hybridn\u00edho nasazen\u00ed b\u011bhem p\u0159echodu. O cenn\u00fdch lekc\u00edch, kter\u00e9 jsme se nau\u010dili, pojedn\u00e1v\u00e1 tento \u010dl\u00e1nek.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Od po\u010d\u00e1tku GitLab.com fungovaly jeho servery v cloudu na virtu\u00e1ln\u00edch stroj\u00edch. T\u011bmito virtu\u00e1ln\u00edmi stroji spravuje Chef a jejich instalace prob\u00edh\u00e1 prost\u0159ednictv\u00edm na\u0161eho <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/install\/#ubuntu\">ofici\u00e1ln\u00edho Linux bal\u00ed\u010dku<\/a><\/noindex>. <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/release\/docs\/-\/blob\/master\/general\/deploy\/gitlab-com-deployer.md\">Strategie nasazen\u00ed<\/a><\/noindex> pro p\u0159\u00edpad, \u017ee je pot\u0159eba aktualizovat aplikaci, spo\u010d\u00edv\u00e1 v jednoduch\u00e9 aktualizaci sady server\u016f koordinovan\u00fdm postupn\u00fdm zp\u016fsobem pomoc\u00ed CI pipeline. Tato metoda \u2014 i kdy\u017e pomal\u00e1 a pon\u011bkud <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/values\/#boring-solutions\">nudn\u00e1<\/a><\/noindex> \u2014 zaru\u010duje, \u017ee GitLab.com pou\u017e\u00edv\u00e1 stejn\u00e9 metody instalace a konfigurace jako u\u017eivatel\u00e9 samostatn\u00fdch <i>(self-managed)<\/i> instalac\u00ed GitLab, vyu\u017e\u00edvaj\u00edc\u00edch k tomu na\u0161e Linux bal\u00ed\u010dky.<\/p>\n<p>Tento p\u0159\u00edstup vyu\u017e\u00edv\u00e1me, proto\u017ee je mimo\u0159\u00e1dn\u011b d\u016fle\u017eit\u00e9 pro\u017e\u00edt v\u0161echny starosti a radosti, kter\u00e9 za\u017e\u00edvaj\u00ed b\u011b\u017en\u00ed \u010dlenov\u00e9 komunity p\u0159i instalaci a nastavov\u00e1n\u00ed sv\u00fdch kopi\u00ed GitLab. Tento p\u0159\u00edstup dob\u0159e fungoval n\u011bjakou dobu, av\u0161ak kdy\u017e po\u010det projekt\u016f na GitLab p\u0159ekro\u010dil 10 milion\u016f, uv\u011bdomili jsme si, \u017ee ji\u017e nevyhovuje na\u0161im pot\u0159eb\u00e1m v oblasti \u0161k\u00e1lovatelnosti a nasazen\u00ed.<\/p>\n<h2>Prvn\u00ed kroky k Kubernetes a cloud-native GitLab<\/h2>\n<p>\nV roce 2017 byl vytvo\u0159en projekt <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\">GitLab Charts<\/a><\/noindex> pro p\u0159\u00edpravu GitLab k nasazen\u00ed v cloudu a tak\u00e9 pro umo\u017en\u011bn\u00ed u\u017eivatel\u016fm instalovat GitLab do Kubernetes klastr\u016f. U\u017e jsme v\u011bd\u011bli, \u017ee p\u0159esun GitLab do Kubernetes zv\u00fd\u0161\u00ed mo\u017enosti \u0161k\u00e1lov\u00e1n\u00ed SaaS platformy, usnadn\u00ed nasazen\u00ed a zlep\u0161\u00ed efektivitu vyu\u017eit\u00ed v\u00fdpo\u010detn\u00edch zdroj\u016f. Z\u00e1rove\u0148 v\u0161ak mnoho funkc\u00ed na\u0161\u00ed aplikace z\u00e1viselo na p\u0159ipojen\u00fdch NFS odd\u00edlech, co\u017e zpomalilo p\u0159echod z virtu\u00e1ln\u00edch stroj\u016f.<\/p>\n<p>Sna\u017een\u00ed o cloud native a Kubernetes umo\u017enilo na\u0161im in\u017een\u00fdr\u016fm pl\u00e1novat postupn\u00fd p\u0159echod, b\u011bhem n\u011bho\u017e jsme se vzdali n\u011bkter\u00fdch z\u00e1vislost\u00ed aplikace na s\u00ed\u0165ov\u00fdch \u00falo\u017ei\u0161t\u00edch, a z\u00e1rove\u0148 pokra\u010dovali ve v\u00fdvoji nov\u00fdch funkc\u00ed. Od t\u00e9 doby, co jsme za\u010dali pl\u00e1novat migraci v l\u00e9t\u011b 2019, bylo odstran\u011bno mnoho t\u011bchto omezen\u00ed a proces p\u0159enosu GitLab.com na Kubernetes nyn\u00ed prob\u00edh\u00e1 na pln\u00e9 obr\u00e1tky!<\/p>\n<h2>Vlastnosti fungov\u00e1n\u00ed GitLab.com v Kubernetes<\/h2>\n<p>\nPro GitLab.com pou\u017e\u00edv\u00e1me jednotn\u00fd region\u00e1ln\u00ed cluster GKE, kter\u00fd zpracov\u00e1v\u00e1 ve\u0161ker\u00fd provoz aplikace. Abychom minimalizovali slo\u017eitost (ji\u017e tak n\u00e1ro\u010dn\u00e9) migrace, soust\u0159ed\u00edme se na slu\u017eby, kter\u00e9 nejsou z\u00e1visl\u00e9 na m\u00edstn\u00edm \u00falo\u017ei\u0161ti nebo NFS. GitLab.com v p\u0159ev\u00e1\u017en\u00e9 v\u011bt\u0161in\u011b vyu\u017e\u00edv\u00e1 monolitn\u00ed k\u00f3dovou z\u00e1kladnu na Rails, a my sm\u011brujeme provoz na z\u00e1klad\u011b charakteristik pracovn\u00ed z\u00e1t\u011b\u017ee na r\u016fzn\u00e9 endpointy, izolovan\u00e9 ve vlastn\u00edch skupin\u00e1ch uzl\u016f.<\/p>\n<p>Pokud jde o frontend, tyto typy se d\u011bl\u00ed na po\u017eadavky na web, API, Git SSH\/HTTPS a Registry. V p\u0159\u00edpad\u011b backendu rozd\u011blujeme \u00falohy ve front\u00e1ch podle r\u016fzn\u00fdch charakteristik v z\u00e1vislosti na <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/blog\/2020\/06\/24\/scaling-our-use-of-sidekiq\/\">p\u0159edem stanoven\u00fdch omezen\u00edch zdroj\u016f<\/a><\/noindex>, kter\u00e9 n\u00e1m umo\u017e\u0148uj\u00ed nastavovat c\u00edlov\u00e9 ukazatele \u00farovn\u011b slu\u017eeb (Service-Level Objectives, SLOs) pro r\u016fzn\u00e9 z\u00e1t\u011b\u017ee.<\/p>\n<p>V\u0161echny tyto slu\u017eby GitLab.com jsou nakonfigurov\u00e1ny pomoc\u00ed nemodifikovan\u00e9ho Helm chartu GitLab. Konfigurace prob\u00edh\u00e1 v subchartech, kter\u00e9 mohou b\u00fdt selektivn\u011b zahrnuty, jak postupn\u011b p\u0159ev\u00e1d\u00edme slu\u017eby do clustru. I p\u0159es rozhodnut\u00ed nezahrnout do migrace n\u011bkter\u00e9 na\u0161e stateful slu\u017eby, jako jsou Redis, Postgres, GitLab Pages a Gitaly, umo\u017e\u0148uje pou\u017eit\u00ed Kubernetes radik\u00e1ln\u011b sn\u00ed\u017eit po\u010det VM, kter\u00e9 nyn\u00ed spravuje Chef.<\/p>\n<h2>Transparentnost a spr\u00e1va konfigurace Kubernetes<\/h2>\n<p>\nV\u0161echna nastaven\u00ed spravuje samotn\u00fd GitLab. K tomu pou\u017e\u00edv\u00e1me t\u0159i konfigura\u010dn\u00ed projekty zalo\u017een\u00e9 na Terraformu a Helm. Sna\u017e\u00edme se v\u0161ude, kde je to mo\u017en\u00e9, vyu\u017e\u00edvat samotn\u00fd GitLab k nasazen\u00ed GitLab, ale pro provozn\u00ed \u00fakoly m\u00e1me samostatnou instalaci GitLab. Ta je pot\u0159ebn\u00e1, aby byla zaji\u0161t\u011bna nez\u00e1vislost na dostupnosti GitLab.com p\u0159i prov\u00e1d\u011bn\u00ed nasazen\u00ed a aktualizac\u00ed GitLab.com.<\/p>\n<p>I kdy\u017e na\u0161e pipeline pro klastr Kubernetes funguj\u00ed na odd\u011blen\u00e9 instalaci GitLabu, existuj\u00ed zrcadla repozit\u00e1\u0159\u016f k\u00f3du, kter\u00e9 jsou ve\u0159ejn\u011b dostupn\u00e9 na n\u00e1sleduj\u00edc\u00edch adres\u00e1ch:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-com\">k8s-workloads\/gitlab-com<\/a><\/noindex> \u2014 konfigura\u010dn\u00ed obal GitLab.com pro Helm chart GitLab;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-helmfiles\/\">k8s-workloads\/gitlab-helmfiles<\/a><\/noindex> \u2014 obsahuje konfigurace pro slu\u017eby, kter\u00e9 nejsou p\u0159\u00edmo spojen\u00e9 s aplikac\u00ed GitLab. Mezi n\u011b pat\u0159\u00ed konfigurace pro protokolov\u00e1n\u00ed a monitorov\u00e1n\u00ed klastru, stejn\u011b jako pro integrovan\u00e9 n\u00e1stroje jako PlantUML;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gitlab-com-infrastructure\">Gitlab-com-infrastructure<\/a><\/noindex> \u2014 konfigurace Terraformu pro Kubernetes a starou (legacy) VM infrastrukturu. Zde se nastavuj\u00ed v\u0161echny zdroje pot\u0159ebn\u00e9 pro spu\u0161t\u011bn\u00ed klastru, v\u010detn\u011b samotn\u00e9ho klastru, pool\u016f uzl\u016f, \u00fa\u010dt\u016f slu\u017eeb a rezervace IP adres.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/612125403171d73106a081bf4244a52b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>P\u0159i prov\u00e1d\u011bn\u00ed zm\u011bn se zobrazuje ve\u0159ejn\u011b p\u0159\u00edstupn\u00e9 <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/k8s-workloads\/gitlab-com\/-\/merge_requests\/315#note_390180361\"><i>stru\u010dn\u00e9 shrnut\u00ed<\/i><\/a><\/noindex><i> s odkazem na podrobn\u00fd diff, kter\u00fd SRE analyzuje p\u0159ed proveden\u00edm zm\u011bn v klastru.<\/i><\/p>\n<p>Pro SRE odkaz vede na podrobn\u00fd diff v instalaci GitLab, kter\u00e1 se pou\u017e\u00edv\u00e1 pro provoz a jej\u00ed\u017e p\u0159\u00edstup je omezen. To umo\u017e\u0148uje zam\u011bstnanc\u016fm a komunit\u011b bez p\u0159\u00edstupu k provozn\u00edmu projektu (ten je otev\u0159en pouze pro SRE) prohl\u00ed\u017eet navrhovan\u00e9 zm\u011bny v konfiguraci. Kombinov\u00e1n\u00edm ve\u0159ejn\u00e9ho instance GitLab pro k\u00f3d s uzav\u0159enou instanc\u00ed pro CI pipeline udr\u017eujeme jednotn\u00fd pracovn\u00ed proces a z\u00e1rove\u0148 zaru\u010dujeme nez\u00e1vislost na GitLab.com p\u0159i aktualizac\u00edch konfigurace.<\/p>\n<h2>Co jsme zjistili b\u011bhem migrace<\/h2>\n<p>\nB\u011bhem p\u0159esunu jsme nasb\u00edrali zku\u0161enosti, kter\u00e9 aplikujeme na nov\u00e9 migrace a nasazen\u00ed v Kubernetes.<\/p>\n<h3>1. R\u016fst n\u00e1klad\u016f kv\u016fli provozn\u00edmu p\u0159enosu mezi z\u00f3nami dostupnosti<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/3dce44b3f803ffcea13e0101e7343d91.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Ka\u017edodenn\u00ed egress statistika (byty za den) pro sadu Git repozit\u00e1\u0159\u016f na GitLab.com<\/i><\/p>\n<p>Google d\u011bl\u00ed svou s\u00ed\u0165 na regiony. Ty se n\u00e1sledn\u011b d\u011bl\u00ed na dostupnostn\u00ed z\u00f3ny (AZ). Git hosting souvis\u00ed s velk\u00fdmi objemy dat, proto je pro n\u00e1s d\u016fle\u017eit\u00e9 kontrolovat s\u00ed\u0165ov\u00fd egress. V p\u0159\u00edpad\u011b vnit\u0159n\u00edho provozu je egress zdarma pouze v p\u0159\u00edpad\u011b, \u017ee z\u016fst\u00e1v\u00e1 v r\u00e1mci jedn\u00e9 dostupnostn\u00ed z\u00f3ny. K datu seps\u00e1n\u00ed tohoto \u010dl\u00e1nku odes\u00edl\u00e1me p\u0159ibli\u017en\u011b 100 TB dat v b\u011b\u017en\u00fd pracovn\u00ed den (a to plat\u00ed pouze pro Git repozit\u00e1\u0159e). Slu\u017eby, kter\u00e9 byly v na\u0161\u00ed star\u00e9 topologii zalo\u017een\u00e9 na VM na stejn\u00fdch virtu\u00e1ln\u00edch stroj\u00edch, nyn\u00ed b\u011b\u017e\u00ed v r\u016fzn\u00fdch pody Kubernetes. To znamen\u00e1, \u017ee \u010d\u00e1st provozu, kter\u00e1 byla d\u0159\u00edve lok\u00e1ln\u00ed pro VM, m\u016f\u017ee potenci\u00e1ln\u011b p\u0159ekro\u010dit hranice dostupnostn\u00edch z\u00f3n.<\/p>\n<p>Region\u00e1ln\u00ed clustry GKE umo\u017e\u0148uj\u00ed pokr\u00fdt n\u011bkolik z\u00f3n dostupnosti pro redundanci. Zva\u017eujeme mo\u017enost <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/delivery\/-\/issues\/1175\">rozd\u011blit region\u00e1ln\u00ed cluster GKE na jednosetov\u00e9 clustry<\/a><\/noindex> pro slu\u017eby, kter\u00e9 generuj\u00ed velk\u00e9 objemy provozu. To povede ke sn\u00ed\u017een\u00ed n\u00e1klad\u016f na egress p\u0159i zachov\u00e1n\u00ed redundance na \u00farovni clusteru.<\/p>\n<h3>2. Limitace, po\u017eadavky na zdroje a \u0161k\u00e1lov\u00e1n\u00ed<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/6e2e1ca4d37666b49358d94cd8660c56.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Po\u010det replik zpracov\u00e1vaj\u00edc\u00edch produk\u010dn\u00ed provoz na registry.gitlab.com. Provoz dosahuje sv\u00e9ho vrcholu kolem ~15:00 UTC.<\/i><\/p>\n<p>Na\u0161e historie migrace za\u010dala v srpnu 2019, kdy\u017e jsme p\u0159esunuli prvn\u00ed slu\u017ebu \u2014 registr kontejner\u016f GitLab (GitLab Container Registry) \u2014 do Kubernetes. Tato kritick\u00e1 slu\u017eba s vysok\u00fdm provozem byla vhodn\u00e1 pro prvn\u00ed migraci, proto\u017ee se jedn\u00e1 o stateless aplikaci s mal\u00fdm po\u010dtem extern\u00edch z\u00e1vislost\u00ed. Prvn\u00edm probl\u00e9mem, kter\u00fd jsme zaznamenali, bylo velk\u00e9 mno\u017estv\u00ed vytla\u010den\u00fdch pod\u016f kv\u016fli nedostatku pam\u011bti na uzlech. To n\u00e1s donutilo zm\u011bnit po\u017eadavky a limity.<\/p>\n<p>Bylo zji\u0161t\u011bno, \u017ee v p\u0159\u00edpad\u011b aplikace, jej\u00ed\u017e spot\u0159eba pam\u011bti roste v \u010dase, n\u00edzk\u00e9 hodnoty po\u017eadavk\u016f (rezervuj\u00edc\u00ed pam\u011b\u0165 pro ka\u017ed\u00fd pod) ve spojen\u00ed s \"\u0161t\u011bdr\u00fdm\" pevn\u00fdm limitem na vyu\u017eit\u00ed vedly k saturaci <i>(saturation)<\/i> uzl\u016f a vysok\u00e9 \u00farovni vytla\u010den\u00ed. Abychom se s t\u00edmto probl\u00e9mem vypo\u0159\u00e1dali, bylo <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/delivery\/-\/issues\/998#note_388983696\">rozhodnuto zv\u00fd\u0161it po\u017eadavky a sn\u00ed\u017eit limity<\/a><\/noindex>. To ulevilo uzl\u016fm a zajistilo \u017eivotn\u00ed cyklus pod\u016f, kter\u00fd nevytv\u00e1\u0159el p\u0159\u00edli\u0161 vysok\u00fd tlak na uzel. Nyn\u00ed za\u010d\u00edn\u00e1me migrace s \u0161t\u011bdr\u00fdmi (a t\u00e9m\u011b\u0159 stejn\u00fdmi) hodnotami po\u017eadavk\u016f a limit\u016f, kter\u00e9 po nutnosti upravujeme.<\/p>\n<h3>3. Met\u0440\u0438\u043a\u0438 a logy<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Na\u0161e z\u00e1v\u011bry po roce migrace GitLab.com na Kubernetes\" src=\"\/wp-content\/uploads\/2020\/09\/66d1fd47b57d5826f13defa6e6a7fe3c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Infrastrukturn\u00ed odd\u011blen\u00ed se zam\u011b\u0159uje na latence, procento chyb a saturaci s nastaven\u00fdmi <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Service-level_objective\"><i>c\u00edli \u00farovn\u011b slu\u017eeb<\/i><\/a><\/noindex><i> (SLO), v\u00e1zan\u00fdmi na <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/dashboards-gitlab-com\/-\/metrics\/sla-dashboard.yml?environment=1790496&amp;duration_seconds=86400\"><i>celkovou dostupnost na\u0161eho syst\u00e9mu<\/i><\/a><\/noindex><i>.<\/i><\/p>\n<p>B\u011bhem uplynul\u00e9ho roku se jedn\u00edm z kl\u00ed\u010dov\u00fdch ud\u00e1lost\u00ed v infrastrukturn\u00edm odd\u011blen\u00ed stala zlep\u0161en\u00ed v monitorov\u00e1n\u00ed a pr\u00e1ci se SLO. SLO n\u00e1m umo\u017enily stanovovat c\u00edle pro jednotliv\u00e9 slu\u017eby, kter\u00e9 jsme pe\u010dliv\u011b sledovali b\u011bhem migrace. Ale i s takto vylep\u0161enou viditelnost\u00ed ne v\u017edy lze okam\u017eit\u011b vid\u011bt probl\u00e9my, pomoc\u00ed metrik a alert\u016f. Nap\u0159\u00edklad, zam\u011b\u0159en\u00edm na latence a procento chyb nekryjeme \u00fapln\u011b v\u0161echny sc\u00e9n\u00e1\u0159e pou\u017e\u00edv\u00e1n\u00ed slu\u017eby, kter\u00e1 proch\u00e1z\u00ed migrac\u00ed.<\/p>\n<p>Tento probl\u00e9m byl odhalen t\u00e9m\u011b\u0159 okam\u017eit\u011b po p\u0159esunu \u010d\u00e1sti pracovn\u00edch z\u00e1t\u011b\u017e\u00ed do clusteru. Zvl\u00e1\u0161\u0165 v\u00fdrazn\u011b se projevoval, kdy\u017e bylo t\u0159eba zkontrolovat funkce, jejich\u017e po\u010det po\u017eadavk\u016f nen\u00ed velk\u00fd, ale maj\u00ed velmi specifick\u00e9 konfigura\u010dn\u00ed z\u00e1vislosti. Jedn\u00edm z kl\u00ed\u010dov\u00fdch pou\u010den\u00ed po migraci byla nutnost br\u00e1t p\u0159i monitorov\u00e1n\u00ed v \u00favahu nejen metriky, ale tak\u00e9 logy a \u201edlouh\u00fd ocas\u201d <i>(jde o <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Long_tail\"><i>takov\u00e9 jejich rozlo\u017een\u00ed<\/i><\/a><\/noindex><i> na grafu \u2014 pozn. p\u0159ekl.)<\/i> chyb. Nyn\u00ed pro ka\u017edou migraci zahrnujeme podrobn\u00fd seznam po\u017eadavk\u016f na logy <i>(log queries)<\/i> a pl\u00e1nujeme jasn\u00e9 postupy pro vr\u00e1cen\u00ed, kter\u00e9 lze v p\u0159\u00edpad\u011b probl\u00e9m\u016f p\u0159en\u00e1\u0161et z jedn\u00e9 sm\u011bny na druhou.<\/p>\n<p>Paraleln\u00ed obsluha stejn\u00fdch po\u017eadavk\u016f na star\u00e9 VM infrastruktu\u0159e a nov\u00e9, zalo\u017een\u00e9 na Kubernetes, p\u0159edstavovala unik\u00e1tn\u00ed \u00fakol. Na rozd\u00edl od migrace typu lift-and-shift <i>(rychl\u00fd p\u0159esun aplikac\u00ed \u201etak jak jsou\u201c do nov\u00e9 infrastruktury; v\u00edce informac\u00ed lze p\u0159e\u010d\u00edst nap\u0159\u00edklad, <\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/www.ibm.com\/cloud\/learn\/lift-and-shift\"><i>zde<\/i><\/a><\/noindex><i> \u2014 pozn. p\u0159ekl.)<\/i>, paraleln\u00ed pr\u00e1ce na \u201estar\u00fdch\u201c VM a Kubernetes vy\u017eaduje, aby n\u00e1stroje pro monitorov\u00e1n\u00ed byly kompatibiln\u00ed s ob\u011bma prost\u0159ed\u00edmi a um\u011bly spojovat metriky do jedn\u00e9 formy. D\u016fle\u017eit\u00e9 je, \u017ee pou\u017e\u00edv\u00e1me stejn\u00e9 dashboardy a dotazy na logy, abychom dos\u00e1hli konzistentn\u00ed observability b\u011bhem p\u0159echodn\u00e9ho obdob\u00ed.<\/p>\n<h3>4. P\u0159epnut\u00ed provozu na nov\u00fd cluster<\/h3>\n<p>\nPro GitLab.com je \u010d\u00e1st server\u016f vyhrazena pro <noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/#canary-testing\">kan\u00e1rkovou (canary) f\u00e1zi<\/a><\/noindex>. Kan\u00e1rkov\u00fd park obsluhuje na\u0161e vnit\u0159n\u00ed projekty a tak\u00e9 m\u016f\u017ee <noindex><a rel=\"nofollow\" href=\"https:\/\/next.gitlab.com\/\">b\u00fdt aktivov\u00e1n u\u017eivateli<\/a><\/noindex>. Ale p\u0159edev\u0161\u00edm je ur\u010den pro testov\u00e1n\u00ed zm\u011bn v infrastruktu\u0159e a aplikaci. Prvn\u00ed p\u0159esunut\u00e1 slu\u017eba za\u010dala p\u0159ij\u00edmat omezen\u00fd objem intern\u00edho provozu a tento p\u0159\u00edstup nad\u00e1le pou\u017e\u00edv\u00e1me, abychom se ujistili, \u017ee SLO je dodr\u017eov\u00e1no, ne\u017e nasm\u011brujeme ve\u0161ker\u00fd provoz do clusteru.<\/p>\n<p>V p\u0159\u00edpad\u011b migrace to znamen\u00e1, \u017ee nejprve jsou do Kubernetes sm\u011brov\u00e1ny po\u017eadavky na intern\u00ed projekty a pot\u00e9 postupn\u011b p\u0159ep\u00edn\u00e1me na cluster zbytek provozu zm\u011bnou v\u00e1hy pro backend p\u0159es HAProxy. B\u011bhem p\u0159echodu z VM na Kubernetes se uk\u00e1zalo, \u017ee je velmi v\u00fdhodn\u00e9 m\u00edt k dispozici jednoduch\u00fd zp\u016fsob p\u0159esm\u011brov\u00e1n\u00ed provozu mezi starou a novou infrastrukturou a t\u00edm p\u00e1dem udr\u017eovat starou infrastrukturu p\u0159ipravenou na zp\u011btn\u00fd n\u00e1vrat v prvn\u00edch n\u011bkolika dnech po migraci.<\/p>\n<h3>5. Rezervn\u00ed kapacity pod\u016f a jejich vyu\u017eit\u00ed<\/h3>\n<p>\nT\u00e9m\u011b\u0159 okam\u017eit\u011b se zjistil n\u00e1sleduj\u00edc\u00ed probl\u00e9m: pody slu\u017eby Registry se spou\u0161t\u011bly rychle, av\u0161ak spu\u0161t\u011bn\u00ed pod\u016f pro Sidekiq trvalo a\u017e <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\/gitlab\/-\/issues\/1775\">dv\u011b minuty<\/a><\/noindex>. Dlouh\u00e9 spu\u0161t\u011bn\u00ed pod\u016f pro Sidekiq se stalo probl\u00e9mem, kdy\u017e jsme za\u010dali migraci pracovn\u00edch z\u00e1t\u011b\u017e\u00ed pro workery do Kubernetes, kte\u0159\u00ed pot\u0159ebuj\u00ed rychle zpracov\u00e1vat joby a rychle \u0161k\u00e1lovat.<\/p>\n<p>V tomto p\u0159\u00edpad\u011b bylo lekc\u00ed to, \u017ee i kdy\u017e Horizontal Pod Autoscaler (HPA) v Kubernetes dob\u0159e zvl\u00e1d\u00e1 zv\u00fd\u0161en\u00ed provozu, je d\u016fle\u017eit\u00e9 zohlednit charakteristiky pracovn\u00edch z\u00e1t\u011b\u017e\u00ed a vy\u010dlenit rezervn\u00ed kapacity pod\u016f (zejm\u00e9na v podm\u00ednk\u00e1ch nerovnom\u011brn\u00e9ho rozlo\u017een\u00ed popt\u00e1vky). V na\u0161em p\u0159\u00edpad\u011b do\u0161lo k n\u00e1hl\u00e9mu n\u00e1r\u016fstu \u00faloh, co\u017e vedlo k rychl\u00e9mu \u0161k\u00e1lov\u00e1n\u00ed, co\u017e m\u011blo za n\u00e1sledek saturaci zdroj\u016f CPU d\u0159\u00edve, ne\u017e jsme byli schopni \u0161k\u00e1lovat pool uzl\u016f.<\/p>\n<p>V\u017edy existuje poku\u0161en\u00ed \u201evy\u017ed\u00edmat\u201c z clusteru co nejv\u00edce, av\u0161ak jakmile jsme se pot\u00fdkali s probl\u00e9my s v\u00fdkonem, za\u010d\u00edn\u00e1me s velkorys\u00fdm rozpo\u010dtem pod\u016f a ten postupn\u011b sni\u017eujeme, pe\u010dliv\u011b sledujeme SLO. Spu\u0161t\u011bn\u00ed pod\u016f pro slu\u017ebu Sidekiq se v\u00fdrazn\u011b zrychlilo a nyn\u00ed v pr\u016fm\u011bru trv\u00e1 kolem 40 sekund. <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-org\/charts\/gitlab\/-\/issues\/1775\">Zkr\u00e1cen\u00ed doby spu\u0161t\u011bn\u00ed pod\u016f<\/a><\/noindex> t\u011b\u017eili jak GitLab.com, tak na\u0161i u\u017eivatel\u00e9 self-managed instalac\u00ed, kte\u0159\u00ed pracuj\u00ed s ofici\u00e1ln\u00edm Helm chartem GitLab.<\/p>\n<h2>Z\u00e1v\u011br<\/h2>\n<p>\nPo p\u0159enosu ka\u017ed\u00e9 slu\u017eby jsme se t\u011b\u0161ili v\u00fdhod\u00e1m pou\u017e\u00edv\u00e1n\u00ed Kubernetes v produkci: rychlej\u0161\u00edmu a bezpe\u010dn\u011bj\u0161\u00edmu nasazen\u00ed aplikace, \u0161k\u00e1lov\u00e1n\u00ed a efektivn\u011bj\u0161\u00edmu rozd\u011blov\u00e1n\u00ed prost\u0159edk\u016f. V\u00fdhody migrace nav\u00edc p\u0159esahuj\u00ed slu\u017eby GitLab.com. Z ka\u017ed\u00e9ho vylep\u0161en\u00ed ofici\u00e1ln\u00edho Helm chartu t\u011b\u017e\u00ed i jeho u\u017eivatel\u00e9.<\/p>\n<p>Douf\u00e1m, \u017ee se v\u00e1m l\u00edbila na\u0161e p\u0159\u00edb\u011bh o na\u0161ich dobrodru\u017estv\u00edch s migrac\u00ed do Kubernetes. Pokra\u010dujeme v p\u0159enosu nov\u00fdch slu\u017eeb do clusteru. Dal\u0161\u00ed informace lze nal\u00e9zt v n\u00e1sleduj\u00edc\u00edch publikac\u00edch:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/infrastructure\/production\/kubernetes\/gitlab-com\/\">Pro\u010d p\u0159ech\u00e1z\u00edme na Kubernetes?<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/about.gitlab.com\/handbook\/engineering\/infrastructure\/production\/architecture\/#gitlab-com-on-kubernetes\">GitLab.com na Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/groups\/gitlab-com\/gl-infra\/-\/epics\/112\">Epic migrace GitLab.com na Kubernetes<\/a><\/noindex>.<\/li>\n<\/ul>\n<p><\/p>\n<h2>P.S. od p\u0159ekladatele<\/h2>\n<p>\nP\u0159e\u010dt\u011bte si tak\u00e9 n\u00e1\u0161 blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/519962\/\">3 roky s Kubernetes v produkci: co jsme pochopili<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/504396\/\">10 typick\u00fdch chyb p\u0159i pou\u017e\u00edv\u00e1n\u00ed Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/335814\/\">P\u0159\u00edb\u011bhy \u00fasp\u011bchu Kubernetes v produkci. \u010c\u00e1st 3: GitHub<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/440278\/\">P\u0159echod Tinderu na Kubernetes<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/520150\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438. \u0422\u0435\u043c \u043d\u0435 \u043c\u0435\u043d\u0435\u0435, \u0434\u043e \u043d\u0435\u0434\u0430\u0432\u043d\u0435\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0430 \u043e\u043d\u043b\u0430\u0439\u043d-\u0441\u0435\u0440\u0432\u0438\u0441\u0430 GitLab.com \u0431\u044b\u043b\u0430 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0430 \u043d\u0430 \u0432\u0438\u0440\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0445 \u043c\u0430\u0448\u0438\u043d\u0430\u0445, \u0438 \u0442\u043e\u043b\u044c\u043a\u043e \u043e\u043a\u043e\u043b\u043e \u0433\u043e\u0434\u0430 \u043d\u0430\u0437\u0430\u0434 \u043d\u0430\u0447\u0430\u043b\u0430\u0441\u044c \u0435\u0451 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u044f \u0432 K8s, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0434\u043e \u0441\u0438\u0445 \u043f\u043e\u0440 \u043d\u0435 \u0437\u0430\u0432\u0435\u0440\u0448\u0435\u043d\u0430. \u0420\u0430\u0434\u044b \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043d\u0435\u0434\u0430\u0432\u043d\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 SRE-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u0430 GitLab \u043e \u0442\u043e\u043c, \u043a\u0430\u043a [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":94978,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-94977","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0430\u0448\u0438 \u0432\u044b\u0432\u043e\u0434\u044b \u0437\u0430 \u0433\u043e\u0434 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u0438 GitLab.com \u043d\u0430 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-09-24T05:43:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-09-24T05:43:00+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Na\u0161e z\u00e1v\u011bry za rok migrace GitLab.com na Kubernetes | ProHoster","description":"Pozn. p\u0159ekl.: adaptaci Kubernetes v GitLab pova\u017euj\u00ed za jeden ze dvou hlavn\u00edch faktor\u016f, kter\u00e9 p\u0159isp\u00edvaj\u00ed k r\u016fstu spole\u010dnosti.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0430\u0448\u0438 \u0432\u044b\u0432\u043e\u0434\u044b \u0437\u0430 \u0433\u043e\u0434 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u0438 GitLab.com \u043d\u0430 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0430\u0434\u0430\u043f\u0442\u0430\u0446\u0438\u044e Kubernetes \u0432 GitLab \u0441\u0447\u0438\u0442\u0430\u044e\u0442 \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u0434\u0432\u0443\u0445 \u0433\u043b\u0430\u0432\u043d\u044b\u0445 \u0444\u0430\u043a\u0442\u043e\u0440\u043e\u0432, \u0441\u043f\u043e\u0441\u043e\u0431\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u0440\u043e\u0441\u0442\u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-24T05:43:00+00:00","article:modified_time":"2020-09-24T05:43:00+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"94977","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:14:45","updated":"2022-10-03 07:39:43","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/94977","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=94977"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/94977\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/94978"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=94977"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=94977"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=94977"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}