{"id":84114,"date":"2020-06-05T07:42:55","date_gmt":"2020-06-05T05:42:55","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah"},"modified":"2020-06-05T07:42:55","modified_gmt":"2020-06-05T05:42:55","slug":"one-cloud-os-urovnya-data-czentra-v-odnoklassnikah","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah","title":{"rendered":"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/fa983acacec59ff2d4ed098f87223971.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ahoj, lidi! Jmenuji se Oleg Anastasyev, pracuji na Odnoklassnik\u00e1ch v t\u00fdmu Platformy. Krom\u011b m\u011b pracuje na Odnoklassnik\u00e1ch spousta techniky. M\u00e1me \u010dty\u0159i datov\u00e1 centra, v nich\u017e je asi 500 rack\u016f s v\u00edce ne\u017e 8000 servery. V ur\u010dit\u00e9m okam\u017eiku jsme si uv\u011bdomili, \u017ee zaveden\u00ed nov\u00e9ho syst\u00e9mu \u0159\u00edzen\u00ed n\u00e1m umo\u017en\u00ed efektivn\u011bji vyu\u017e\u00edt techniku, usnadnit spr\u00e1vu p\u0159\u00edstup\u016f, automatizovat (p\u0159e)rozd\u011blov\u00e1n\u00ed v\u00fdpo\u010detn\u00edch zdroj\u016f, zrychlit spu\u0161t\u011bn\u00ed nov\u00fdch slu\u017eeb a zrychlit reakce na masivn\u00ed hav\u00e1rie. <\/p>\n<p><\/p>\n<p>Co z toho vzniklo? <\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Krom\u011b m\u011b a spousty techniky jsou tu je\u0161t\u011b lid\u00e9, kte\u0159\u00ed s touto technikou pracuj\u00ed: in\u017een\u00fd\u0159i, kte\u0159\u00ed jsou p\u0159\u00edmo v datov\u00fdch centrech; s\u00ed\u0165a\u0159i, kte\u0159\u00ed nastavuj\u00ed s\u00ed\u0165ovou infrastrukturu; administr\u00e1to\u0159i, tedy SRE, kte\u0159\u00ed zaji\u0161\u0165uj\u00ed odolnost infrastruktury; a t\u00fdmy v\u00fdvoj\u00e1\u0159\u016f, z nich\u017e ka\u017ed\u00e1 odpov\u00edd\u00e1 za \u010d\u00e1st funkc\u00ed port\u00e1lu. Software, kter\u00fd vytv\u00e1\u0159ej\u00ed, funguje takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/592fd0acfa7322eb80fbb85601a92918.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u017d\u00e1dosti u\u017eivatel\u016f p\u0159ich\u00e1zej\u00ed jak na fronty hlavn\u00edho port\u00e1lu <noindex><a rel=\"nofollow\" href=\"http:\/\/www.ok.ru\/\">www.ok.ru<\/a><\/noindex>, tak i na dal\u0161\u00ed, nap\u0159\u00edklad na fronty API hudby. Pro zpracov\u00e1n\u00ed obchodn\u00ed logiky vyvol\u00e1vaj\u00ed server aplikac\u00ed, kter\u00fd p\u0159i zpracov\u00e1n\u00ed \u017e\u00e1dosti vol\u00e1 pot\u0159ebn\u00e9 specializovan\u00e9 mikroservisy - one-graph (graf soci\u00e1ln\u00edch vazeb), user-cache (cache u\u017eivatelsk\u00fdch profil\u016f) atd.<\/p>\n<p><\/p>\n<p>Ka\u017ed\u00e1 z t\u011bchto slu\u017eeb je nasazena na mnoha stroj\u00edch a ka\u017ed\u00e1 z nich m\u00e1 zodpov\u011bdn\u00e9 v\u00fdvoj\u00e1\u0159e, kte\u0159\u00ed odpov\u00eddaj\u00ed za spr\u00e1vu modul\u016f, jejich provoz a technologick\u00fd rozvoj. V\u0161echny tyto slu\u017eby se spou\u0161t\u011bj\u00ed na fyzick\u00fdch serverech a a\u017e do ned\u00e1vna jsme spou\u0161t\u011bli p\u0159esn\u011b jednu \u00falohu na jeden server, tj. byl specializov\u00e1n pro konkr\u00e9tn\u00ed \u00fakol.<\/p>\n<p><\/p>\n<p>Pro\u010d tomu tak bylo? Tento p\u0159\u00edstup m\u011bl n\u011bkolik v\u00fdhod:<\/p>\n<p><\/p>\n<ul>\n<li>Usnad\u0148uje <strong>masivn\u00ed spr\u00e1vu<\/strong>. P\u0159edpokl\u00e1dejme, \u017ee \u00faloha vy\u017eaduje n\u011bjak\u00e9 knihovny, n\u011bjak\u00e1 nastaven\u00ed. A pak je server p\u0159i\u0159azen pouze jedn\u00e9 konkr\u00e9tn\u00ed skupin\u011b, popisuje se politika cfengine pro tuto skupinu (nebo u\u017e je pops\u00e1na) a tato konfigurace je centr\u00e1ln\u011b a automaticky nasazena na v\u0161echny servery t\u00e9to skupiny.<\/li>\n<li>Zjednodu\u0161uje <strong>diagnostiku<\/strong>. P\u0159edstavte si, \u017ee sledujete zv\u00fd\u0161enou z\u00e1t\u011b\u017e procesoru a uv\u011bdomujete si, \u017ee tuto z\u00e1t\u011b\u017e mohla vygenerovat pouze ta \u00faloha, kter\u00e1 b\u011b\u017e\u00ed na tomto fyzick\u00e9m procesoru. Hled\u00e1n\u00ed vin\u00edka kon\u010d\u00ed velmi rychle.<\/li>\n<li>Zjednodu\u0161uje <strong>monitoring<\/strong>. Pokud je se serverem n\u011bco v nepo\u0159\u00e1dku, monitor o tom informuje a vy p\u0159esn\u011b v\u00edte, kdo je za to odpov\u011bdn\u00fd.<\/li>\n<\/ul>\n<p><\/p>\n<p>Slu\u017eba slo\u017een\u00e1 z v\u00edce replik z\u00edsk\u00e1v\u00e1 n\u011bkolik server\u016f \u2014 po jednom na ka\u017edou. V\u00fdpo\u010detn\u00ed zdroje pro slu\u017ebu jsou tedy p\u0159i\u0159azeny velmi jednodu\u0161e: kolik server\u016f slu\u017eba m\u00e1, tolik maxim\u00e1ln\u00edch zdroj\u016f m\u016f\u017ee spot\u0159ebovat. \"Jednodu\u0161e\" tady neznamen\u00e1, \u017ee je to snadn\u00e9 pou\u017e\u00edt, ale \u017ee rozd\u011blen\u00ed zdroj\u016f prob\u00edh\u00e1 ru\u010dn\u011b.<\/p>\n<p><\/p>\n<p>Takov\u00fd p\u0159\u00edstup n\u00e1m tak\u00e9 umo\u017enil prov\u00e1d\u011bt <strong>specializovan\u00e9 hardwarov\u00e9 konfigurace<\/strong> pod \u00falohu, kter\u00e1 b\u011b\u017e\u00ed na tomto serveru. Pokud \u00faloha uchov\u00e1v\u00e1 velk\u00e1 mno\u017estv\u00ed dat, pou\u017eijeme 4U server s \u0161asi pro 38 disk\u016f. Pokud je \u00faloha \u010dist\u011b v\u00fdpo\u010detn\u00ed, m\u016f\u017eeme zakoupit levn\u011bj\u0161\u00ed 1U server. To je efektivn\u00ed z pohledu v\u00fdpo\u010detn\u00edch zdroj\u016f. Tento p\u0159\u00edstup n\u00e1m tak\u00e9 umo\u017e\u0148uje pou\u017e\u00edvat \u010dty\u0159ikr\u00e1t m\u00e9n\u011b stroj\u016f p\u0159i z\u00e1t\u011b\u017ei srovnateln\u00e9 s jednou p\u0159\u00e1telskou soci\u00e1ln\u00ed s\u00edt\u00ed. <\/p>\n<p><\/p>\n<p>Takov\u00e1 efektivita vyu\u017eit\u00ed v\u00fdpo\u010detn\u00edch zdroj\u016f by m\u011bla zajistit i ekonomickou efektivitu, pokud vych\u00e1z\u00edme z p\u0159edpokladu, \u017ee nejdra\u017e\u0161\u00ed jsou servery. Dlouho to bylo pr\u00e1v\u011b hardwarov\u00e1 za\u0159\u00edzen\u00ed, kter\u00e1 st\u00e1la nejv\u00edce, a v\u011bnovali jsme mnoho \u00fasil\u00ed s c\u00edlem sn\u00ed\u017eit n\u00e1klady na hardware vym\u00fd\u0161len\u00edm algoritm\u016f pro zaji\u0161t\u011bn\u00ed odolnosti proti poruch\u00e1m za \u00fa\u010delem sn\u00ed\u017een\u00ed po\u017eadavk\u016f na spolehlivost za\u0159\u00edzen\u00ed. Dnes jsme se dostali do f\u00e1ze, kdy cena serveru ji\u017e p\u0159estala b\u00fdt ur\u010duj\u00edc\u00ed. Pokud nebereme v \u00favahu \u010derstvou exotiku, konkr\u00e9tn\u00ed konfigurace server\u016f v racku nem\u00e1 z\u00e1sadn\u00ed v\u00fdznam. Nyn\u00ed m\u00e1me jin\u00fd probl\u00e9m \u2014 cena m\u00edsta, kter\u00e9 server zab\u00edr\u00e1 v datov\u00e9m centru, tedy m\u00edsta v racku.<\/p>\n<p><\/p>\n<p>Kdy\u017e jsme si uv\u011bdomili, \u017ee je to tak, rozhodli jsme se spo\u010d\u00edtat, jak efektivn\u011b vyu\u017e\u00edv\u00e1me racky.<br \/>\nVzali jsme cenu nejsiln\u011bj\u0161\u00edho serveru z ekonomicky v\u00fdhodn\u00fdch a spo\u010d\u00edtali, kolik takov\u00fdch server\u016f se vejde do rack\u016f, kolik \u00faloh bychom na n\u011bm spustili na z\u00e1klad\u011b star\u00e9ho modelu \u201ejeden server = jedna \u00faloha\u201c a jak by takov\u00e9 \u00falohy mohly vyu\u017e\u00edvat hardware. Spo\u010d\u00edtali jsme to \u2013 a rozplakali se. Uk\u00e1zalo se, \u017ee efektivita vyu\u017eit\u00ed rack\u016f je u n\u00e1s kolem 11 %. Z\u00e1v\u011br je jasn\u00fd: je pot\u0159eba zvy\u0161ovat efektivitu vyu\u017eit\u00ed datov\u00fdch center. \u0158e\u0161en\u00ed se zd\u00e1 b\u00fdt evidentn\u00ed: na jednom serveru spustit v\u00edce \u00faloh. Ale zde nast\u00e1vaj\u00ed komplikace. <\/p>\n<p><\/p>\n<p>Hromadn\u00e1 konfigurace se drasticky komplikuje \u2014 nyn\u00ed nen\u00ed mo\u017en\u00e9 p\u0159i\u0159adit serveru n\u011bjakou jednu skupinu. Na jednom serveru mohou b\u00fdt spu\u0161t\u011bny r\u016fzn\u00e9 \u00falohy r\u016fzn\u00fdch t\u00fdm\u016f. D\u00e1le m\u016f\u017ee b\u00fdt konfigurace konfliktuj\u00edc\u00ed pro r\u016fzn\u00e9 aplikace. Diagnostika je tak\u00e9 slo\u017eit\u011bj\u0161\u00ed: pokud vid\u00edte zv\u00fd\u0161enou spot\u0159ebu procesor\u016f nebo disk\u016f na serveru, nev\u00edte, kter\u00e1 z \u00faloh zp\u016fsobuje probl\u00e9my.<\/p>\n<p><\/p>\n<p>Ale hlavn\u00ed je, \u017ee mezi \u00falohami spu\u0161t\u011bn\u00fdmi na jednom stroji nen\u00ed izolace. Nap\u0159\u00edklad graf pr\u016fm\u011brn\u00e9 doby odezvy serverov\u00e9 \u00falohy p\u0159ed a po spu\u0161t\u011bn\u00ed dal\u0161\u00edho, zcela nesouvisej\u00edc\u00edho v\u00fdpo\u010dtov\u00e9ho programu na stejn\u00e9m serveru \u2013 doba odezvy hlavn\u00ed \u00falohy se v\u00fdrazn\u011b zv\u00fd\u0161ila.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/c7c07359ae572ca8c84a4c63559e4083.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je z\u0159ejm\u00e9, \u017ee \u00falohy by m\u011bly b\u00fdt spou\u0161t\u011bny bu\u010f v kontejnerech, nebo ve virtu\u00e1ln\u00edch stroj\u00edch. Proto\u017ee t\u00e9m\u011b\u0159 v\u0161echny \u00falohy b\u011b\u017e\u00ed pod jedn\u00edm opera\u010dn\u00edm syst\u00e9mem (Linux) nebo jsou mu p\u0159izp\u016fsobeny, nepot\u0159ebujeme podporovat \u0159adu r\u016fzn\u00fdch opera\u010dn\u00edch syst\u00e9m\u016f. Vzhledem k tomu nebude virtualizace pot\u0159ebn\u00e1, proto\u017ee kv\u016fli dodate\u010dn\u00fdm n\u00e1klad\u016fm bude m\u00e9n\u011b efektivn\u00ed ne\u017e kontejnerizace.<\/p>\n<p><\/p>\n<p>Docker je dobr\u00fdm kandid\u00e1tem pro implementaci kontejner\u016f pro spou\u0161t\u011bn\u00ed \u00faloh p\u0159\u00edmo na serverech. Obrazy souborov\u00fdch syst\u00e9m\u016f dob\u0159e \u0159e\u0161\u00ed probl\u00e9my s konfliktuj\u00edc\u00edmi konfiguracemi. To, \u017ee obrazy mohou b\u00fdt slo\u017eeny z n\u011bkolika vrstev, n\u00e1m umo\u017e\u0148uje zna\u010dn\u011b sn\u00ed\u017eit objem dat pot\u0159ebn\u00fd pro jejich nasazen\u00ed na infrastruktur\u011b, kdy\u017e spole\u010dn\u00e9 \u010d\u00e1sti odd\u011bl\u00edme do samostatn\u00fdch z\u00e1kladn\u00edch vrstev. Z\u00e1kladn\u00ed (a nejobjemn\u011bj\u0161\u00ed) vrstvy se pak dostate\u010dn\u011b rychle ukl\u00e1daj\u00ed do cache v cel\u00e9 infrastruktu\u0159e a pro dod\u00e1n\u00ed mnoha r\u016fzn\u00fdch typ\u016f aplikac\u00ed a verz\u00ed bude pot\u0159eba p\u0159en\u00e1\u0161et pouze mal\u00e9 objemov\u00e9 vrstvy. <\/p>\n<p><\/p>\n<p>Nav\u00edc, hotov\u00fd registr a tagov\u00e1n\u00ed obraz\u016f v Dockeru n\u00e1m poskytuj\u00ed p\u0159ipraven\u00e9 primitivy pro verzov\u00e1n\u00ed a dod\u00e1v\u00e1n\u00ed k\u00f3du do produkce.<\/p>\n<p><\/p>\n<p>Docker, jako ka\u017ed\u00e1 jin\u00e1 podobn\u00e1 technologie, n\u00e1m z boxu poskytuje ur\u010dit\u00fd stupe\u0148 izolace kontejner\u016f. Nap\u0159\u00edklad izolace podle pam\u011bti \u2014 ka\u017ed\u00e9mu kontejneru je p\u0159id\u011blen limit na pou\u017e\u00edv\u00e1n\u00ed pam\u011bti stroje, nad kter\u00fdm nezpot\u0159ebuje v\u00edce. Tak\u00e9 je mo\u017en\u00e9 izolovat kontejnery podle pou\u017eit\u00ed CPU. Pro n\u00e1s v\u0161ak standardn\u00ed izolace byla nedostate\u010dn\u00e1. Ale o tom d\u00e1le.<\/p>\n<p><\/p>\n<p>P\u0159\u00edm\u00e9 spou\u0161t\u011bn\u00ed kontejner\u016f na serverech je jen \u010d\u00e1st probl\u00e9m\u016f. Dal\u0161\u00ed \u010d\u00e1st se t\u00fdk\u00e1 um\u00edst\u011bn\u00ed kontejner\u016f na serverech. Je t\u0159eba pochopit, kter\u00fd kontejner m\u016f\u017ee b\u00fdt um\u00edst\u011bn na kter\u00fd server. To nen\u00ed tak jednoduch\u00fd \u00fakol, proto\u017ee kontejnery je t\u0159eba um\u00edstit na servery co nejhust\u011bji, ani\u017e by se sn\u00ed\u017eila rychlost jejich pr\u00e1ce. Takov\u00e9 um\u00edst\u011bn\u00ed m\u016f\u017ee b\u00fdt slo\u017eit\u00e9 i z hlediska odolnosti proti v\u00fdpadk\u016fm. \u010casto chceme umis\u0165ovat repliky stejn\u00e9 slu\u017eby do r\u016fzn\u00fdch sk\u0159\u00edn\u00ed nebo dokonce do r\u016fzn\u00fdch s\u00e1l\u016f datov\u00e9ho centra, abychom p\u0159i v\u00fdpadku sk\u0159\u00edn\u011b nebo s\u00e1lu neztratili okam\u017eit\u011b v\u0161echny repliky slu\u017eby. <\/p>\n<p><\/p>\n<p>Ru\u010dn\u00ed rozd\u011blov\u00e1n\u00ed kontejner\u016f nen\u00ed mo\u017enost, kdy\u017e m\u00e1\u0161 8 tis\u00edc server\u016f a 8\u201416 tis\u00edc kontejner\u016f. <\/p>\n<p><\/p>\n<p>Krom\u011b toho jsme cht\u011bli d\u00e1t v\u00fdvoj\u00e1\u0159\u016fm v\u011bt\u0161\u00ed samostatnost v rozd\u011blov\u00e1n\u00ed zdroj\u016f, aby mohli sami umis\u0165ovat sv\u00e9 slu\u017eby do produkce, bez pomoci administr\u00e1tora. P\u0159itom jsme cht\u011bli zachovat kontrolu, aby n\u011bjak\u00e1 vedlej\u0161\u00ed slu\u017eba nezpot\u0159ebovala v\u0161echny zdroje na\u0161ich datov\u00fdch center. <\/p>\n<p><\/p>\n<p>Je z\u0159ejm\u00e9, \u017ee pot\u0159ebujeme \u0159\u00eddic\u00ed vrstvu, kter\u00e1 by se t\u00edmto automaticky zab\u00fdvala.<\/p>\n<p><\/p>\n<p>Tak jsme se dostali k jednoduch\u00e9mu a srozumiteln\u00e9mu obr\u00e1zku, kter\u00fd maj\u00ed v\u0161ichni architekti r\u00e1di: t\u0159i \u010dtverce. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/079df6e79874ef55b0d967fde3d5feca.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>one-cloud masters \u2014 vysoce dostupn\u00fd cluster odpov\u011bdn\u00fd za orchestraci cloudu. Developer pos\u00edl\u00e1 do masteru manifest, kter\u00fd obsahuje v\u0161echny pot\u0159ebn\u00e9 informace pro nasazen\u00ed slu\u017eby. Master na jeho z\u00e1klad\u011b d\u00e1v\u00e1 p\u0159\u00edkazy vybran\u00fdm minion\u016fm (stroj\u016fm ur\u010den\u00fdm k spu\u0161t\u011bn\u00ed kontejner\u016f). Na minionech je n\u00e1\u0161 agent, kter\u00fd p\u0159ij\u00edm\u00e1 p\u0159\u00edkaz, vyd\u00e1v\u00e1 u\u017e sv\u00e9 p\u0159\u00edkazy Dockeru a Docker konfiguruje linuxov\u00fd kernel pro spu\u0161t\u011bn\u00ed odpov\u00eddaj\u00edc\u00edho kontejneru. Krom\u011b vykon\u00e1v\u00e1n\u00ed p\u0159\u00edkaz\u016f agent neust\u00e1le informuje mastera o zm\u011bn\u00e1ch stavu jak stroje-miniona, tak spu\u0161t\u011bn\u00fdch na n\u00ed kontejner\u016f.<\/p>\n<p><\/p>\n<h2 id=\"raspredelenie-resursov\">Rozd\u011blen\u00ed zdroj\u016f<\/h2>\n<p><\/p>\n<p>A nyn\u00ed se pod\u00edv\u00e1me na \u00fakol slo\u017eit\u011bj\u0161\u00edho rozd\u011blen\u00ed zdroj\u016f pro v\u00edce minion\u016f.<\/p>\n<p><\/p>\n<p>V\u00fdpo\u010detn\u00ed zdroj v one-cloud \u2014 to je:<\/p>\n<p><\/p>\n<ul>\n<li>V\u00fdpo\u010detn\u00ed v\u00fdkon procesoru spot\u0159ebovan\u00fd konkr\u00e9tn\u00edm \u00fakolem. <\/li>\n<li>Objem pam\u011bti dostupn\u00fd \u00fakolu. <\/li>\n<li>S\u00ed\u0165ov\u00fd provoz. Ka\u017ed\u00fd z minion\u016f m\u00e1 konkr\u00e9tn\u00ed s\u00ed\u0165ov\u00e9 rozhran\u00ed s omezenou \u0161\u00ed\u0159kou p\u00e1sma, tak\u017ee nelze distribuovat \u00fakoly bez zohledn\u011bn\u00ed objemu dat p\u0159en\u00e1\u0161en\u00fdch po s\u00edti. <\/li>\n<li>Disky. Krom\u011b, o\u010dividn\u011b, m\u00edsta pro data \u00fakolu, tak\u00e9 vy\u010dle\u0148ujeme typ disku: HDD nebo SSD. Disky mohou obsluhovat kone\u010dn\u00fd po\u010det po\u017eadavk\u016f za sekundu \u2014 IOPS. Proto pro \u00fakoly, kter\u00e9 generuj\u00ed v\u00edce IOPS, ne\u017e m\u016f\u017ee jeden disk obslou\u017eit, tak\u00e9 vy\u010dle\u0148ujeme \u201espindly\u201c \u2014 tedy diskov\u00e1 za\u0159\u00edzen\u00ed, kter\u00e1 je t\u0159eba v\u00fdhradn\u011b rezervovat pro \u00fakol.<\/li>\n<\/ul>\n<p><\/p>\n<p>Tak pro n\u011bjakou slu\u017ebu, nap\u0159\u00edklad user-cache, m\u016f\u017eeme zapsat spot\u0159ebov\u00e1van\u00e9 zdroje t\u00edmto zp\u016fsobem: 400 procesorov\u00fdch jader, 2,5 T\u010d pam\u011bti, 50 Gb\/s provozu ob\u011bma sm\u011bry, 6 T\u010d m\u00edsta na HDD um\u00edst\u011bn\u00e9ho na 100 spindlech. Nebo v n\u00e1m zn\u00e1m\u011bj\u0161\u00ed form\u011b takto:<\/p>\n<p><\/p>\n<pre><code>alloc:\n    cpu: 400\n    mem: 2500\n    lan_in: 50g\n    lan_out: 50g\n    hdd:100x6T<\/code><\/pre>\n<p><\/p>\n<p>Zdroje slu\u017eby user-cache spot\u0159ebov\u00e1vaj\u00ed pouze \u010d\u00e1st v\u0161ech dostupn\u00fdch zdroj\u016f v produk\u010dn\u00ed infrastruktu\u0159e. Proto chceme zajistit, aby n\u00e1hle, kv\u016fli chyb\u011b oper\u00e1tora nebo ne, user-cache nespot\u0159eboval v\u00edce zdroj\u016f, ne\u017e mu bylo p\u0159id\u011bleno. To znamen\u00e1, \u017ee mus\u00edme limitovat zdroje. Ale na co bychom mohli nav\u00e1zat kv\u00f3tu?<\/p>\n<p><\/p>\n<p>Poj\u010fme se vr\u00e1tit k na\u0161\u00ed zna\u010dn\u011b zjednodu\u0161en\u00e9 sch\u00e9m\u011b interakce komponent a p\u0159ekreslit ji s v\u011bt\u0161\u00edm mno\u017estv\u00edm detail\u016f \u2014 takto: <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/27e5bfbffadd3d4b9fdc6bf138d135ed.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Co okam\u017eit\u011b zas\u00e1hne:<\/p>\n<p><\/p>\n<ul>\n<li>Webov\u00fd frontend a hudba pou\u017e\u00edvaj\u00ed izolovan\u00e9 klastrov\u00e9 instance stejn\u00e9ho aplika\u010dn\u00edho serveru.<\/li>\n<li>Lze vy\u010dlenit logick\u00e9 vrstvy, do kter\u00fdch tyto clustery pat\u0159\u00ed: fronty, ke\u0161ov\u00e1n\u00ed, \u00falo\u017ei\u0161t\u011b a spr\u00e1va dat.<\/li>\n<li>Frontend nen\u00ed homogenn\u00ed, skl\u00e1d\u00e1 se z r\u016fzn\u00fdch funk\u010dn\u00edch subsyst\u00e9m\u016f. <\/li>\n<li>Ke\u0161ov\u00e1n\u00ed lze tak\u00e9 rozpt\u00fdlit podle subsyst\u00e9mu, jeho\u017e data ke\u0161uj\u00ed.<\/li>\n<\/ul>\n<p><\/p>\n<p>Znovu p\u0159ekresl\u00edme obr\u00e1zek:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/905dacc0aca859e65bd33ef751a557cf.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Aha! Vid\u00edme hierarchii! To znamen\u00e1, \u017ee m\u016f\u017eeme rozd\u011blit zdroje ve v\u011bt\u0161\u00edch celc\u00edch: p\u0159i\u0159adit odpov\u011bdn\u00e9ho v\u00fdvoj\u00e1\u0159e ke uzlu t\u00e9to hierarchie, odpov\u00eddaj\u00edc\u00edho funk\u010dn\u00edmu subsyst\u00e9mu (jako je \"music\" na obr\u00e1zku), a k t\u00e9to \u00farovni hierarchie tak\u00e9 p\u0159i\u0159adit kv\u00f3tu. Takov\u00e1 hierarchie n\u00e1m tak\u00e9 umo\u017e\u0148uje flexibiln\u011bji organizovat slu\u017eby pro usnadn\u011bn\u00ed spr\u00e1vy. Nap\u0159\u00edklad v\u0161echny webov\u00e9 slu\u017eby, jeliko\u017e se jedn\u00e1 o velmi velkou skupinu server\u016f, rozd\u011blujeme na n\u011bkolik men\u0161\u00edch skupin, kter\u00e9 jsou na obr\u00e1zku zobrazeny jako group1, group2.<\/p>\n<p><\/p>\n<p>Odstran\u011bn\u00edm zbyte\u010dn\u00fdch \u010dar m\u016f\u017eeme zapsat ka\u017ed\u00fd uzel na\u0161eho obr\u00e1zku v ploch\u011bj\u0161\u00ed podob\u011b: <strong>group1.web.front<\/strong>, <strong>api.music.front<\/strong>, <strong>user-cache.cache<\/strong>.<\/p>\n<p><\/p>\n<p>Tak se dost\u00e1v\u00e1me k pojmu \"hierarchick\u00e1 fronta\". M\u00e1 sv\u00e9 jm\u00e9no, nap\u0159\u00edklad \"group1.web.front\". Na tuto frontu se p\u0159i\u0159azuje kv\u00f3ta na zdroje a pr\u00e1va u\u017eivatel\u016f. \u010clov\u011bku z DevOps ud\u011bl\u00edme pr\u00e1va na odes\u00edl\u00e1n\u00ed slu\u017eby do fronty, a takov\u00fd zam\u011bstnanec m\u016f\u017ee spou\u0161t\u011bt n\u011bco ve front\u011b, zat\u00edmco \u010dlov\u011bku z OpsDev ud\u011bl\u00edme administr\u00e1torsk\u00e1 pr\u00e1va, tak\u017ee nyn\u00ed m\u016f\u017ee frontu spravovat, p\u0159i\u0159azovat do n\u00ed lidi, ud\u011blovat t\u011bmto lidem pr\u00e1va atd. Slu\u017eby spou\u0161t\u011bn\u00e9 v t\u00e9to front\u011b budou vykon\u00e1v\u00e1ny v r\u00e1mci kv\u00f3ty fronty. Pokud v\u00fdpo\u010detn\u00ed kv\u00f3ty fronty nesta\u010d\u00ed pro sou\u010dasn\u00e9 proveden\u00ed v\u0161ech slu\u017eeb, budou se vykon\u00e1vat sekven\u010dn\u011b, a t\u00edm se vytvo\u0159\u00ed skute\u010dn\u00e1 fronta. <\/p>\n<p><\/p>\n<p>Pod\u00edvejme se na slu\u017eby podrobn\u011bji. Slu\u017eba m\u00e1 pln\u00e9 jm\u00e9no, kter\u00e9 v\u017edy zahrnuje jm\u00e9no fronty. Tak slu\u017eba webov\u00e9ho frontu bude m\u00edt n\u00e1zev <strong>ok-web.group1.web.front<\/strong>. A slu\u017eba aplika\u010dn\u00edho serveru, na kterou se obrac\u00ed, bude m\u00edt n\u00e1zev <strong>ok-app.group1.web.front<\/strong>. Ka\u017ed\u00e1 slu\u017eba m\u00e1 manifest, ve kter\u00e9m jsou uvedeny v\u0161echny pot\u0159ebn\u00e9 informace pro nasazen\u00ed na konkr\u00e9tn\u00edch stroj\u00edch: kolik zdroj\u016f tato \u00faloha pot\u0159ebuje, jak\u00e1 konfigurace je pro ni pot\u0159ebn\u00e1, kolik replik by m\u011blo b\u00fdt, vlastnosti pro zpracov\u00e1n\u00ed selh\u00e1n\u00ed t\u00e9to slu\u017eby. A po nasazen\u00ed slu\u017eby p\u0159\u00edmo na stroj\u00edch se objevuj\u00ed jej\u00ed instance. Ty jsou tak\u00e9 jednozna\u010dn\u011b pojmenov\u00e1ny \u2014 jako \u010d\u00edslo instance a n\u00e1zev slu\u017eby: <strong>1.ok-web.group1.web.front, 2.ok-web.group1.web.front, \u2026<\/strong><\/p>\n<p><\/p>\n<p>To je velmi pohodln\u00e9: pod\u00edvejte se pouze na n\u00e1zev spu\u0161t\u011bn\u00e9ho kontejneru, ihned z n\u011bj m\u016f\u017eeme zjistit mnoho informac\u00ed.<\/p>\n<p><\/p>\n<p>A nyn\u00ed se bl\u00ed\u017ee seznamme s t\u00edm, co vlastn\u011b tyto instance d\u011blaj\u00ed: s \u00falohami.<\/p>\n<p><\/p>\n<h2 id=\"klassy-izolyacii-zadach\">T\u0159\u00eddy izolace \u00faloh<\/h2>\n<p><\/p>\n<p>V\u0161echny \u00falohy v OK (a pravd\u011bpodobn\u011b i jinde) lze rozd\u011blit do skupin:<\/p>\n<p><\/p>\n<ul>\n<li><strong>\u00dalohy s kr\u00e1tk\u00fdm zpo\u017ed\u011bn\u00edm \u2014 prod<\/strong>. Pro takov\u00e9 \u00falohy a slu\u017eby je velmi d\u016fle\u017eit\u00e9 zpo\u017ed\u011bn\u00ed odpov\u011bdi (latency), jak rychle bude ka\u017ed\u00fd z po\u017eadavk\u016f zpracov\u00e1n syst\u00e9mem. P\u0159\u00edklady \u00faloh: webov\u00e9 fronty, cache, aplika\u010dn\u00ed servery, OLTP \u00falo\u017ei\u0161t\u011b atd.<\/li>\n<li><strong>\u00dalohy v\u00fdpo\u010detn\u00ed \u2014 batch<\/strong>. Zde nen\u00ed d\u016fle\u017eit\u00e1 rychlost zpracov\u00e1n\u00ed ka\u017ed\u00e9ho konkr\u00e9tn\u00edho po\u017eadavku. D\u016fle\u017eit\u00e9 je, kolik celkov\u00fdch v\u00fdpo\u010dt\u016f za ur\u010den\u00e9 (velk\u00e9) \u010dasov\u00e9 obdob\u00ed tato \u00faloha provede (throughput). Takov\u00e9 budou jak\u00e9koli \u00falohy MapReduce, Hadoop, strojov\u00e9 u\u010den\u00ed, statistika.<\/li>\n<li><strong>\u00dalohy na pozad\u00ed \u2014 idle<\/strong>. Pro takov\u00e9 \u00falohy nejsou d\u016fle\u017eit\u00e1 ani latency, ani throughput. Sem pat\u0159\u00ed r\u016fzn\u00e9 testy, migrace, p\u0159epo\u010dty, konverze dat z jednoho form\u00e1tu do druh\u00e9ho. Z jedn\u00e9 strany se podobaj\u00ed v\u00fdpo\u010detn\u00edm, z druh\u00e9 strany n\u00e1m nen\u00ed p\u0159\u00edli\u0161 jedno, jak rychle dokon\u010d\u00ed. <\/li>\n<\/ul>\n<p><\/p>\n<p>Pod\u00edvejme se, jak takov\u00e9 \u00falohy spot\u0159ebov\u00e1vaj\u00ed zdroje, nap\u0159\u00edklad centr\u00e1ln\u00ed procesor.<\/p>\n<p><\/p>\n<p><strong>\u00dalohy s kr\u00e1tk\u00fdm zpo\u017ed\u011bn\u00edm.<\/strong> Tato \u00faloha bude m\u00edt vzor spot\u0159eby CPU podobn\u00fd tomuto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/1ed82c0df76325eb30056ad9af86e86e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na zpracov\u00e1n\u00ed p\u0159ijde po\u017eadavek od u\u017eivatele, \u00faloha za\u010dne pou\u017e\u00edvat v\u0161echny dostupn\u00e9 j\u00e1dra CPU, zpracov\u00e1v\u00e1, vrac\u00ed odpov\u011b\u010f, \u010dek\u00e1 na dal\u0161\u00ed po\u017eadavek a je v klidu. Dal\u0161\u00ed po\u017eadavek p\u0159i\u0161el \u2014 op\u011bt zabralo v\u0161e, co bylo, vypo\u010d\u00edtalo, \u010dek\u00e1me na dal\u0161\u00ed.<\/p>\n<p><\/p>\n<p>Abychom zajistili minim\u00e1ln\u00ed zpo\u017ed\u011bn\u00ed pro takovou \u00falohu, mus\u00edme vz\u00edt maximum spot\u0159ebov\u00e1van\u00fdch zdroj\u016f a rezervovat pot\u0159ebn\u00fd po\u010det jader na minionu (stroji, kter\u00fd bude \u00falohu vykon\u00e1vat). Pak bude vzorec rezervace pro na\u0161i \u00falohu vypadat takto:<\/p>\n<p><\/p>\n<pre><code>alloc: cpu = 4 (max)<\/code><\/pre>\n<p><\/p>\n<p>A pokud m\u00e1me minion s 16 j\u00e1dry, m\u016f\u017eeme na n\u011bm um\u00edstit p\u0159esn\u011b \u010dty\u0159i takov\u00e9 \u00falohy. Je t\u0159eba zd\u016fraznit, \u017ee pr\u016fm\u011brn\u00e1 spot\u0159eba procesoru u t\u011bchto \u00faloh je \u010dasto velmi n\u00edzk\u00e1 \u2014 co\u017e je o\u010dividn\u00e9, proto\u017ee v\u011bt\u0161inu \u010dasu je \u00faloha v o\u010dek\u00e1v\u00e1n\u00ed po\u017eadavku a nic ned\u011bl\u00e1.<\/p>\n<p><\/p>\n<p><strong>V\u00fdpo\u010dtov\u00e9 \u00falohy.<\/strong> Jejich vzor bude pon\u011bkud odli\u0161n\u00fd:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/35b37fa1d70a137287cafdbfe3bcfc2e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pr\u016fm\u011brn\u00e1 spot\u0159eba procesorov\u00fdch zdroj\u016f u t\u011bchto \u00faloh je dost vysok\u00e1. \u010casto chceme, aby v\u00fdpo\u010dtov\u00e1 \u00faloha byla dokon\u010dena za ur\u010ditou dobu, proto je pot\u0159eba rezervovat minim\u00e1ln\u00ed po\u010det procesor\u016f, kter\u00e9 pot\u0159ebuje, aby cel\u00fd v\u00fdpo\u010det skon\u010dil v p\u0159ijateln\u00e9m \u010dase. Jej\u00ed rezerva\u010dn\u00ed vzorec bude vypadat takto:<\/p>\n<p><\/p>\n<pre><code>alloc: cpu = [1,*)<\/code><\/pre>\n<p><\/p>\n<p><em>\u201eUm\u00edsti pros\u00edm na minion, kde je alespo\u0148 jedno voln\u00e9 j\u00e1dro, a pak a\u0165 se spot\u0159ebuje, kolik je pot\u0159eba.\u201c<\/em><\/p>\n<p><\/p>\n<p>Zde je efektivita vyu\u017eit\u00ed ji\u017e v\u00fdrazn\u011b lep\u0161\u00ed ne\u017e u \u00faloh s kr\u00e1tk\u00fdm zpo\u017ed\u011bn\u00edm. Ale zisk bude mnohem v\u011bt\u0161\u00ed, pokud zkombinujeme oba typy \u00faloh na jednom minionu a budeme dynamicky rozd\u011blovat jej\u00ed zdroje. Kdy\u017e \u00faloha s kr\u00e1tk\u00fdm zpo\u017ed\u011bn\u00edm pot\u0159ebuje procesor \u2014 dostane ho okam\u017eit\u011b, a kdy\u017e zdroje nebudou pot\u0159ebn\u00e9 \u2014 p\u0159ed\u00e1vaj\u00ed se v\u00fdpo\u010dtov\u00e9 \u00faloze, tedy n\u011bjak takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/875ad8d3f6a01e4fd0a89d0931e986c1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale jak to ud\u011blat?<\/p>\n<p><\/p>\n<p>Nejd\u0159\u00edve se zam\u011b\u0159\u00edme na prod a jeho alloc: cpu = 4. Mus\u00edme rezervovat \u010dty\u0159i j\u00e1dra. V Docker run lze toho dos\u00e1hnout dv\u011bma zp\u016fsoby: <\/p>\n<p><\/p>\n<ul>\n<li>Pomoc\u00ed volby <code>--cpuset=1-4<\/code>, tedy vy\u010dlenit \u00faloze \u010dty\u0159i konkr\u00e9tn\u00ed j\u00e1dra na stroji.<\/li>\n<li>Pou\u017e\u00edt <code>--cpuquota=400_000 --cpuperiod=100_000<\/code>, p\u0159id\u011blit kv\u00f3tu na procesorov\u00fd \u010das, tedy uv\u00e9st, \u017ee ka\u017ed\u00fdch 100 ms skute\u010dn\u00e9ho \u010dasu \u00faloha spot\u0159ebov\u00e1v\u00e1 maxim\u00e1ln\u011b 400 ms procesorov\u00e9ho \u010dasu. Z\u00edsk\u00e1v\u00e1me stejn\u00e1 \u010dty\u0159i j\u00e1dra. <\/li>\n<\/ul>\n<p><\/p>\n<p>Ale kter\u00fd z t\u011bchto zp\u016fsob\u016f bude vhodn\u00fd?<\/p>\n<p><\/p>\n<p>Pom\u011brn\u011b atraktivn\u011b vypad\u00e1 cpuset. \u00daloha m\u00e1 \u010dty\u0159i vyhrazen\u00e1 j\u00e1dra, co\u017e znamen\u00e1, \u017ee procesorov\u00e9 cache budou fungovat maxim\u00e1ln\u011b efektivn\u011b. To m\u00e1 v\u0161ak i svou nev\u00fdhodu: museli bychom se postarat o distribuci v\u00fdpo\u010dt\u016f na nezat\u00ed\u017een\u00e1 j\u00e1dra stroje m\u00edsto opera\u010dn\u00edho syst\u00e9mu, co\u017e je pom\u011brn\u011b netrivi\u00e1ln\u00ed \u00fakol, zvl\u00e1\u0161\u0165 kdy\u017e se pokus\u00edme um\u00edstit na tento stroj batch-\u00falohy. Testy uk\u00e1zaly, \u017ee tady l\u00e9pe funguje varianta s kv\u00f3tou: tak m\u00e1 opera\u010dn\u00ed syst\u00e9m v\u00edce svobody p\u0159i v\u00fdb\u011bru j\u00e1dra pro vykon\u00e1n\u00ed \u00falohy v aktu\u00e1ln\u00edm okam\u017eiku a procesorov\u00fd \u010das se rozd\u011bluje efektivn\u011bji.<\/p>\n<p><\/p>\n<p>Poj\u010fme si objasnit, jak v Dockeru zajistit rezervaci na minim\u00e1ln\u00ed po\u010det jader. Kv\u00f3ta pro batch-\u00falohy u\u017e nen\u00ed pou\u017eiteln\u00e1, proto\u017ee omezovat maximum nen\u00ed pot\u0159eba, sta\u010d\u00ed pouze zajistit minimum. A tady dob\u0159e zapad\u00e1 mo\u017enost <code>docker run --cpushares<\/code>.<\/p>\n<p><\/p>\n<p>Dohodli jsme se, \u017ee pokud batch vy\u017eaduje zaru\u010den\u00ed min na jednom j\u00e1d\u0159e, tak uvedeme <code>--cpushares=1024<\/code>, a pokud minimum na dvou j\u00e1drech, tak uvedeme <code>--cpushares=2048<\/code>. Cpu shares nijak nezasahuj\u00ed do rozd\u011blen\u00ed procesorov\u00e9ho \u010dasu, dokud ho je dostatek. Tak\u017ee pokud prod v dan\u00e9m okam\u017eiku nevyu\u017e\u00edv\u00e1 v\u0161echna \u010dty\u0159i j\u00e1dra \u2013 nic neomezuje batch-\u00falohy, a ty mohou vyu\u017e\u00edt dal\u0161\u00ed procesorov\u00fd \u010das. Naopak v situaci nedostatku procesoru, pokud prod spot\u0159eboval v\u0161echna \u010dty\u0159i j\u00e1dra a narazil na kv\u00f3tu \u2013 zb\u00fdvaj\u00edc\u00ed procesorov\u00fd \u010das bude rozd\u011blen proporcion\u00e1ln\u011b podle cpushares, t.j. v situaci t\u0159\u00ed voln\u00fdch jader jedno dostane \u00faloha se 1024 cpushares a zb\u00fdvaj\u00edc\u00ed dv\u011b \u00falohy se 2048 cpushares.<\/p>\n<p><\/p>\n<p>Ale pou\u017eit\u00ed kv\u00f3ty a shares nen\u00ed dostate\u010dn\u00e9. Mus\u00edme zajistit, aby \u00faloha s kr\u00e1tkou latenc\u00ed m\u011bla prioritu p\u0159ed batch-\u00falohou p\u0159i rozd\u011blov\u00e1n\u00ed procesorov\u00e9ho \u010dasu. Bez takov\u00e9 prioritizace by batch-\u00faloha zabrala ve\u0161ker\u00fd procesorov\u00fd \u010das ve chv\u00edli, kdy ho pot\u0159ebuje prod. V Docker run nejsou \u017e\u00e1dn\u00e9 mo\u017enosti prioritizace kontejner\u016f, ale pomoci p\u0159ich\u00e1zej\u00ed politiky pl\u00e1nova\u010de centr\u00e1ln\u00edho procesoru v Linuxu. O nich si m\u016f\u017eete podrobn\u011bji p\u0159e\u010d\u00edst <noindex><a rel=\"nofollow\" href=\"http:\/\/man7.org\/linux\/man-pages\/man2\/sched_setscheduler.2.html\">tady<\/a><\/noindex>, a v r\u00e1mci tohoto \u010dl\u00e1nku si je projdeme stru\u010dn\u011b:<\/p>\n<p><\/p>\n<ul>\n<li><strong>SCHED_OTHER<\/strong><br \/>\nStandardn\u011b dost\u00e1vaj\u00ed v\u0161echny b\u011b\u017en\u00e9 u\u017eivatelsk\u00e9 procesy na Linuxov\u00e9m stroji.<\/li>\n<li><strong>SCHED_BATCH<\/strong><br \/>\nUr\u010deno pro n\u00e1ro\u010dn\u00e9 procesy. P\u0159i um\u00edst\u011bn\u00ed \u00falohy do procesoru se zav\u00e1d\u00ed tzv. trest za aktivaci: takov\u00e1 \u00faloha s men\u0161\u00ed pravd\u011bpodobnost\u00ed z\u00edsk\u00e1 procesorov\u00e9 zdroje, pokud je pr\u00e1v\u011b v tu chv\u00edli procesorem obsazena \u00faloha se SCHED_OTHER.<\/li>\n<li><strong>SCHED_IDLE<\/strong><br \/>\nPozad\u00ed proces s velmi n\u00edzkou prioritou, dokonce ni\u017e\u0161\u00ed ne\u017e nice \u201319. Pou\u017e\u00edv\u00e1me na\u0161i knihovnu s otev\u0159en\u00fdm zdrojov\u00fdm k\u00f3dem. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/odnoklassniki\/one-nio\">one-nio<\/a><\/noindex>, aby byla stanovena nezbytn\u00e1 politika p\u0159i spu\u0161t\u011bn\u00ed kontejneru vol\u00e1n\u00edm<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"java\">one.nio.os.Proc.sched_setscheduler( pid, Proc.SCHED_IDLE )<\/code><\/pre>\n<p><\/p>\n<p>Ale i pokud neprogramujete v Jav\u011b, to sam\u00e9 lze ud\u011blat pomoc\u00ed p\u0159\u00edkazu chrt:<\/p>\n<p><\/p>\n<pre><code class=\"bash\">chrt -i 0 $pid<\/code><\/pre>\n<p><\/p>\n<p>Shrneme v\u0161echny na\u0161e \u00farovn\u011b izolace do jedn\u00e9 tabulky pro p\u0159ehlednost:<\/p>\n<p><\/p>\n<p>T\u0159\u00edda izolace<br \/>\nP\u0159\u00edklad alloc<br \/>\nMo\u017enosti Docker run<br \/>\nsched_setscheduler chrt*<\/p>\n<p>Prod<br \/>\ncpu = 4<br \/>\n<code>--cpuquota=400000<\/code> <code>--cpuperiod=100000<\/code><br \/>\nSCHED_OTHER<\/p>\n<p>Batch<br \/>\nCpu = [1, * )<br \/>\n<code>--cpushares=1024<\/code><br \/>\nSCHED_BATCH<\/p>\n<p>Idle<br \/>\nCpu= [2, *)<br \/>\n<code>--cpushares=2048<\/code><br \/>\nSCHED_IDLE<\/p>\n<p><\/p>\n<p>*Pokud prov\u00e1d\u00edte chrt uvnit\u0159 kontejneru, m\u016f\u017ee b\u00fdt pot\u0159eba capability sys_nice, proto\u017ee standardn\u011b Docker tuto capability p\u0159i spu\u0161t\u011bn\u00ed kontejneru odebere.<\/p>\n<p><\/p>\n<p>Ale \u00falohy spot\u0159ebov\u00e1vaj\u00ed nejen procesor, ale tak\u00e9 provoz, kter\u00fd m\u00e1 je\u0161t\u011b v\u011bt\u0161\u00ed vliv na zpo\u017ed\u011bn\u00ed s\u00ed\u0165ov\u00e9 \u00falohy ne\u017e nespr\u00e1vn\u00e9 rozd\u011blen\u00ed procesorov\u00fdch zdroj\u016f. Proto samoz\u0159ejm\u011b chceme z\u00edskat identick\u00fd obraz i pro provoz. Tedy, kdy\u017e prod-\u00faloha pos\u00edl\u00e1 n\u011bjak\u00e9 pakety do s\u00edt\u011b, kvotujeme maxim\u00e1ln\u00ed rychlost (vzorec <em>alloc: lan=[*,500mbps)<\/em> ), s jakou prod m\u016f\u017ee toto d\u011blat. A pro batch garantujeme pouze minim\u00e1ln\u00ed propustnost, ale neomezen\u00edm maxim\u00e1ln\u00ed (vzorec <em>alloc: lan=[10Mbps,*)<\/em> ) V tomto p\u0159\u00edpad\u011b by m\u011bl provoz prod m\u00edt prioritu p\u0159ed batch-\u00falohami.<br \/>\nZde Docker nem\u00e1 \u017e\u00e1dn\u00e9 primitiva, kter\u00e1 bychom mohli pou\u017e\u00edt. Ale pomoci n\u00e1m p\u0159ich\u00e1z\u00ed <noindex><a rel=\"nofollow\" href=\"http:\/\/lartc.org\/\">Linux Traffic Control<\/a><\/noindex>. Dok\u00e1zali jsme dos\u00e1hnout po\u017eadovan\u00e9ho v\u00fdsledku d\u00edky discipl\u00edn\u011b <noindex><a rel=\"nofollow\" href=\"http:\/\/linux-ip.net\/articles\/hfsc.en\/\">Hierarchical Fair Service Curve<\/a><\/noindex>. S jej\u00ed\u017e pomoc\u00ed vy\u010dle\u0148ujeme dv\u011b t\u0159\u00eddy provozu: vysoce prioritn\u00ed prod a n\u00edzko prioritn\u00ed batch\/idle. Nakonec konfigurace pro odchoz\u00ed provoz vypad\u00e1 takto:<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/gh\/5k\/kf\/gh5kkfwkhwv0ilmbdmdbo83dp6k.png\"><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/3f52a968118b6021bd0c920af17a00c0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p><\/p>\n<p>zde 1:0 \u2014 &#171;ko\u0159enov\u00fd qdisc&#187; discipl\u00edny hsfc; 1:1 \u2014 pod\u0159\u00edzen\u00e1 t\u0159\u00edda hsfc s celkov\u00fdm limitem propustnosti 8 Gbit\/s, pod kterou spadaj\u00ed pod\u0159\u00edzen\u00e9 t\u0159\u00eddy v\u0161ech kontejner\u016f; 1:2 \u2014 pod\u0159\u00edzen\u00e1 t\u0159\u00edda hsfc spole\u010dn\u00e1 pro v\u0161echny batch a idle \u00falohy s &#171;dynami\u010dt\u00fdm&#187; limitem, o kter\u00e9m bude n\u00ed\u017ee. Ostatn\u00ed pod\u0159\u00edzen\u00e9 t\u0159\u00eddy hsfc jsou vyhrazen\u00e9 t\u0159\u00eddy pro aktu\u00e1ln\u011b b\u011b\u017e\u00edc\u00ed prod-kontejnery s limity odpov\u00eddaj\u00edc\u00edmi jejich manifest\u016fm \u2014 450 a 400 Mbit\/s. Ka\u017ed\u00e9 t\u0159\u00edd\u011b hsfc je p\u0159i\u0159azena qdisc fronta fq nebo fq_codel, v z\u00e1vislosti na verzi j\u00e1dra linux, aby se p\u0159ede\u0161lo ztr\u00e1t\u00e1m paket\u016f p\u0159i spikech provozu. <\/p>\n<p><\/p>\n<p>Obvykle discipl\u00edny tc slou\u017e\u00ed pouze k prioritizaci odchoz\u00edho provozu. Ale chceme prioritizovat i p\u0159\u00edchoz\u00ed provoz \u2014 n\u011bjak\u00fd batch \u00fakol by mohl snadno zabrat celou p\u0159\u00edchoz\u00ed linku, nap\u0159\u00edklad z\u00edsk\u00e1n\u00edm velk\u00e9ho bal\u00edku vstupn\u00edch dat pro map&amp;reduce. K tomu pou\u017e\u00edv\u00e1me modul <noindex><a rel=\"nofollow\" href=\"https:\/\/serverfault.com\/questions\/350023\/tc-ingress-policing-and-ifb-mirroring\">ifb<\/a><\/noindex>, kter\u00fd vytv\u00e1\u0159\u00ed virtu\u00e1ln\u00ed rozhran\u00ed ifbX pro ka\u017ed\u00e9 s\u00ed\u0165ov\u00e9 rozhran\u00ed a p\u0159esm\u011brov\u00e1v\u00e1 p\u0159\u00edchoz\u00ed provoz z rozhran\u00ed do odchoz\u00edho na ifbX. Pro ifbX pak funguj\u00ed v\u0161echny stejn\u00e9 discipl\u00edny pro kontrolu odchoz\u00edho provozu, pro kter\u00e9 bude konfigurace hsfc velmi podobn\u00e1:<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/iz\/ao\/-k\/izao-kgthgu_ptgyq9cjb1il0wm.png\"><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/967da9c0637cc70ba195af781db18adf.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p><\/p>\n<p>B\u011bhem experiment\u016f jsme zjistili, \u017ee nejlep\u0161\u00ed v\u00fdsledky hsfc ukazuje, kdy\u017e je t\u0159\u00edda 1:2 nep\u0159ednostn\u00edho batch\/idle provozu omezena na stroj\u00edch-minionech na maxim\u00e1ln\u011b ur\u010ditou volnou \u0161\u00ed\u0159ku p\u00e1sma. Jinak nep\u0159ednostn\u00ed provoz p\u0159\u00edli\u0161 siln\u011b ovliv\u0148uje zpo\u017ed\u011bn\u00ed prod \u00fakol\u016f. Aktu\u00e1ln\u00ed velikost voln\u00e9 \u0161\u00ed\u0159ky p\u00e1sma ur\u010duje miniond ka\u017edou sekundu m\u011b\u0159en\u00edm pr\u016fm\u011brn\u00e9 spot\u0159eby provozu v\u0161emi prod \u00fakoly dan\u00e9ho miniona <img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/9d6f38110d3b1693afe222b01430b9bb.jpg\" style=\"display:block;margin: 0 auto;\" \/> a ode\u010d\u00edt\u00e1n\u00edm ji od \u0161\u00ed\u0159ky p\u00e1sma s\u00ed\u0165ov\u00e9ho rozhran\u00ed <img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/d5608ec4c58e44b8571ab0b1d0cc7701.jpg\" style=\"display:block;margin: 0 auto;\" \/> s malou rezervou, t. j.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/fb01347c2224bd5bbd82169861eeb7e4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u0160\u00ed\u0159ky p\u00e1sma jsou ur\u010dov\u00e1ny pro p\u0159\u00edchoz\u00ed a odchoz\u00ed provoz nez\u00e1visle. A podle nov\u00fdch hodnot miniond znovu konfiguruje limit nep\u0159ednostn\u00ed t\u0159\u00eddy 1:2.<\/p>\n<p><\/p>\n<p>T\u00edmto zp\u016fsobem jsme implementovali v\u0161echny t\u0159i t\u0159\u00eddy izolace: prod, batch a idle. Tyto t\u0159\u00eddy maj\u00ed siln\u00fd vliv na charakteristiky vykon\u00e1v\u00e1n\u00ed \u00fakol\u016f. Proto jsme se rozhodli um\u00edstit tento znak na vrchol hierarchie, abychom p\u0159i pohledu na n\u00e1zev hierarchick\u00e9 fronty okam\u017eit\u011b pochopili, s \u010d\u00edm m\u00e1me co do \u010din\u011bn\u00ed: <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/34dc89c461db711b19a0f4eccc50efce.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u0161ichni na\u0161i zn\u00e1m\u00ed <strong>web<\/strong> a <strong>music<\/strong> fronty jsou pak um\u00edst\u011bny v hierarchii pod prod. Nap\u0159\u00edklad pod batch um\u00edst\u00edme slu\u017ebu <strong>music catalog<\/strong>, kter\u00fd pravideln\u011b sestavuje katalog skladeb ze soubor\u016f mp3 nahran\u00fdch do \u201eOdnoklassniki\u201c. P\u0159\u00edkladem slu\u017eby pod idle m\u016f\u017ee b\u00fdt <strong>hudebn\u00ed transform\u00e1tor<\/strong>, normalizuj\u00edc\u00ed \u00farove\u0148 hlasitosti hudby.<\/p>\n<p><\/p>\n<p>Op\u011bt odstran\u011bn\u00edm nadbyte\u010dn\u00fdch \u0159\u00e1dk\u016f m\u016f\u017eeme zapsat n\u00e1zvy na\u0161ich slu\u017eeb v\u00edce plo\u0161e, p\u0159id\u00e1n\u00edm t\u0159\u00eddy izolace \u00fakolu na konec \u00fapln\u00e9ho n\u00e1zvu slu\u017eby: <strong>web.front.prod<\/strong>, <strong>catalog.music.batch<\/strong>, <strong>transformer.music.idle<\/strong>.<\/p>\n<p><\/p>\n<p>A nyn\u00ed, kdy\u017e se pod\u00edv\u00e1me na n\u00e1zev slu\u017eby, ch\u00e1peme nejen to, jakou funkci pln\u00ed, ale i jej\u00ed t\u0159\u00eddu izolace, co\u017e znamen\u00e1 jej\u00ed kriti\u010dnost atd.<\/p>\n<p><\/p>\n<p>V\u0161echno je skv\u011bl\u00e9, ale je tu jedna ho\u0159k\u00e1 pravda. \u00dapln\u011b izolovat \u00fakoly b\u011b\u017e\u00edc\u00ed na jednom stroji nen\u00ed mo\u017en\u00e9.<\/p>\n<p><\/p>\n<p>Co se n\u00e1m poda\u0159ilo dos\u00e1hnout: pokud batch intenzivn\u011b spot\u0159ebov\u00e1v\u00e1 <strong>pouze<\/strong> prost\u0159edky procesoru, vestav\u011bn\u00fd pl\u00e1nova\u010d CPU Linux se svou \u00falohou velmi dob\u0159e vyrovn\u00e1v\u00e1 a vliv na prod-\u00fakol prakticky nen\u00ed. Ale pokud tento batch-\u00fakol za\u010dne aktivn\u011b pracovat s pam\u011bt\u00ed, vz\u00e1jemn\u00fd vliv u\u017e se projev\u00ed. D\u011bje se to proto, \u017ee z prod-\u00fakolu \u201evymizely\u201c procesorov\u00e9 cache pam\u011bti \u2014 nakonec se v cache zvy\u0161uj\u00ed proh\u0159e\u0161ky a procesor zpracov\u00e1v\u00e1 prod-\u00fakol pomaleji. Takov\u00fd batch-\u00fakol m\u016f\u017ee zv\u00fd\u0161it latenci na\u0161eho typick\u00e9ho prod-kontejneru o 10 %.<\/p>\n<p><\/p>\n<p>Izolovat provoz je je\u0161t\u011b slo\u017eit\u011bj\u0161\u00ed, proto\u017ee modern\u00ed s\u00ed\u0165ov\u00e9 karty maj\u00ed intern\u00ed frontu paket\u016f. Pokud se paket z batch-\u00fakolu dostal dovnit\u0159 jako prvn\u00ed, bude tak\u00e9 p\u0159ed\u00e1n p\u0159es kabel jako prvn\u00ed, a s t\u00edm se ned\u00e1 nic d\u011blat.<\/p>\n<p><\/p>\n<p>Nav\u00edc se n\u00e1m zat\u00edm poda\u0159ilo vy\u0159e\u0161it pouze probl\u00e9m prioritizace TCP-provozu: pro UDP p\u0159\u00edstup s hsfc nefunguje. A i v p\u0159\u00edpad\u011b TCP-provozu, pokud batch-\u00fakol generuje mnoho provozu, to tak\u00e9 zvy\u0161uje latenci prod-\u00fakolu o p\u0159ibli\u017en\u011b 10 %.<\/p>\n<p><\/p>\n<h2 id=\"otkazoustoychivost\">Odolnost proti v\u00fdpadk\u016fm<\/h2>\n<p><\/p>\n<p>Jedn\u00edm z c\u00edl\u016f p\u0159i v\u00fdvoji one-cloud bylo zlep\u0161en\u00ed odolnosti Odnoklassniki. Proto bych se cht\u011bl podrobn\u011bji pod\u00edvat na mo\u017en\u00e9 sc\u00e9n\u00e1\u0159e selh\u00e1n\u00ed a hav\u00e1rie. Za\u010dn\u011bme jednoduch\u00fdm sc\u00e9n\u00e1\u0159em \u2014 selh\u00e1n\u00edm kontejneru. <\/p>\n<p><\/p>\n<p>Kontejner m\u016f\u017ee selhat n\u011bkolika zp\u016fsoby. M\u016f\u017ee to b\u00fdt experiment, chyba nebo probl\u00e9m v manifestu, kv\u016fli kter\u00e9mu produk\u010dn\u00ed \u00faloha za\u010dne spot\u0159ebov\u00e1vat v\u00edce zdroj\u016f, ne\u017e je uvedeno v manifestu. M\u011bli jsme p\u0159\u00edpad, kdy v\u00fdvoj\u00e1\u0159 implementoval slo\u017eit\u00fd algoritmus, n\u011bkolikr\u00e1t ho p\u0159etv\u00e1\u0159el, a\u017e se s\u00e1m zamotal, co\u017e vedlo k tomu, \u017ee \u00faloha se za\u010dala ne\u010dekan\u011b cyklit. A proto\u017ee produk\u010dn\u00ed \u00faloha m\u00e1 vy\u0161\u0161\u00ed prioritu ne\u017e v\u0161echny ostatn\u00ed na stejn\u00fdch minionech, za\u010dala spot\u0159ebov\u00e1vat v\u0161echny dostupn\u00e9 procesorov\u00e9 zdroje. V t\u00e9to situaci n\u00e1m pomohla izolace, p\u0159esn\u011bji \u0159e\u010deno kv\u00f3ta na procesorov\u00fd \u010das. Pokud je \u00faloze p\u0159id\u011blena kv\u00f3ta, nespot\u0159ebuje v\u00edce. Proto si d\u00e1vkov\u00e1 a dal\u0161\u00ed produk\u010dn\u00ed \u00falohy, kter\u00e9 pracovaly na stejn\u00e9 ma\u0161in\u011b, ni\u010deho nev\u0161imly. <\/p>\n<p><\/p>\n<p>Druhou mo\u017enou nep\u0159\u00edjemnost\u00ed je p\u00e1d kontejneru. A zde n\u00e1s zachra\u0148uje restartovac\u00ed politika, kterou v\u0161ichni znaj\u00ed, Docker s t\u00edm skv\u011ble pracuje. Prakticky v\u0161echny produk\u010dn\u00ed \u00falohy maj\u00ed politiku restartu always. N\u011bkdy pou\u017e\u00edv\u00e1me on_failure pro d\u00e1vkov\u00e9 \u00falohy nebo pro lad\u011bn\u00ed produk\u010dn\u00edch kontejner\u016f.<\/p>\n<p><\/p>\n<p>Co se d\u00e1 d\u011blat, kdy\u017e je cel\u00e1 ma\u0161ina nedostupn\u00e1?<\/p>\n<p><\/p>\n<p>Je jasn\u00e9, \u017ee je t\u0159eba spustit kontejner na jin\u00e9m stroji. Nejzaj\u00edmav\u011bj\u0161\u00ed je, co se stane s IP adresou (adresami) p\u0159id\u011blen\u00fdmi kontejneru. <\/p>\n<p><\/p>\n<p>M\u016f\u017eeme p\u0159id\u011blovat kontejner\u016fm stejn\u00e9 IP adresy jako maj\u00ed stroje-miniony, na kter\u00fdch jsou tyto kontejnery spu\u0161t\u011bny. Pak p\u0159i spu\u0161t\u011bn\u00ed kontejneru na jin\u00e9m stroji se jeho IP adresa zm\u011bn\u00ed a v\u0161ichni klienti mus\u00ed pochopit, \u017ee kontejner se p\u0159est\u011bhoval, nyn\u00ed je pot\u0159eba se obr\u00e1tit na jinou adresu, co\u017e vy\u017eaduje samostatnou slu\u017ebu pro objevov\u00e1n\u00ed slu\u017eeb. <\/p>\n<p><\/p>\n<p>Objevov\u00e1n\u00ed slu\u017eeb je praktick\u00e9. Na trhu je mnoho \u0159e\u0161en\u00ed r\u016fzn\u00fdch stup\u0148\u016f v\u00fdpadkovosti pro organizaci registru slu\u017eeb. \u010casto v t\u011bchto \u0159e\u0161en\u00edch je realizov\u00e1na logika vyva\u017eova\u010de z\u00e1t\u011b\u017ee, ukl\u00e1d\u00e1n\u00ed dal\u0161\u00edch konfigurac\u00ed ve form\u011b KV storage a podobn\u011b.<br \/>\nR\u00e1di bychom se v\u0161ak obe\u0161li bez nutnosti zav\u00e1d\u011bt samostatn\u00fd registr, proto\u017ee by to znamenalo zav\u00e9st kritick\u00fd syst\u00e9m, kter\u00fd pou\u017e\u00edvaj\u00ed v\u0161echny slu\u017eby v produkci. To znamen\u00e1 potenci\u00e1ln\u00ed bod selh\u00e1n\u00ed, a je t\u0159eba vyb\u00edrat nebo vyv\u00edjet velmi spolehliv\u00e9 \u0159e\u0161en\u00ed, co\u017e je zjevn\u011b velmi slo\u017eit\u00e9, \u010dasov\u011b n\u00e1ro\u010dn\u00e9 a drah\u00e9. <\/p>\n<p><\/p>\n<p>A dal\u0161\u00ed velkou nev\u00fdhodou je, \u017ee abychom na\u0161i starou infrastrukturu spojili s novou, museli bychom p\u0159epsat v\u0161echny \u00fakoly pro pou\u017eit\u00ed n\u011bjak\u00e9ho syst\u00e9mu Service Discovery. Pr\u00e1ce by bylo OPRAVDU hodn\u011b, a m\u00edsty by to bylo t\u00e9m\u011b\u0159 nemo\u017en\u00e9, zejm\u00e9na kdy\u017e se jedn\u00e1 o n\u00edzko\u00farov\u0148ov\u00e1 za\u0159\u00edzen\u00ed, kter\u00e1 pracuj\u00ed na \u00farovni j\u00e1dra OS nebo p\u0159\u00edmo s hardwarem. Implementace t\u00e9to funk\u010dnosti pomoc\u00ed osv\u011bd\u010den\u00fdch vzor\u016f \u0159e\u0161en\u00ed, jako nap\u0159\u00edklad <noindex><a rel=\"nofollow\" href=\"https:\/\/linkerd.io\">side-car<\/a><\/noindex> by znamenala m\u00edsty dodate\u010dn\u00e9 zat\u00ed\u017een\u00ed, jindy \u2014 zt\u00ed\u017een\u00ed provozu a dal\u0161\u00ed sc\u00e9n\u00e1\u0159e selh\u00e1n\u00ed. Cht\u011bli jsme se vyhnout komplikac\u00edm, a proto jsme se rozhodli ud\u011blat pou\u017eit\u00ed Service Discovery voliteln\u00fdm. <\/p>\n<p><\/p>\n<p>V one-cloud IP n\u00e1sleduje kontejner, tj. ka\u017ed\u00fd exempl\u00e1\u0159 \u00falohy m\u00e1 svou vlastn\u00ed IP adresu. Tato adresa je \u201estatick\u00e1\u201c: je p\u0159id\u011blena ka\u017ed\u00e9mu exempl\u00e1\u0159i v okam\u017eiku prvn\u00ed odesl\u00e1n\u00ed slu\u017eeb do cloudu. Pokud m\u011bl b\u011bhem sv\u00e9 existence slu\u017eba r\u016fzn\u00fd po\u010det exempl\u00e1\u0159\u016f \u2014 nakonec bude m\u00edt tolik IP adres, kolik bylo maxim\u00e1ln\u011b exempl\u00e1\u0159\u016f.<\/p>\n<p><\/p>\n<p>Tyto adresy se n\u00e1sledn\u011b nem\u011bn\u00ed: jsou p\u0159id\u011bleny jednor\u00e1zov\u011b a z\u016fst\u00e1vaj\u00ed platn\u00e9 po celou dobu existence slu\u017eby v produkci. IP adresy n\u00e1sleduj\u00ed za kontejnery p\u0159es s\u00ed\u0165. Pokud je kontejner p\u0159esunut na jin\u00fd minion, pak adresa p\u0159evezme s n\u00edm. <\/p>\n<p><\/p>\n<p>Tak\u017ee mapov\u00e1n\u00ed n\u00e1zvu slu\u017eby na seznam jej\u00edch IP adres se m\u011bn\u00ed velmi z\u0159\u00eddka. Pokud se je\u0161t\u011b jednou pod\u00edv\u00e1me na n\u00e1zvy exempl\u00e1\u0159\u016f slu\u017eby, kter\u00e9 jsme zm\u00ednili na za\u010d\u00e1tku \u010dl\u00e1nku (<strong>1.ok-web.group1.web.front.prod, 2.ok-web.group1.web.front.prod, \u2026<\/strong>), si v\u0161imneme, \u017ee p\u0159ipom\u00ednaj\u00ed FQDN pou\u017e\u00edvan\u00e9 v DNS. A skute\u010dn\u011b, pro zobrazen\u00ed n\u00e1zv\u016f instanc\u00ed slu\u017eeb v jejich IP adres\u00e1ch pou\u017e\u00edv\u00e1me protokol DNS. Tento DNS nav\u00edc vrac\u00ed v\u0161echny rezervovan\u00e9 IP adresy v\u0161ech kontejner\u016f \u2014 jak b\u011b\u017e\u00edc\u00edch, tak zastaven\u00fdch (p\u0159edpokl\u00e1dejme, \u017ee se pou\u017e\u00edvaj\u00ed t\u0159i repliky, a m\u00e1me p\u011bt adres rezervovan\u00fdch \u2014 v\u0161ech p\u011bt bude vr\u00e1ceno). Klienti, kte\u0159\u00ed z\u00edskaj\u00ed tyto informace, se pokus\u00ed nav\u00e1zat spojen\u00ed se v\u0161emi p\u011bti replikami \u2014 a t\u00edm zjist\u00ed, kter\u00e9 funguj\u00ed. Tento zp\u016fsob ur\u010dov\u00e1n\u00ed dostupnosti je v\u00fdrazn\u011b spolehliv\u011bj\u0161\u00ed, nezapojuje do n\u011bj ani DNS, ani Service Discovery, co\u017e znamen\u00e1, \u017ee neexistuj\u00ed slo\u017eit\u00e9 probl\u00e9my s udr\u017eov\u00e1n\u00edm aktu\u00e1lnosti informac\u00ed a odolnost\u00ed t\u011bchto syst\u00e9m\u016f. Nav\u00edc u kritick\u00fdch slu\u017eeb, na kter\u00fdch z\u00e1vis\u00ed fungov\u00e1n\u00ed cel\u00e9ho port\u00e1lu, m\u016f\u017eeme v\u016fbec nepou\u017e\u00edvat DNS, ale prost\u011b zadat IP adresy do konfigurace.<\/p>\n<p><\/p>\n<p>Realizace takov\u00e9ho p\u0159enosu IP za kontejnery m\u016f\u017ee b\u00fdt netrivi\u00e1ln\u00ed \u2014 a zastav\u00edme se u toho, jak to funguje, na n\u00e1sleduj\u00edc\u00edm p\u0159\u00edkladu:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/faf99c2609a57daa1bdc193cb0f4cb31.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0159edstavme si, \u017ee master one-cloud d\u00e1v\u00e1 pokyn minionu M1 spustit <strong>1.ok-web.group1.web.front.prod<\/strong> s adresou 1.1.1.1. Na minionu b\u011b\u017e\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Bird_Internet_routing_daemon\">BIRD<\/a><\/noindex>, kter\u00fd oznamuje tuto adresu speci\u00e1ln\u00edm server\u016fm <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Route_reflector\">route reflector<\/a><\/noindex>. Ti maj\u00ed BGP relaci se s\u00ed\u0165ov\u00fdm za\u0159\u00edzen\u00edm, do kter\u00e9 se retransmituje cesta adresy 1.1.1.1 na M1. M1 pak sm\u011bruje pakety do kontejneru pomoc\u00ed prost\u0159edk\u016f Linuxu. Server\u016f route reflector jsou t\u0159i, proto\u017ee to je velmi kritick\u00e1 \u010d\u00e1st infrastruktury one-cloud \u2014 bez nich by s\u00ed\u0165 v one-cloud nefungovala. Um\u00eds\u0165ujeme je do r\u016fzn\u00fdch sk\u0159\u00edn\u00ed, pokud mo\u017eno um\u00edst\u011bn\u00fdch v r\u016fzn\u00fdch s\u00e1lech datacentra, abychom sn\u00ed\u017eili pravd\u011bpodobnost sou\u010dasn\u00e9ho selh\u00e1n\u00ed v\u0161ech t\u0159\u00ed.<\/p>\n<p><\/p>\n<p>P\u0159edpokl\u00e1dejme nyn\u00ed, \u017ee spojen\u00ed mezi masterem one-cloud a minionem M1 se p\u0159eru\u0161ilo. Master one-cloud nyn\u00ed bude jednat na z\u00e1klad\u011b p\u0159edpokladu, \u017ee M1 zcela selhal. To znamen\u00e1, \u017ee d\u00e1 pokyn minionu M2 spustit <strong>web.group1.web.front.prod<\/strong> se stejnou IP adresou 1.1.1.1. Nyn\u00ed m\u00e1me dv\u011b konfliktn\u00ed trasy v s\u00edti pro 1.1.1.1: na M1 a na M2. Pro \u0159e\u0161en\u00ed t\u011bchto konflikt\u016f pou\u017e\u00edv\u00e1me Multi Exit Discriminator, kter\u00fd je uveden v BGP ozn\u00e1men\u00ed. Toto \u010d\u00edslo ukazuje v\u00e1hu inzerovan\u00e9 trasy. Z konfliktuj\u00edc\u00edch se vybere trasa s men\u0161\u00ed hodnotou MED. Master one-cloud podporuje MED jako integr\u00e1ln\u00ed \u010d\u00e1st IP adres kontejner\u016f. Poprv\u00e9 je adresa vyps\u00e1na s dostate\u010dn\u011b vysok\u00fdm MED = 1 000 000. V p\u0159\u00edpad\u011b nouzov\u00e9ho p\u0159esunu kontejneru master sni\u017euje MED a M2 obdr\u017e\u00ed p\u0159\u00edkaz inzerovat adresu 1.1.1.1 s MED = 999 999. Instance b\u011b\u017e\u00edc\u00ed na M1 z\u016fstane bez spojen\u00ed a jej\u00ed dal\u0161\u00ed osud n\u00e1s p\u0159\u00edli\u0161 nezaj\u00edm\u00e1 do doby obnoven\u00ed spojen\u00ed s masterem, kdy bude zastavena jako star\u00fd dvojn\u00edk.<\/p>\n<p><\/p>\n<h2 id=\"avarii\">Hav\u00e1rie<\/h2>\n<p><\/p>\n<p>V\u0161echny syst\u00e9my spr\u00e1vy datov\u00fdch center v\u017edy adekv\u00e1tn\u011b zvl\u00e1daj\u00ed drobn\u00e9 v\u00fdpadky. Selh\u00e1n\u00ed kontejneru je normou prakticky v\u0161ude.<\/p>\n<p><\/p>\n<p>Pod\u00edvejme se, jak zvl\u00e1d\u00e1me hav\u00e1rii, nap\u0159\u00edklad v\u00fdpadek energie v jedn\u00e9 nebo v\u00edce m\u00edstnostech datov\u00e9ho centra.<\/p>\n<p><\/p>\n<p>Co znamen\u00e1 hav\u00e1rie pro syst\u00e9m spr\u00e1vy datov\u00e9ho centra? V prvn\u00ed \u0159ad\u011b to je masivn\u00ed jednor\u00e1zov\u00e9 selh\u00e1n\u00ed mnoha stroj\u016f a syst\u00e9m spr\u00e1vy mus\u00ed sou\u010dasn\u011b migrovat velk\u00e9 mno\u017estv\u00ed kontejner\u016f. Ale pokud je hav\u00e1rie velmi rozs\u00e1hl\u00e1, m\u016f\u017ee se st\u00e1t, \u017ee v\u0161echny \u00fakoly nebudou moci b\u00fdt p\u0159esunuty na jin\u00e9 miniony, proto\u017ee kapacita datov\u00e9ho centra klesne pod 100 % zat\u00ed\u017een\u00ed. <\/p>\n<p><\/p>\n<p>Hav\u00e1rie \u010dasto prov\u00e1z\u00ed selh\u00e1n\u00ed i \u0159\u00edd\u00edc\u00ed vrstvy. To m\u016f\u017ee nastat kv\u016fli selh\u00e1n\u00ed jej\u00edch za\u0159\u00edzen\u00ed, ale \u010dast\u011bji kv\u016fli tomu, \u017ee hav\u00e1rie nejsou testov\u00e1ny, a \u0159\u00edd\u00edc\u00ed vrstva sama selh\u00e1v\u00e1 z d\u016fvodu zv\u00fd\u0161en\u00e9 z\u00e1t\u011b\u017ee. <\/p>\n<p><\/p>\n<p>Co s t\u00edm v\u0161\u00edm lze ud\u011blat?<\/p>\n<p><\/p>\n<p>Masivn\u00ed migrace znamenaj\u00ed, \u017ee v infrastruktu\u0159e vznik\u00e1 velk\u00e9 mno\u017estv\u00ed akc\u00ed, migrac\u00ed a um\u00edst\u011bn\u00ed. Ka\u017ed\u00e1 z migrac\u00ed m\u016f\u017ee zabrat n\u011bjak\u00fd \u010das, kter\u00fd je pot\u0159eba k dod\u00e1n\u00ed a rozbalen\u00ed obraz\u016f kontejner\u016f na minionech, spu\u0161t\u011bn\u00ed a inicializaci kontejner\u016f atd. Proto je \u017e\u00e1douc\u00ed, aby d\u016fle\u017eit\u011bj\u0161\u00ed \u00fakoly byly spu\u0161t\u011bny p\u0159ed m\u00e9n\u011b d\u016fle\u017eit\u00fdmi.<\/p>\n<p><\/p>\n<p>Znovu se pod\u00edvejme na na\u0161i zn\u00e1mou hierarchii slu\u017eeb a pokusme se rozhodnout, kter\u00e9 \u00fakoly chceme spustit jako prvn\u00ed.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/2e827271adb8d3ff3d385e53553aaacf.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Samoz\u0159ejm\u011b, to jsou ty procesy, kter\u00e9 se p\u0159\u00edmo pod\u00edlej\u00ed na zpracov\u00e1n\u00ed po\u017eadavk\u016f u\u017eivatel\u016f, tj. prod. Uv\u00e1d\u00edme to pomoc\u00ed <strong>priorita um\u00edst\u011bn\u00ed<\/strong> \u2014 \u010d\u00edslo, kter\u00e9 m\u016f\u017ee b\u00fdt p\u0159i\u0159azeno front\u011b. Pokud m\u00e1 n\u011bjak\u00e1 fronta vy\u0161\u0161\u00ed prioritu, jej\u00ed slu\u017eby se umis\u0165uj\u00ed jako prvn\u00ed.<\/p>\n<p><\/p>\n<p>Na prod p\u0159i\u0159azujeme vy\u0161\u0161\u00ed priority, 0; na batch \u2014 o n\u011bco ni\u017e\u0161\u00ed, 100; na idle \u2014 je\u0161t\u011b ni\u017e\u0161\u00ed, 200. Priority se aplikuj\u00ed hierarchicky. V\u0161echny \u00fakoly ni\u017e\u0161\u00ed v hierarchii budou m\u00edt odpov\u00eddaj\u00edc\u00ed prioritu. Pokud chceme, aby se v r\u00e1mci prod cache spou\u0161t\u011bly p\u0159ed frontami, nastav\u00edme priority pro cache = 0 a pro front pod\u0159\u00edzen\u00e9 = 1. Pokud nap\u0159\u00edklad chceme, aby se z front spou\u0161t\u011bl jako prvn\u00ed hlavn\u00ed port\u00e1l a a\u017e pot\u00e9 music front, m\u016f\u017eeme posledn\u00edmu p\u0159i\u0159adit ni\u017e\u0161\u00ed prioritu \u2014 10.<\/p>\n<p><\/p>\n<p>Dal\u0161\u00edm probl\u00e9mem je nedostatek zdroj\u016f. Tak\u017ee n\u00e1m selhalo velk\u00e9 mno\u017estv\u00ed za\u0159\u00edzen\u00ed, cel\u00e9 s\u00e1ly datov\u00e9ho centra, a my jsme spustili tolik slu\u017eeb, \u017ee nyn\u00ed nen\u00ed dostatek zdroj\u016f pro v\u0161echny. Mus\u00edme rozhodnout, kter\u00fdm \u00fakol\u016fm ob\u011btovat, aby fungovaly hlavn\u00ed kritick\u00e9 slu\u017eby. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/1b1ad3da16df6677dd0ba4b038cdd7d6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na rozd\u00edl od priority um\u00edst\u011bn\u00ed nem\u016f\u017eeme obecn\u011b ob\u011btovat v\u0161echny batch-\u00fakoly, n\u011bkter\u00e9 z nich jsou d\u016fle\u017eit\u00e9 pro fungov\u00e1n\u00ed port\u00e1lu. Proto jsme vy\u010dlenili zvl\u00e1\u0161\u0165 <strong>prioritu vykazov\u00e1n\u00ed<\/strong> \u00faloh. P\u0159i um\u00edst\u011bn\u00ed \u00fakol s vy\u0161\u0161\u00ed prioritou m\u016f\u017ee vyk\u00e1zat, t.j. zastavit \u00fakol s ni\u017e\u0161\u00ed prioritou, pokud ji\u017e nejsou k dispozici \u017e\u00e1dn\u00ed minioni. P\u0159itom \u00fakol s n\u00edzkou prioritou pravd\u011bpodobn\u011b z\u016fstane neumi\u0161t\u011bn\u00fd, tj. pro n\u011bj ji\u017e nebude vhodn\u00fd minion s dostate\u010dn\u00fdm po\u010dtem voln\u00fdch zdroj\u016f.<\/p>\n<p><\/p>\n<p>V na\u0161\u00ed hierarchii je velmi snadn\u00e9 nastavit takovou prioritu vykazov\u00e1n\u00ed, aby prod- a batch-\u00fakoly vykazovaly nebo zastavovaly idle-\u00fakoly, ale nezastavovaly navz\u00e1jem, p\u0159i\u010dem\u017e pro idle nastav\u00edme prioritu rovnaj\u00edc\u00ed se 200. Stejn\u011b jako v p\u0159\u00edpad\u011b priority um\u00edst\u011bn\u00ed m\u016f\u017eeme vyu\u017e\u00edt na\u0161i hierarchii k popisu slo\u017eit\u011bj\u0161\u00edch pravidel. Nap\u0159\u00edklad m\u016f\u017eeme uv\u00e9st, \u017ee ob\u011btujeme funkci hudby, pokud n\u00e1m nebude sta\u010dit zdroj\u016f pro hlavn\u00ed webov\u00fd port\u00e1l, a nastav\u00edme odpov\u00eddaj\u00edc\u00ed uzly s ni\u017e\u0161\u00ed prioritou: 10.<\/p>\n<p><\/p>\n<h2 id=\"avarii-dc-celikom\">Hav\u00e1rie DDC \u00fapln\u011b<\/h2>\n<p><\/p>\n<p>Pro\u010d m\u016f\u017ee selhat cel\u00e9 datov\u00e9 centrum? P\u0159\u00edrodn\u00ed katastrofa. Byl dobr\u00fd p\u0159\u00edsp\u011bvek, jak <noindex><a rel=\"nofollow\" href=\"https:\/\/habrahabr.ru\/company\/dataline\/blog\/333578\/\">hurik\u00e1n ovlivnil fungov\u00e1n\u00ed datov\u00e9ho centra<\/a><\/noindex>. Elementem lze pova\u017eovat bezdomovce, kte\u0159\u00ed kdysi zap\u00e1lili optiku v kanalizaci a datov\u00e9 centrum \u00fapln\u011b ztratilo spojen\u00ed s ostatn\u00edmi lokalitami. P\u0159\u00ed\u010dinou v\u00fdpadku m\u016f\u017ee b\u00fdt i lidsk\u00fd faktor: oper\u00e1tor vyd\u00e1 takov\u00fd p\u0159\u00edkaz, \u017ee cel\u00e9 datov\u00e9 centrum spadne. To se m\u016f\u017ee st\u00e1t kv\u016fli velk\u00e9 chyb\u011b. Obecn\u011b plat\u00ed, \u017ee datov\u00e1 centra padaj\u00ed \u2014 to nen\u00ed nic neobvykl\u00e9ho. U n\u00e1s k tomu doch\u00e1z\u00ed jednou za n\u011bkolik m\u011bs\u00edc\u016f. <\/p>\n<p><\/p>\n<p>A tady je to, co d\u011bl\u00e1me, aby nikdo #ok\u017eiv\u00ed nepostoval na Twitter.<\/p>\n<p><\/p>\n<p>Prvn\u00ed strategie \u2014 izolace. Ka\u017ed\u00fd instance one-cloud je izolovan\u00e1 a m\u016f\u017ee spravovat stroje pouze jedn\u00e9 datov\u00e9 centrum. To znamen\u00e1, \u017ee ztr\u00e1ta cloudu kv\u016fli chyb\u00e1m nebo chybn\u00e9mu p\u0159\u00edkazu oper\u00e1tora \u2014 je to ztr\u00e1ta pouze jednoho datov\u00e9ho centra. Na to jsme p\u0159ipraveni: m\u00e1me politiku z\u00e1lohov\u00e1n\u00ed, kdy jsou repliky aplikace a dat um\u00edst\u011bny ve v\u0161ech datov\u00fdch centrech. Pou\u017e\u00edv\u00e1me odoln\u00e9 datab\u00e1ze a pravideln\u011b testujeme v\u00fdpadky.<br \/>\nProto\u017ee dnes m\u00e1me \u010dty\u0159i datov\u00e1 centra, m\u00e1me tak\u00e9 \u010dty\u0159i \u00fapln\u011b izolovan\u00e9 instance one-cloud.<\/p>\n<p><\/p>\n<p>Takov\u00fd p\u0159\u00edstup nejen chr\u00e1n\u00ed p\u0159ed fyzick\u00fdm selh\u00e1n\u00edm, ale m\u016f\u017ee tak\u00e9 chr\u00e1nit p\u0159ed chybami oper\u00e1tora.<\/p>\n<p><\/p>\n<p>A co dal\u0161\u00edho lze ud\u011blat s lidsk\u00fdm faktorem? Kdy\u017e oper\u00e1tor vyd\u00e1v\u00e1 cloudu n\u011bjak\u00fd podivn\u00fd nebo potenci\u00e1ln\u011b nebezpe\u010dn\u00fd p\u0159\u00edkaz, m\u016f\u017ee b\u00fdt n\u00e1hle po\u017e\u00e1d\u00e1n, aby vy\u0159e\u0161il mal\u00fd \u00fakol, aby ov\u011b\u0159il, jak dob\u0159e o tom p\u0159em\u00fd\u0161lel. Nap\u0159\u00edklad, pokud jde o hromadn\u00e9 zastaven\u00ed mnoha replik nebo prost\u011b podivn\u00fd p\u0159\u00edkaz \u2014 sn\u00ed\u017een\u00ed po\u010dtu replik nebo zm\u011bna n\u00e1zvu obrazu, a ne pouze \u010d\u00edsla verze v nov\u00e9m manifestu.<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/vh\/vw\/0d\/vhvw0dzobo9sd4x7zih_cnfnlu8.png\"><img decoding=\"async\" alt=\"One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra v Odnoklassnik\u00e1ch\" src=\"\/wp-content\/uploads\/2020\/06\/f7ee44a77e30612a3cd99a6f7aee3820.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p><\/p>\n<h2 id=\"itogi\">Z\u00e1v\u011bry<\/h2>\n<p><\/p>\n<p>Vlastnosti one-cloud: <\/p>\n<p><\/p>\n<ul>\n<li><strong>Hierarchick\u00e1 a p\u0159ehledn\u00e1 sch\u00e9ma pojmenov\u00e1n\u00ed slu\u017eeb a kontejner\u016f<\/strong>, kter\u00e1 umo\u017e\u0148uje velmi rychle zjistit, o jak\u00fd \u00fakol jde, k \u010demu se vztahuje a jak funguje a kdo za n\u011bj odpov\u00edd\u00e1. <\/li>\n<li>Pou\u017e\u00edv\u00e1me svou <strong>techniku kombinov\u00e1n\u00ed prod- a batch-<\/strong>\u00fakol\u016f na minionech, abychom zv\u00fd\u0161ili efektivitu sd\u00edlen\u00ed stroj\u016f. M\u00edsto cpuset pou\u017e\u00edv\u00e1me CPU kv\u00f3ty, pod\u00edly, politiky pl\u00e1nova\u010de CPU a Linux QoS.<\/li>\n<li>\u00dapln\u011b izolovat kontejnery b\u011b\u017e\u00edc\u00ed na jednom stroji se n\u00e1m nepoda\u0159ilo, ale jejich vz\u00e1jemn\u00fd vliv z\u016fst\u00e1v\u00e1 v rozmez\u00ed do 20%.<\/li>\n<li>Organizace slu\u017eeb do hierarchie pom\u00e1h\u00e1 p\u0159i automatick\u00e9m odstra\u0148ov\u00e1n\u00ed hav\u00e1ri\u00ed pomoc\u00ed <strong>priorit um\u00edst\u011bn\u00ed a vyt\u011bsn\u011bn\u00ed<\/strong>.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"chavo\">\u010cASTO KLADEN\u00c9 OT\u00c1ZKY<\/h2>\n<p><\/p>\n<p>Pro\u010d jsme nezvolili hotov\u00e9 \u0159e\u0161en\u00ed.<\/p>\n<p><\/p>\n<ul>\n<li>R\u016fzn\u00e9 t\u0159\u00eddy izolace \u00faloh vy\u017eaduj\u00ed odli\u0161nou logiku p\u0159i umis\u0165ov\u00e1n\u00ed na minionech. Zat\u00edmco \u00falohy v produkci lze umis\u0165ovat jednoduch\u00fdm rezervov\u00e1n\u00edm zdroj\u016f, pro d\u00e1vkov\u00e9 a ne\u010dinn\u00e9 \u00falohy je nutn\u00e9 umis\u0165ovat je s ohledem na skute\u010dn\u00e9 vyu\u017eit\u00ed zdroj\u016f na stroj\u00edch-minionech. <\/li>\n<li>Nutnost sledovat spot\u0159ebov\u00e1van\u00e9 zdroje \u00faloh, jako jsou: \n<ul>\n<li>\u0161\u00ed\u0159ka p\u00e1sma s\u00edt\u011b;<\/li>\n<li>typy a 'spindly' disk\u016f.<\/li>\n<\/ul>\n<\/li>\n<li>Nutnost uv\u00e1d\u011bt priority slu\u017eeb p\u0159i odstra\u0148ov\u00e1n\u00ed hav\u00e1ri\u00ed, pr\u00e1va a kv\u00f3ty t\u00fdm\u016f na zdroje, co\u017e se \u0159e\u0161\u00ed pomoc\u00ed hierarchick\u00fdch front v one-cloud.<\/li>\n<li>Nutnost m\u00edt lidsk\u00e1 n\u00e1zvoslov\u00ed pro kontejnery, aby se zkr\u00e1tila doba reakc\u00ed na hav\u00e1rie a incidenty.<\/li>\n<li>Nemo\u017enost sou\u010dasn\u00e9ho plo\u0161n\u00e9ho nasazen\u00ed Service Discovery; nutnost dlouho koexistovat s \u00falohami um\u00edst\u011bn\u00fdmi na fyzick\u00fdch hostech \u2014 co\u017e se \u0159e\u0161\u00ed 'statick\u00fdmi' IP adresami navazuj\u00edc\u00edmi na kontejnery, a t\u00edm i nutnost unik\u00e1tn\u00ed integrace s rozs\u00e1hlou s\u00ed\u0165ovou infrastrukturou.<\/li>\n<\/ul>\n<p><\/p>\n<p>V\u0161echny tyto funkce by si vy\u017e\u00e1daly v\u00fdznamn\u00e9 \u00fapravy st\u00e1vaj\u00edc\u00edch \u0159e\u0161en\u00ed, a po zhodnocen\u00ed mno\u017estv\u00ed pr\u00e1ce jsme si uv\u011bdomili, \u017ee budeme schopni vyvinout na\u0161e \u0159e\u0161en\u00ed s p\u0159ibli\u017en\u011b stejn\u00fdm pracovn\u00edm \u00fasil\u00edm. Ale na\u0161e \u0159e\u0161en\u00ed bude podstatn\u011b jednodu\u0161\u0161\u00ed na provoz a rozvoj \u2014 neobsahuje \u017e\u00e1dn\u00e9 nadbyte\u010dn\u00e9 abstrakce podporuj\u00edc\u00ed pro n\u00e1s nepot\u0159ebnou funkcionalitu. <\/p>\n<p><\/p>\n<p>D\u011bkujeme v\u0161em, kte\u0159\u00ed p\u0159e\u010detli posledn\u00ed \u0159\u00e1dky, za trp\u011blivost a pozornost!<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/346868\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0410\u043b\u043e\u0445\u0430, \u043f\u0438\u043f\u043b! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041e\u043b\u0435\u0433 \u0410\u043d\u0430\u0441\u0442\u0430\u0441\u044c\u0435\u0432, \u044f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 \u041f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b. \u0410 \u043a\u0440\u043e\u043c\u0435 \u043c\u0435\u043d\u044f, \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u0443\u0447\u0430 \u0436\u0435\u043b\u0435\u0437\u0430. \u0423 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u0447\u0435\u0442\u044b\u0440\u0435 \u0426\u041e\u0414\u0430, \u0432 \u043d\u0438\u0445 \u043e\u043a\u043e\u043b\u043e 500 \u0441\u0442\u043e\u0435\u043a \u0431\u043e\u043b\u0435\u0435 \u0447\u0435\u043c \u0441 8 \u0442\u044b\u0441\u044f\u0447\u0430\u043c\u0438 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432. \u0412 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u044b\u0439 \u043c\u043e\u043c\u0435\u043d\u0442 \u043c\u044b \u043f\u043e\u043d\u044f\u043b\u0438, \u0447\u0442\u043e \u0432\u043d\u0435\u0434\u0440\u0435\u043d\u0438\u0435 \u043d\u043e\u0432\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u043f\u043e\u0437\u0432\u043e\u043b\u0438\u0442 \u043d\u0430\u043c \u0431\u043e\u043b\u0435\u0435 \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e \u0437\u0430\u0433\u0440\u0443\u0437\u0438\u0442\u044c \u0442\u0435\u0445\u043d\u0438\u043a\u0443, \u043e\u0431\u043b\u0435\u0433\u0447\u0438\u0442\u044c \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":84115,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-84114","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0410\u043b\u043e\u0445\u0430, \u043f\u0438\u043f\u043b!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47One-cloud \u2014 \u041e\u0421 \u0443\u0440\u043e\u0432\u043d\u044f \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u0430 \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0410\u043b\u043e\u0445\u0430, \u043f\u0438\u043f\u043b!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-06-05T05:42:55+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-06-05T05:42:55+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47One-cloud \u2014 opera\u010dn\u00ed syst\u00e9m na \u00farovni datov\u00e9ho centra na Odnoklassniki | ProHoster","description":"Aloha, lidi!","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47One-cloud \u2014 \u041e\u0421 \u0443\u0440\u043e\u0432\u043d\u044f \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u0430 \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 | ProHoster","og:description":"\u0410\u043b\u043e\u0445\u0430, \u043f\u0438\u043f\u043b!","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/one-cloud-os-urovnya-data-czentra-v-odnoklassnikah","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-06-05T05:42:55+00:00","article:modified_time":"2020-06-05T05:42:55+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"84114","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:04:42","updated":"2022-10-02 14:53:14","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/84114","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=84114"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/84114\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/84115"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=84114"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=84114"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=84114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}