{"id":34841,"date":"2019-10-31T22:00:43","date_gmt":"2019-10-31T19:00:43","guid":{"rendered":"https:\/\/prohoster.info\/blog\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible\/"},"modified":"2019-10-31T22:00:43","modified_gmt":"2019-10-31T19:00:43","slug":"avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible","title":{"rendered":"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/c60a2326ab13e7766eafda3ae5ac39c5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAhoj v\u0161em. Pracuji jako vedouc\u00ed syst\u00e9mov\u00fd administr\u00e1tor v O\u041a a odpov\u00edd\u00e1m za stabiln\u00ed provoz port\u00e1lu. Chci se pod\u011blit o to, jak jsme vytvo\u0159ili proces automatick\u00e9 v\u00fdm\u011bny disk\u016f a jak jsme pot\u00e9 ze procesu vylou\u010dili administr\u00e1tora a nahradili ho botem.<\/p>\n<p>Tento \u010dl\u00e1nek je jakousi transkripc\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=5WhbG3FQveE&amp;list=FLKN7KW1sxju7fWJvKuyKxEQ\">vystoupen\u00ed<\/a><\/noindex> na HighLoad+ 2018<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Vytvo\u0159en\u00ed procesu v\u00fdm\u011bny disk\u016f<\/h2>\n<p><\/p>\n<h3>Nejprve n\u011bkolik \u010d\u00edsel<\/h3>\n<p>\nO\u041a \u2014 to je obrovsk\u00fd servis, kter\u00fd pou\u017e\u00edvaj\u00ed miliony lid\u00ed. Jeho provoz zabezpe\u010duje kolem 7 tis\u00edc server\u016f, kter\u00e9 se nach\u00e1zej\u00ed ve 4 r\u016fzn\u00fdch datov\u00fdch centrech. Na serverech je v\u00edce ne\u017e 70 tis\u00edc disk\u016f. Pokud bychom je naskl\u00e1dali na sebe, dostali bychom v\u011b\u017e vy\u0161\u0161\u00ed ne\u017e 1 km. <\/p>\n<p>Pevn\u00e9 disky jsou sou\u010d\u00e1st\u00ed serveru, kter\u00e1 se porouch\u00e1v\u00e1 nej\u010dast\u011bji. P\u0159i takov\u00fdch objemech mus\u00edme m\u011bnit kolem 30 disk\u016f t\u00fddn\u011b, a tento proces se stal nep\u0159\u00edjemnou rutinou.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/5e915e9077994efcc4e74a26fc6286fa.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Incidenty<\/h3>\n<p>\nV na\u0161\u00ed spole\u010dnosti je zavedena plnohodnotn\u00e1 spr\u00e1va incident\u016f. Ka\u017ed\u00fd incident zaznamen\u00e1v\u00e1me do Jiry, a pot\u00e9 jej \u0159e\u0161\u00edme a analyzujeme. Pokud m\u011bl incident vliv na u\u017eivatele, ur\u010dit\u011b se sejdeme a p\u0159em\u00fd\u0161l\u00edme, jak rychleji reagovat v takov\u00fdch p\u0159\u00edpadech, jak sn\u00ed\u017eit dopad a samoz\u0159ejm\u011b, jak p\u0159edej\u00edt opakov\u00e1n\u00ed.<\/p>\n<p>\u00dalo\u017en\u00e1 za\u0159\u00edzen\u00ed nejsou v\u00fdjimkou. O jejich stavu se star\u00e1 Zabbix. Monitorujeme zpr\u00e1vy v Syslog na p\u0159\u00edtomnost chyb p\u0159i z\u00e1pisu\/\u010dten\u00ed, analyzujeme stav HW\/SW RAID\u016f, sledujeme SMART, pro SSD po\u010d\u00edt\u00e1me opot\u0159eben\u00ed. <\/p>\n<h3>Jak se d\u0159\u00edve m\u011bnily disky<\/h3>\n<p>\nKdy\u017e se v Zabbixu rozsv\u00edt\u00ed n\u011bjak\u00fd spou\u0161t\u011b\u010d, v Jize se vytvo\u0159\u00ed incident a automaticky se p\u0159id\u011bluje p\u0159\u00edslu\u0161n\u00fdm in\u017een\u00fdr\u016fm v datov\u00fdch centrech. Toto d\u011bl\u00e1me se v\u0161emi HW incidenty, tedy takov\u00fdmi, kter\u00e9 vy\u017eaduj\u00ed fyzickou pr\u00e1ci s vybaven\u00edm v datov\u00e9m centru. <br \/>\nIn\u017een\u00fdr datov\u00e9ho centra je \u010dlov\u011bk, kter\u00fd \u0159e\u0161\u00ed ot\u00e1zky spojen\u00e9 se \u017eelezem, odpov\u00edd\u00e1 za instalaci, \u00fadr\u017ebu a demont\u00e1\u017e server\u016f. Po obdr\u017een\u00ed tiketu se in\u017een\u00fdr pust\u00ed do pr\u00e1ce. V diskov\u00fdch polic\u00edch m\u011bn\u00ed disky s\u00e1m. Pokud v\u0161ak nem\u00e1 p\u0159\u00edstup k pot\u0159ebn\u00e9mu za\u0159\u00edzen\u00ed, obr\u00e1t\u00ed se na pohotovostn\u00ed syst\u00e9mov\u00e9 administr\u00e1tory o pomoc. Nejd\u0159\u00edve je t\u0159eba disk vyjmout z rotace. K tomu je t\u0159eba prov\u00e9st nezbytn\u00e9 zm\u011bny na serveru, zastavit aplikace a odpojit disk.<\/p>\n<p>Pohotovostn\u00ed syst\u00e9mov\u00fd administr\u00e1tor b\u011bhem pracovn\u00ed sm\u011bny odpov\u00edd\u00e1 za spr\u00e1vu cel\u00e9ho port\u00e1lu. Vy\u0161et\u0159uje incidenty, prov\u00e1d\u00ed opravy a pom\u00e1h\u00e1 v\u00fdvoj\u00e1\u0159\u016fm plnit drobn\u00e9 \u00fakoly. Nezab\u00fdv\u00e1 se pouze pevn\u00fdm diskem.<\/p>\n<p>D\u0159\u00edve in\u017een\u00fd\u0159i datov\u00fdch center komunikovali se syst\u00e9mov\u00fdm administr\u00e1torem v chatu. In\u017een\u00fd\u0159i pos\u00edlali odkazy na Jira tickety, administr\u00e1tor je proch\u00e1zel a vedl si z\u00e1znamy o pr\u00e1ci v n\u011bjak\u00e9m bloku. Ale pro takov\u00e9 \u00fakoly jsou chaty nepohodln\u00e9: informace tam nejsou strukturovan\u00e9 a rychle se ztr\u00e1c\u00ed. A administr\u00e1tor mohl jednodu\u0161e opustit po\u010d\u00edta\u010d a n\u011bjakou dobu neodpov\u00eddat na po\u017eadavky, zat\u00edmco in\u017een\u00fdr st\u00e1l u serveru s hromadou disk\u016f a \u010dekal.<\/p>\n<p>Nejhor\u0161\u00ed bylo, \u017ee administr\u00e1to\u0159i nev\u011bd\u011bli, jak\u00e1 je celkov\u00e1 situace: jak\u00e9 diskov\u00e9 incidenty existuj\u00ed a kde by mohly nastat potenci\u00e1ln\u00ed probl\u00e9my. To je d\u00e1no t\u00edm, \u017ee v\u0161echny HW incidenty p\u0159ed\u00e1v\u00e1me in\u017een\u00fdr\u016fm. Ano, mohli bychom zobrazit v\u0161echny incidenty na administr\u00e1torsk\u00e9m dashboardu. Ale je jich p\u0159\u00edli\u0161 mnoho a administr\u00e1tor byl zapojen jen do n\u011bkter\u00fdch z nich.<\/p>\n<p>Krom\u011b toho in\u017een\u00fdr nemohl spr\u00e1vn\u011b stanovit priority, proto\u017ee nic nev\u011bd\u011bl o \u00fa\u010delu konkr\u00e9tn\u00edch server\u016f a o rozd\u011blen\u00ed informac\u00ed mezi diskami.<\/p>\n<h3>Nov\u00fd postup v\u00fdm\u011bny<\/h3>\n<p>\nPrvn\u00ed, co jsme ud\u011blali, bylo vy\u010dlenit v\u0161echny diskov\u00e9 incidenty do samostatn\u00e9ho typu \"HW-disk\" a p\u0159idat k n\u011bmu pole \"n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed\", \"velikost\" a \"typ disku\", aby tyto informace byly uchov\u00e1v\u00e1ny v tiketu a nemusely se neust\u00e1le vym\u011b\u0148ovat v chatu. <\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/c4526bf668e31d90e0fbd061de44f1a7.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nTak\u00e9 jsme se dohodli, \u017ee v r\u00e1mci jednoho incidentu vym\u011bn\u00edme pouze jeden disk. To v\u00fdrazn\u011b usnadnilo n\u00e1sledn\u00fd proces automatizace, sb\u011br statistik a pr\u00e1ci. <\/p>\n<p>Krom\u011b toho jsme p\u0159idali pole \"odpov\u011bdn\u00fd administr\u00e1tor\". Tam se automaticky dopl\u0148uje pohotovostn\u00ed syst\u00e9mov\u00fd administr\u00e1tor. To je velmi v\u00fdhodn\u00e9, proto\u017ee in\u017een\u00fdr nyn\u00ed v\u017edy vid\u00ed, kdo je zodpov\u011bdn\u00fd. Nen\u00ed pot\u0159eba chodit do kalend\u00e1\u0159e a hledat. Pr\u00e1v\u011b toto pole umo\u017enilo vyv\u011bsit na administr\u00e1torsk\u00e9m dashboardu tikety, ve kter\u00fdch by mohla b\u00fdt jeho pomoc pot\u0159ebn\u00e1.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/5097e5dcca1fddd68d1e5f296c09f60a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAby v\u0161ichni \u00fa\u010dastn\u00edci mohli z\u00edskat maxim\u00e1ln\u00ed prosp\u011bch z novinek, vytvo\u0159ili jsme filtry a dashboardy a informovali jsme o nich v\u0161echny z\u00fa\u010dastn\u011bn\u00e9. Kdy\u017e lid\u00e9 ch\u00e1pou zm\u011bny, nedistancuj\u00ed se od nich jako od n\u011b\u010deho zbyte\u010dn\u00e9ho. Pro in\u017een\u00fdra je d\u016fle\u017eit\u00e9 zn\u00e1t \u010d\u00edslo racku, kde se nach\u00e1z\u00ed server, velikost a typ disku. Spr\u00e1vci je d\u016fle\u017eit\u00e9 p\u0159edev\u0161\u00edm pochopit, o jakou skupinu server\u016f se jedn\u00e1 a jak\u00fd m\u016f\u017ee b\u00fdt efekt p\u0159i v\u00fdm\u011bn\u011b disku.<\/p>\n<p>Existence pol\u00ed a jejich zobrazen\u00ed je u\u017eite\u010dn\u00e9, ale nezbavili jsme se t\u00edm pot\u0159eby pou\u017e\u00edvat chaty. Museli jsme zm\u011bnit pracovn\u00ed proces. <\/p>\n<p>D\u0159\u00edve to vypadalo takto:<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/2b79cdb7ec88b5ae3d2917ad8b164c6d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nDnes takto st\u00e1le pracuj\u00ed in\u017een\u00fd\u0159i, kdy\u017e nepot\u0159ebuj\u00ed pomoc spr\u00e1vce.<\/p>\n<p>Prvn\u00ed, co jsme ud\u011blali, bylo zaveden\u00ed nov\u00e9ho stavu <b>Investigate<\/b>. V t\u00e9to f\u00e1zi je tiket, kdy\u017e in\u017een\u00fdr je\u0161t\u011b nerozhodl, zda bude pot\u0159ebovat spr\u00e1vce, nebo ne. Skrze tento stav m\u016f\u017ee in\u017een\u00fdr p\u0159edat tiket spr\u00e1vci. Krom\u011b toho t\u00edmto stavem ozna\u010dujeme tickety, kdy\u017e je t\u0159eba vym\u011bnit disk, ale disk na m\u00edst\u011b nen\u00ed. To se st\u00e1v\u00e1 v p\u0159\u00edpad\u011b CDN a vzd\u00e1len\u00fdch lokalit.<\/p>\n<p>Tak\u00e9 jsme p\u0159idali stav <b>Ready<\/b>. Do n\u011bj se tiket p\u0159ev\u00e1d\u00ed po v\u00fdm\u011bn\u011b disku. To znamen\u00e1, \u017ee v\u0161e ji\u017e bylo ud\u011bl\u00e1no, ale na serveru se synchronizuje HW\/SW RAID. To m\u016f\u017ee trvat docela dlouho.<\/p>\n<p>Pokud je do pr\u00e1ce zapojen spr\u00e1vce, sch\u00e9ma se m\u00edrn\u011b komplikuje.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/5b6aacbb0d4049e62e7cb9ee28eb6498.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nZ stavu <b>Otev\u0159\u00edt<\/b> m\u016f\u017ee tiket p\u0159ev\u00e9st jak syst\u00e9mov\u00fd administr\u00e1tor, tak in\u017een\u00fdr. Ve stavu <b>In progress<\/b> administr\u00e1tor odpojuje disk z rotace, aby in\u017een\u00fdr mohl prost\u011b vyt\u00e1hnout: zapne osv\u011btlen\u00ed, odmontuje disk, zastav\u00ed aplikace, v z\u00e1vislosti na konkr\u00e9tn\u00ed skupin\u011b server\u016f.<\/p>\n<p>Pot\u00e9 se tiket p\u0159evede do <b>Ready to change<\/b>: to je sign\u00e1l pro in\u017een\u00fdra, \u017ee disk lze vyt\u00e1hnout. V\u0161echna pole v Jira jsou ji\u017e vypln\u011bna, in\u017een\u00fdr v\u00ed, jak\u00fd typ a velikost disku. Tato data jsou zad\u00e1van\u00e1 bu\u010f automaticky p\u0159i p\u0159edchoz\u00edm stavu, nebo spr\u00e1vcem.<\/p>\n<p>Po v\u00fdm\u011bn\u011b disku se tiket p\u0159evede do stavu <b>Changed<\/b>. Kontroluje se, \u017ee byl vlo\u017een spr\u00e1vn\u00fd disk, prov\u00e1d\u00ed se rozlo\u017een\u00ed, spust\u00ed se aplikace a n\u011bkter\u00e9 \u00fakoly pro obnovu dat. Tak\u00e9 m\u016f\u017ee b\u00fdt tiket p\u0159eveden do stavu <b>Ready<\/b>, v tom p\u0159\u00edpad\u011b z\u016fst\u00e1v\u00e1 odpov\u011bdn\u00fdm spr\u00e1vcem, proto\u017ee on disk do rotace zav\u00e1d\u011bl. Celkov\u00e9 sch\u00e9ma vypad\u00e1 takto.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/32f8d488e2ff6a28a3341114b66de0a3.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nP\u0159id\u00e1n\u00ed nov\u00fdch pol\u00ed n\u00e1m v\u00fdrazn\u011b usnadnilo \u017eivot. Kluci za\u010dali pracovat se strukturovan\u00fdmi informacemi, tak\u017ee bylo jasn\u00e9, co a v jak\u00e9 f\u00e1zi je pot\u0159eba ud\u011blat. Prioritizace se stala mnohem relevantn\u011bj\u0161\u00ed, proto\u017ee je nyn\u00ed nastavuje administr\u00e1tor.<\/p>\n<p>Odpadla pot\u0159eba v chatech. Samoz\u0159ejm\u011b, administr\u00e1tor m\u016f\u017ee napsat in\u017een\u00fdrovi \"tady je pot\u0159eba vym\u011bnit rychleji\" nebo \"u\u017e je ve\u010der, stihne\u0161 to vym\u011bnit?\". Ale u\u017e se o t\u011bchto v\u011bcech ka\u017ed\u00fd den v chatech nebav\u00edme.<\/p>\n<p>Disky se za\u010daly m\u011bnit po skupin\u00e1ch. Pokud administr\u00e1tor p\u0159ijde do pr\u00e1ce o n\u011bco d\u0159\u00edve, m\u00e1 voln\u00fd \u010das a zat\u00edm se nic nestalo, m\u016f\u017ee p\u0159ipravit \u0159adu server\u016f k v\u00fdm\u011bn\u011b: nastavit pole, vy\u0159adit disky z rotace a p\u0159edat \u00fakol in\u017een\u00fdrovi. In\u017een\u00fdr p\u0159ijde do datov\u00e9ho centra o chv\u00edli pozd\u011bji, vid\u00ed \u00fakol, vezme pot\u0159ebn\u00e9 \u00falo\u017ei\u0161t\u011b ze skladu a okam\u017eit\u011b je vym\u011bn\u00ed. V\u00fdsledkem je zv\u00fd\u0161en\u00ed rychlosti v\u00fdm\u011bny.<\/p>\n<h3>Nastaven\u00e9 zku\u0161enosti p\u0159i budov\u00e1n\u00ed Workflow<\/h3>\n<p><\/p>\n<ul>\n<li><b>P\u0159i vytv\u00e1\u0159en\u00ed procedury je t\u0159eba shrom\u00e1\u017edit informace z r\u016fzn\u00fdch zdroj\u016f.<\/b><br \/>\nN\u011bkte\u0159\u00ed na\u0161i administr\u00e1to\u0159i nev\u011bd\u011bli, \u017ee in\u017een\u00fdr m\u011bn\u00ed disky s\u00e1m. N\u011bkte\u0159\u00ed si mysleli, \u017ee za synchronizaci MD RAID dohl\u00ed\u017eej\u00ed in\u017een\u00fd\u0159i, p\u0159esto\u017ee n\u011bkte\u0159\u00ed z nich nem\u011bli k tomu p\u0159\u00edstup. N\u011bkte\u0159\u00ed vedouc\u00ed in\u017een\u00fd\u0159i to d\u011blali, ale ne v\u017edy, proto\u017ee proces nikde nebyl pops\u00e1n.<\/li>\n<li><b>Procedura by m\u011bla b\u00fdt jednoduch\u00e1 a srozumiteln\u00e1.<\/b><br \/>\nJe t\u011b\u017ek\u00e9 udr\u017eet na pam\u011bti mnoho krok\u016f. Nejpodstatn\u011bj\u0161\u00ed sousedn\u00ed stavy v Jira je pot\u0159eba um\u00edstit na hlavn\u00ed obrazovku. M\u016f\u017eeme je p\u0159ejmenovat, nap\u0159\u00edklad In progress nazveme Ready to change. Ostatn\u00ed stavy m\u016f\u017eeme skr\u00fdt do rozbalovac\u00edho menu, aby nevy\u010dn\u00edvaly. Ale je lep\u0161\u00ed neomezen\u00ed lid\u00ed, d\u00e1t mo\u017enost prov\u00e9st p\u0159echod.<br \/>\nVysv\u011btlete hodnotu novinek. Kdy\u017e lid\u00e9 ch\u00e1pou, l\u00e9pe p\u0159ij\u00edmaj\u00ed novou proceduru. Pro n\u00e1s bylo velmi d\u016fle\u017eit\u00e9, aby lid\u00e9 neklikali na cel\u00fd proces, ale \u0161li j\u00edm. Pak jsme na tom budovali automatizaci.<\/li>\n<li><b>\u010cekat, analyzovat, zkoumat.<\/b><br \/>\nTrvalo n\u00e1m asi m\u011bs\u00edc vybudovat proceduru, technicky ji realizovat, uspo\u0159\u00e1dat sch\u016fzky a diskuse. A na zaveden\u00ed \u2014 v\u00edce ne\u017e t\u0159i m\u011bs\u00edce. Vid\u011bl jsem, jak lid\u00e9 pomalu za\u010d\u00ednaj\u00ed novinku pou\u017e\u00edvat. V prvn\u00ed f\u00e1zi bylo hodn\u011b negativn\u00edch reakc\u00ed. Ale to nijak nesouviselo s procedurou nebo jej\u00ed technickou realizac\u00ed. Nap\u0159\u00edklad jeden administr\u00e1tor pou\u017e\u00edval nikoli Jira, ale Jira plugin v Confluence, a n\u011bkter\u00e9 v\u011bci mu nebyly dostupn\u00e9. Uk\u00e1zali jsme mu Jira, jeho produktivita vzrostla jak v b\u011b\u017en\u00fdch \u00fakolech, tak i p\u0159i v\u00fdm\u011bn\u00e1ch disk\u016f.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Automatizace v\u00fdm\u011bny disk\u016f<\/h2>\n<p>\nK automatizaci v\u00fdm\u011bny disk\u016f jsme se pokusili p\u0159istoupit n\u011bkolikr\u00e1t. M\u011bli jsme u\u017e n\u011bjak\u00e9 skripty, ale v\u0161echny pracovaly bu\u010f v interaktivn\u00edm nebo v manu\u00e1ln\u00edm re\u017eimu, vy\u017eadovaly spu\u0161t\u011bn\u00ed. A teprve po zaveden\u00ed nov\u00e9 procedury jsme si uv\u011bdomili, \u017ee n\u00e1m pr\u00e1v\u011b to chyb\u011blo.<\/p>\n<p>Proto\u017ee jsme nyn\u00ed proces v\u00fdm\u011bny rozd\u011blili na f\u00e1ze, kde ka\u017edou m\u00e1 na starosti ur\u010dit\u00fd vykonavatel a seznam \u00fakon\u016f, m\u016f\u017eeme zap\u00ednat automatizaci postupn\u011b, a ne hned celou. Nap\u0159\u00edklad nejjednodu\u0161\u0161\u00ed f\u00e1zi \u2014 Ready (kontrola synchronizace RAID\/dat) lze snadno delegovat botovi. Jakmile se bot trochu nau\u010d\u00ed, m\u016f\u017eeme mu d\u00e1t zodpov\u011bdn\u011bj\u0161\u00ed \u00fakol \u2014 za\u0159azen\u00ed disku do rotace atd.<\/p>\n<h3>Zoologick\u00e1 zahrada konfigurac\u00ed<\/h3>\n<p>\nNe\u017e za\u010dneme mluvit o botovi, ud\u011bl\u00e1me si kr\u00e1tk\u00fd v\u00fdlet do na\u0161\u00ed zoologick\u00e9 zahrady instalac\u00ed. V prvn\u00ed \u0159ad\u011b je to d\u00e1no obrovskou velikost\u00ed na\u0161\u00ed infrastruktury. Zadruh\u00e9 se sna\u017e\u00edme pro ka\u017ed\u00fd servis vybrat optim\u00e1ln\u00ed konfiguraci hardwaru. M\u00e1me asi 20 model\u016f hardwarov\u00fdch RAID, p\u0159ev\u00e1\u017en\u011b LSI a Adaptec, ale objevuj\u00ed se i HP a DELL r\u016fzn\u00fdch verz\u00ed. Ka\u017ed\u00fd RAID kontrol\u00e9r m\u00e1 svou vlastn\u00ed \u0159\u00edd\u00edc\u00ed utilitu. Sada p\u0159\u00edkaz\u016f a v\u00fdstupy se mohou li\u0161it mezi verzemi ka\u017ed\u00e9ho RAID kontrol\u00e9ru. Tam, kde se nepou\u017e\u00edv\u00e1 HW-RAID, m\u016f\u017ee b\u00fdt mdraid.<\/p>\n<p>S t\u00e9m\u011b\u0159 v\u0161emi nov\u00fdmi instalacemi se sna\u017e\u00edme obej\u00edt diskov\u00e9 z\u00e1lohov\u00e1n\u00ed. Sna\u017e\u00edme se v\u00edce nepou\u017e\u00edvat hardwarov\u00e9 a softwarov\u00e9 RAID, proto\u017ee z\u00e1lohujeme na\u0161e syst\u00e9my na \u00farovni datov\u00fdch center, nikoli server\u016f. Ale samoz\u0159ejm\u011b existuje mnoho star\u0161\u00edch server\u016f, kter\u00e9 je t\u0159eba podporovat.<\/p>\n<p>N\u011bkde jsou disky v RAID kontrolerech spojeny jako raw za\u0159\u00edzen\u00ed, jinde se pou\u017e\u00edv\u00e1 JBOD. Existuj\u00ed konfigurace se samotn\u00fdm syst\u00e9mov\u00fdm diskem v serveru, a pokud je t\u0159eba ho vym\u011bnit, je nutn\u00e9 server p\u0159einstalovat s opera\u010dn\u00edm syst\u00e9mem a aplikacemi, a to ve stejn\u00fdch verz\u00edch, pot\u00e9 p\u0159idat konfigura\u010dn\u00ed soubory a spustit aplikace. Tak\u00e9 existuje \u0159ada skupin server\u016f, kde se z\u00e1lohov\u00e1n\u00ed prov\u00e1d\u00ed nikoli na \u00farovni diskov\u00e9ho subsyst\u00e9mu, ale p\u0159\u00edmo v samotn\u00fdch aplikac\u00edch.<\/p>\n<p>Celkov\u011b m\u00e1me v\u00edc ne\u017e 400 unik\u00e1tn\u00edch skupin server\u016f, na kter\u00fdch b\u011b\u017e\u00ed kolem 100 r\u016fzn\u00fdch aplikac\u00ed. Abychom pokryli tak obrovsk\u00e9 mno\u017estv\u00ed variant, pot\u0159ebovali jsme multifunk\u010dn\u00ed automatiza\u010dn\u00ed n\u00e1stroj. Ide\u00e1ln\u011b s jednoduch\u00fdm DSL, aby ho mohl udr\u017eovat nejen ten, kdo ho napsal.<\/p>\n<p>Vybrali jsme Ansible, proto\u017ee je bezagentov\u00fd: nebylo t\u0159eba p\u0159ipravovat infrastrukturu, rychl\u00fd start. Nav\u00edc je napsan\u00fd v Pythonu, kter\u00fd je v t\u00fdmu p\u0159ijat jako standard.<\/p>\n<h3>Obecn\u00e1 sch\u00e9ma<\/h3>\n<p>\nPod\u00edvejme se na obecn\u00fd diagram automatizace na p\u0159\u00edkladu jednoho incidentu. Zabbix detekuje, \u017ee disk sdb selhal, spust\u00ed se trigger, vytvo\u0159\u00ed se tiket v Jira. Administr\u00e1tor se na n\u011bj pod\u00edv\u00e1, pochop\u00ed, \u017ee to nen\u00ed duplik\u00e1t a nen\u00ed to false positive, tedy \u017ee je t\u0159eba disk vym\u011bnit, a p\u0159evede tiket do stavu In progress.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/673df0fe9600bc65633841222f436d37.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAplikace DiskoBot napsan\u00e1 v Pythonu periodicky prohled\u00e1v\u00e1 Jira na nov\u00e9 tikety. Zjist\u00ed, \u017ee se objevil nov\u00fd tiket ve stavu In progress, spust\u00ed se odpov\u00eddaj\u00edc\u00ed thread, kter\u00fd spust\u00ed playbook v Ansible (to se d\u011bl\u00e1 pro ka\u017ed\u00fd stav v Jira). V tomto p\u0159\u00edpad\u011b se spou\u0161t\u00ed Prepare2change.<\/p>\n<p>Ansible se odes\u00edl\u00e1 na hostitele, vy\u0159azuje disk z rotace a hl\u00e1s\u00ed stav aplikaci prost\u0159ednictv\u00edm Callbacks. <\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/8bfc13129f8deca640eb63ea81f616c7.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nNa z\u00e1klad\u011b v\u00fdsledk\u016f bot automaticky p\u0159evede tiket do stavu Ready to change. In\u017een\u00fdr dost\u00e1v\u00e1 ozn\u00e1men\u00ed a jde vym\u011bnit disk, po \u010dem\u017e p\u0159evede tiket do stavu Changed. <\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/eeda06b6966f435e2172d3f9c61b341e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nPodle v\u00fd\u0161e popsan\u00e9ho sch\u00e9matu se tiket vr\u00e1t\u00ed zp\u011bt k botovi, ten spust\u00ed jin\u00fd playbook, jde na hostitele a uvede disk do rotace. Bot uzav\u00edr\u00e1 tiket. Hur\u00e1!<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/51b8c7050b0c10c16ad6a6d4974c3ebf.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nTe\u010f se poj\u010fme pod\u00edvat na n\u011bkter\u00e9 komponenty syst\u00e9mu.<\/p>\n<h3>Diskobot<\/h3>\n<p>\nTato aplikace je napsan\u00e1 v Pythonu. Vyb\u00edr\u00e1 tikety z Jira podle JQL. V z\u00e1vislosti na stavu tiketu, posledn\u00ed \u00fakol p\u0159ech\u00e1z\u00ed k odpov\u00eddaj\u00edc\u00edmu zpracovateli, kter\u00fd na opl\u00e1tku spust\u00ed odpov\u00eddaj\u00edc\u00ed Ansible playbook podle stavu.<\/p>\n<p>JQL a intervaly dotaz\u016f jsou definov\u00e1ny v konfigura\u010dn\u00edm souboru aplikace.<\/p>\n<pre><code class=\"plaintext\">jira_states:\n  investigate:\n    jql: '\u2026 status = Open and \"Disk Size\" is EMPTY'\n    interval: 180\n\n  inprogress:\n    jql: '\u2026  and \"Disk Size\" is not EMPTY and \"Device Name\" is not EMPTY'\n \n  ready:\n    jql: '\u2026 and (labels not in (\"dbot_ignore\") or labels is EMPTY)'\n    interval: 7200\n<\/code><\/pre>\n<p>\nNap\u0159\u00edklad, mezi tikety ve stavu In progress, se vyb\u00edraj\u00ed pouze ty, ve kter\u00fdch jsou vypln\u011bna pole Disk size a Device name. Device name je n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed pot\u0159ebn\u00e9ho pro proveden\u00ed playbooku. Disk size je pot\u0159ebn\u00fd, aby in\u017een\u00fdr v\u011bd\u011bl, jak velk\u00fd disk je t\u0159eba.<\/p>\n<p>A mezi tikety se stavem Ready se filtruj\u00ed tikety s labelem dbot_ignore. Mimochodem, Jira labely pou\u017e\u00edv\u00e1me jak pro takov\u00e9 filtrov\u00e1n\u00ed, tak pro ozna\u010dov\u00e1n\u00ed duplicitn\u00edch tiket\u016f a sb\u011br statistiky.<\/p>\n<p>V p\u0159\u00edpad\u011b selh\u00e1n\u00ed playbooku Jira p\u0159i\u0159ad\u00ed label dbot_failed, aby bylo mo\u017en\u00e9 se pozd\u011bji vy\u0159e\u0161it. <\/p>\n<h3>Interakce s Ansible<\/h3>\n<p>\nAplikace interaguje s Ansible prost\u0159ednictv\u00edm <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.ansible.com\/ansible\/latest\/dev_guide\/developing_api.html\">Ansible Python API<\/a><\/noindex>. V playbook_executor p\u0159ed\u00e1v\u00e1me n\u00e1zev souboru a sadu prom\u011bnn\u00fdch. To umo\u017e\u0148uje uchov\u00e1vat Ansible projekt jako b\u011b\u017en\u00e9 yml soubory, nikoli jej popisovat v Python k\u00f3du. <\/p>\n<p>Tak\u00e9 v Ansible jsou p\u0159es *extra_vars* p\u0159ed\u00e1v\u00e1ny n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed, stav tiketu a callback_url, kde je zak\u00f3dov\u00e1n issue key \u2014 je pou\u017e\u00edv\u00e1n pro callback v HTTP.<\/p>\n<p>Pro ka\u017ed\u00e9 spu\u0161t\u011bn\u00ed se generuje do\u010dasn\u00e1 inventory, kter\u00e1 se skl\u00e1d\u00e1 z jednoho hostitele a skupiny, do kter\u00e9 tento hostitel pat\u0159\u00ed, aby se uplatnily group_vars.<\/p>\n<p>Zde je p\u0159\u00edklad \u00fakolu, ve kter\u00e9m je implementov\u00e1n HTTP callback.<\/p>\n<p>V\u00fdsledek proveden\u00ed playbook\u016f z\u00edsk\u00e1v\u00e1me pomoc\u00ed callback(-\u016f). Jsou dvou typ\u016f:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/docs.ansible.com\/ansible\/latest\/plugins\/callback.html\">Ansible callback plugin<\/a><\/noindex>, poskytuje \u00fadaje o v\u00fdsledc\u00edch proveden\u00ed playbooku. Popisuje \u00fakoly, kter\u00e9 byly spu\u0161t\u011bny, \u00fasp\u011b\u0161n\u011b vykon\u00e1ny nebo ne\u00fasp\u011b\u0161n\u011b vykon\u00e1ny. Tento callback se vyvol\u00e1 po skon\u010den\u00ed p\u0159ehr\u00e1v\u00e1n\u00ed playbooku.<\/li>\n<li>HTTP callback pro z\u00edsk\u00e1v\u00e1n\u00ed informac\u00ed b\u011bhem p\u0159ehr\u00e1v\u00e1n\u00ed playbooku. V Ansible \u00fakolu prov\u00e1d\u00edme POST\/GET dotaz sm\u011brem k na\u0161\u00ed aplikaci.<\/li>\n<\/ul>\n<p>\nPomoc\u00ed HTTP callback(-\u016f) se p\u0159ed\u00e1vaj\u00ed prom\u011bnn\u00e9, kter\u00e9 byly definov\u00e1ny p\u0159i prov\u00e1d\u011bn\u00ed playbooku a kter\u00e9 chceme zachovat a pou\u017e\u00edt p\u0159i dal\u0161\u00edch spu\u0161t\u011bn\u00edch. Tyto \u00fadaje zapisujeme do sqlite.<\/p>\n<p>Tak\u00e9 pomoc\u00ed HTTP callbacku zanech\u00e1v\u00e1me koment\u00e1\u0159e a m\u011bn\u00edme stav tiketu.<\/p>\n<p><b class=\"spoiler_title\">HTTP callback<\/b><\/p>\n<pre><code class=\"plaintext\"># Make callback to Diskobot App\n# Variables:\n#    callback_post_body: # A dict with follow keys. All keys are optional\n#       msg: If exist it would be posted to Jira as comment\n#       data: If exist it would be saved in Incident.variables\n#       desire_state: Set desire_state for incident\n#       status: If exist Proceed issue to that status\n\n  - name: Callback to Diskobot app (jira comment\/status)\n    uri:\n      url: \"{{ callback_url }}\/{{ devname }}\"\n      user: \"{{ diskobot_user }}\"\n      password: \"{{ diskobot_pass }}\"\n      force_basic_auth: True\n      method: POST\n      body: \"{{ callback_post_body | to_json }}\"\n      body_format: json\n    delegate_to: 127.0.0.1\n<\/code><\/pre>\n<p>Stejn\u011b jako u mnoha podobn\u00fdch \u00faloh jsme to vy\u010dlenili do samostatn\u00e9ho b\u011b\u017en\u00e9ho souboru a zahrnujeme ho podle pot\u0159eby, abychom se v playbooku neust\u00e1le neopakovali. Zde se objevuje callback_url, ve kter\u00e9m jsou zak\u00f3dov\u00e1ny issue key a host name. Kdy\u017e Ansible provede tento POST po\u017eadavek, bot rozum\u00ed, \u017ee p\u0159i\u0161el v r\u00e1mci ur\u010dit\u00e9ho incidentu.<\/p>\n<p>A tady je p\u0159\u00edklad z playbooku, ve kter\u00e9m jsme odpoj\u00ed disk z MD za\u0159\u00edzen\u00ed:<\/p>\n<pre><code class=\"plaintext\">  # Save mdadm configuration\n  - include: common\/callback.yml\n    vars:\n      callback_post_body:\n        status: 'Ready to change'\n        msg: \"Removed disk from mdraid {{ mdadm_remove_disk.msg | comment_jira }}\"\n        data:\n          mdadm_data: \"{{ mdadm_remove_disk.removed }}\"\n          parted_info: \"{{ parted_info | default() }}\"\n    when:\n      - mdadm_remove_disk | changed\n      - mdadm_remove_disk.removed\n<\/code><\/pre>\n<p>\nTato \u00faloha p\u0159ev\u00e1d\u00ed Jira tiket do stavu \u201eReady to change\u201c a p\u0159id\u00e1v\u00e1 koment\u00e1\u0159. Tak\u00e9 se v prom\u011bnn\u00e9 mdam_data ukl\u00e1d\u00e1 seznam md za\u0159\u00edzen\u00ed, ze kter\u00fdch byl disk odstran\u011bn, a v parted_info \u2014 dump partition od parted. <\/p>\n<p>Kdy\u017e in\u017een\u00fdr vlo\u017e\u00ed nov\u00fd disk, m\u016f\u017eeme pou\u017e\u00edt tyto prom\u011bnn\u00e9 k obnoven\u00ed dumpu partition a tak\u00e9 p\u0159idat disk do t\u011bch md za\u0159\u00edzen\u00ed, ze kter\u00fdch byl odstran\u011bn.<\/p>\n<h3>Ansible kontroln\u00ed re\u017eim<\/h3>\n<p>\nSpou\u0161t\u011bn\u00ed automatizace bylo d\u011bsiv\u00e9. Proto jsme se rozhodli spou\u0161t\u011bt v\u0161echny playbooky v re\u017eimu <br \/>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/docs.ansible.com\/ansible\/latest\/user_guide\/playbooks_checkmode.html\">dry run<\/a><\/noindex>, ve kter\u00e9m Ansible na serverech neprov\u00e1d\u00ed \u017e\u00e1dn\u00e9 akce, ale pouze je emuluje. <\/p>\n<p>Takov\u00e9 spu\u0161t\u011bn\u00ed prob\u00edh\u00e1 prost\u0159ednictv\u00edm samostatn\u00e9ho callback modulu a v\u00fdsledek proveden\u00ed playbooku se ukl\u00e1d\u00e1 do Jira jako koment\u00e1\u0159.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/b02d7f1dc428f2797ae8556cd496b115.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZa prv\u00e9, to umo\u017enilo validovat pr\u00e1ci bota a playbook\u016f. Za druh\u00e9, zv\u00fd\u0161ilo to d\u016fv\u011bru administr\u00e1tor\u016f v bota. <\/p>\n<p>Kdy\u017e jsme pro\u0161li validac\u00ed a pochopili, \u017ee m\u016f\u017eeme spou\u0161t\u011bt Ansible nejen v re\u017eimu dry run, vytvo\u0159ili jsme v Jira tla\u010d\u00edtko Run Diskobot pro spu\u0161t\u011bn\u00ed stejn\u00e9ho playbooku se stejn\u00fdmi prom\u011bnn\u00fdmi na stejn\u00e9m hostu, ale v b\u011b\u017en\u00e9m re\u017eimu. <\/p>\n<p>Krom\u011b toho se tla\u010d\u00edtko pou\u017e\u00edv\u00e1 pro op\u011btovn\u00e9 spu\u0161t\u011bn\u00ed playbooku v p\u0159\u00edpad\u011b jeho selh\u00e1n\u00ed.<\/p>\n<h3>Struktura Playbooks<\/h3>\n<p>\nU\u017e jsem zm\u00ednil, \u017ee v z\u00e1vislosti na stavu Jira tiketu bot spou\u0161t\u00ed r\u016fzn\u00e9 playbooky.<\/p>\n<p>Za prv\u00e9, tak je mnohem snaz\u0161\u00ed organizovat p\u0159\u00edstup. <br \/>\nZa druh\u00e9, v n\u011bkter\u00fdch p\u0159\u00edpadech je to zcela nezbytn\u00e9. <\/p>\n<p>Nap\u0159\u00edklad p\u0159i v\u00fdm\u011bn\u011b syst\u00e9mov\u00e9ho disku je nutn\u00e9 nejd\u0159\u00edve j\u00edt do syst\u00e9mu nasazen\u00ed, vytvo\u0159it \u00fakol, a po spr\u00e1vn\u00e9m nasazen\u00ed server bude dostupn\u00fd p\u0159es ssh a lze na n\u011bj nasadit aplikaci. Kdybychom to v\u0161echno d\u011blali v jednom playbooku, Ansible by ho nemohl prov\u00e9st kv\u016fli nedostupnosti hosta.<\/p>\n<p>Pou\u017e\u00edv\u00e1me Ansible role pro ka\u017edou skupinu server\u016f. Zde je vid\u011bt, jak jsou playbooky uspo\u0159\u00e1d\u00e1ny v jednom z nich. <\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/7a14035392b33f01d3182d5fff779321.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to pohodln\u00e9, proto\u017ee je hned jasn\u00e9, kde se nach\u00e1zej\u00ed jednotliv\u00e9 \u00fakoly. V main.yml, kter\u00fd slou\u017e\u00ed jako vstup pro Ansible roli, m\u016f\u017eeme m\u00edt prost\u011b include podle stavu tiketu nebo obecn\u00e9 \u00fakoly, kter\u00e9 jsou pot\u0159ebn\u00e9 pro v\u0161echny, nap\u0159\u00edklad proveden\u00ed identifikace nebo z\u00edsk\u00e1n\u00ed tokenu.<\/p>\n<h4>Investigation.yml<\/h4>\n<p>\nSpou\u0161t\u00ed se pro tikety ve stavu Investigation a Open. Nejd\u016fle\u017eit\u011bj\u0161\u00ed pro tento playbook je n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed. Tyto informace nejsou v\u017edy k dispozici. <\/p>\n<p>Pro jejich z\u00edsk\u00e1n\u00ed analyzujeme shrnut\u00ed Jira, posledn\u00ed hodnotu ze Zabbix triggeru. Tam m\u016f\u017ee b\u00fdt uveden n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed \u2014 m\u00e1me \u0161t\u011bst\u00ed. M\u016f\u017ee tam b\u00fdt tak\u00e9 mount point \u2014 pak je nutn\u00e9 j\u00edt na server, zpracovat a vypo\u010d\u00edtat pot\u0159ebn\u00fd disk. Trigger m\u016f\u017ee tak\u00e9 poskytnout scsi adresu nebo n\u011bjak\u00e9 jin\u00e9 informace. Ale n\u011bkdy se stane, \u017ee \u017e\u00e1dn\u00e9 stopy nejsou a je t\u0159eba analyzovat.<\/p>\n<p>Zjistiv\u0161i n\u00e1zev blokov\u00e9ho za\u0159\u00edzen\u00ed, shroma\u017e\u010fujeme informace o typu a velikosti disku pro vypln\u011bn\u00ed pol\u00ed v Jira. Takt\u00e9\u017e zaznamen\u00e1v\u00e1me informace o vendorovi, modelu, firmwaru, ID, SMART, a to v\u0161e vkl\u00e1d\u00e1me do koment\u00e1\u0159e v Jira tiketu. Administr\u00e1tor a in\u017een\u00fdr nyn\u00ed nemus\u00ed tyto \u00fadaje hledat. \ud83d\ude42<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/259bf8712c013ee00d238f111b4280fb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h4>prepare2change.yml<\/h4>\n<p>\nVytahov\u00e1n\u00ed disku z rotace, p\u0159\u00edprava na v\u00fdm\u011bnu. Nejd\u016fle\u017eit\u011bj\u0161\u00ed a nejt\u011b\u017e\u0161\u00ed f\u00e1ze. Pr\u00e1v\u011b zde m\u016f\u017eete zastavit aplikaci, kdy\u017e ji nen\u00ed mo\u017en\u00e9 zastavit. Nebo vyt\u00e1hnout disk, u kter\u00e9ho chyb\u011bly repliky, a t\u00edm ovlivnit u\u017eivatele, ztratit n\u011bjak\u00e1 data. Zde m\u00e1me nejv\u00edce kontrol a notifikac\u00ed v chatu.<\/p>\n<p>V nejjednodu\u0161\u0161\u00edm p\u0159\u00edpad\u011b jde o odstran\u011bn\u00ed disku z HW\/MD RAID. <\/p>\n<p>V komplikovan\u011bj\u0161\u00edch situac\u00edch (v na\u0161ich \u00falo\u017ei\u0161t\u00edch), kdy je z\u00e1lohov\u00e1n\u00ed prov\u00e1d\u011bno na \u00farovni aplikace, je nutn\u00e9 j\u00edt do aplikace p\u0159es API, informovat o vyjmut\u00ed disku, deaktivovat ho a zah\u00e1jit obnovu.<\/p>\n<p>Pr\u00e1v\u011b te\u010f masov\u011b migrujeme do <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/company\/odnoklassniki\/blog\/346868\">cloudu<\/a><\/noindex>, a pokud je server cloudov\u00fd, Diskobot se obrac\u00ed na API cloudu, \u0159\u00edk\u00e1, \u017ee se chyst\u00e1 pracovat s t\u00edmto minionem \u2014 serverem, na kter\u00e9m b\u011b\u017e\u00ed kontejnery, a \u017e\u00e1d\u00e1 \u201emigruj v\u0161echny kontejnery z tohoto miniona\u201c. A z\u00e1rove\u0148 zap\u00edn\u00e1 osv\u011btlen\u00ed disku, aby in\u017een\u00fdr okam\u017eit\u011b vid\u011bl, kter\u00fd je t\u0159eba vyt\u00e1hnout.<\/p>\n<h4>changed.yml<\/h4>\n<p>\nPo v\u00fdm\u011bn\u011b disku nejprve kontrolujeme jeho dostupnost. <\/p>\n<p>In\u017een\u00fd\u0159i ne v\u017edy vkl\u00e1daj\u00ed nov\u00e9 disky, proto jsme p\u0159idali kontrolu vyhovuj\u00edc\u00edch hodnot SMART.<\/p>\n<p><b class=\"spoiler_title\">Jak\u00e9 atributy sledujeme<\/b>Reallocated Sectors Count (5) &lt; 100<br \/>\nCurrent Pending Sector Count (107) == 0<\/p>\n<p>Pokud disk neprojde kontrolou, in\u017een\u00fdrovi se ozn\u00e1m\u00ed pot\u0159eba v\u00fdm\u011bny. Pokud je v\u0161e v po\u0159\u00e1dku, kontrolka se vypne, je provedena zna\u010den\u00ed a disk je za\u0159azen do rotace.<\/p>\n<h4>ready.yml<\/h4>\n<p>\nNejjednodu\u0161\u0161\u00ed p\u0159\u00edpad: kontrola synchronizace HW\/SW RAID nebo ukon\u010den\u00ed synchronizace dat v aplikaci.<\/p>\n<h3>API aplikac\u00ed<\/h3>\n<p>\nN\u011bkolikr\u00e1t jsem zm\u00ednil, \u017ee robot \u010dasto p\u0159istupuje k API aplikac\u00ed. Samoz\u0159ejm\u011b, \u017ee ne v\u0161echny aplikace m\u011bly pot\u0159ebn\u00e9 metody, proto bylo nutn\u00e9 je doplnit. Zde jsou nejd\u016fle\u017eit\u011bj\u0161\u00ed metody, kter\u00e9 pou\u017e\u00edv\u00e1me:<\/p>\n<ul>\n<li>Status. Stav clusteru nebo disku, abychom pochopili, zda s n\u00edm lze pracovat;\n<\/li>\n<li>Start\/stop. Aktivace-deaktivace disku;\n<\/li>\n<li>Migrate\/restore. Migrace a obnova dat b\u011bhem a po v\u00fdm\u011bn\u011b.\n<\/li>\n<\/ul>\n<p><\/p>\n<h3>Z\u00edskan\u00e9 zku\u0161enosti s Ansible<\/h3>\n<p>\nAnsible m\u00e1m velmi r\u00e1d. Ale \u010dasto, kdy\u017e se d\u00edv\u00e1m na r\u016fzn\u00e9 open-source projekty a vid\u00edm, jak lid\u00e9 p\u00ed\u0161\u00ed playbooky, tro\u0161ku se d\u011bs\u00edm. Slo\u017eit\u00e1 logick\u00e1 prov\u00e1zanost s when\/loop, nedostatek flexibility a idempotence kv\u016fli \u010dast\u00e9mu pou\u017e\u00edv\u00e1n\u00ed shell\/command.<\/p>\n<p>Rozhodli jsme se v\u0161e maxim\u00e1ln\u011b zjednodu\u0161it a vyu\u017e\u00edt v\u00fdhodu Ansible - modularitu. Na nejvy\u0161\u0161\u00ed \u00farovni jsou playbooky, kter\u00e9 m\u016f\u017ee ps\u00e1t jak\u00fdkoli administr\u00e1tor, extern\u00ed v\u00fdvoj\u00e1\u0159, kter\u00fd n\u011bco v\u00ed o Ansible.<\/p>\n<pre><code class=\"plaintext\">- name: Blink disk\n  become: True\n  register: locate_action\n  disk_locate:\n      locate: '{{ locate }}'\n      devname: '{{ devname }}'\n      ids: '{{ locate_ids | default(pd_id) | default(omit) }}'\n<\/code><\/pre>\n<p>Pokud je n\u011bjakou logiku t\u011b\u017ek\u00e9 realizovat v playbook\u00e1ch, p\u0159esouv\u00e1me ji do Ansible modulu nebo filtru. Skripty mohou b\u00fdt ps\u00e1ny bu\u010f v Pythonu, nebo v jak\u00e9mkoli jin\u00e9m jazyce. <\/p>\n<p>Jsou snadn\u00e9 a rychl\u00e9 na psan\u00ed. Nap\u0159\u00edklad modul pro blik\u00e1n\u00ed disku, jeho\u017e p\u0159\u00edklad pou\u017eit\u00ed je uveden v\u00fd\u0161e, se skl\u00e1d\u00e1 z 265 \u0159\u00e1dk\u016f.<\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/08e0384bfad24f3ee61e0643ca087852.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa nejni\u017e\u0161\u00ed \u00farovni je knihovna. Pro tento projekt jsme napsali samostatnou aplikaci, jakousi abstrakci nad hardwarov\u00fdmi a softwarov\u00fdmi RAIDs, kter\u00e9 prov\u00e1d\u011bj\u00ed odpov\u00eddaj\u00edc\u00ed po\u017eadavky. <\/p>\n<p><img decoding=\"async\" alt=\"Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible\" src=\"\/wp-content\/uploads\/2019\/06\/0edda3182026ce6279bd84c99613e3d6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNejvy\u0161\u0161\u00edmi siln\u00fdmi str\u00e1nkami Ansible jsou jednoduchost a srozumiteln\u00e9 playbooky. Domn\u00edv\u00e1m se, \u017ee je t\u0159eba toho vyu\u017e\u00edvat a ne\u017eene generovat zd\u011b\u0161en\u00e9 yaml-soubory a velk\u00e9 mno\u017estv\u00ed podm\u00ednek, shell-k\u00f3du a smy\u010dek.<\/p>\n<p>Pokud chcete zopakovat na\u0161i zku\u0161enost s Ansible API, m\u011bjte na pam\u011bti dv\u011b v\u011bci:<\/p>\n<ul>\n<li>Playbook_executor a v\u016fbec playbook nelze p\u0159edat \u010dasov\u00fd limit. Existuje \u010dasov\u00fd limit na SSH relaci, ale na playbook nen\u00ed. Pokud se pokus\u00edme odpojit disk, kter\u00fd v syst\u00e9mu ji\u017e neexistuje, playbook se bude vykon\u00e1vat nekone\u010dn\u011b, tak\u017ee jsme museli jeho spu\u0161t\u011bn\u00ed zabalit do samostatn\u00e9ho wrapperu a zab\u00edjet podle \u010dasov\u00e9ho limitu.\n<\/li>\n<li>Ansible funguje na principu fork-proces\u016f, proto jeho API nen\u00ed thread-safe. V\u0161echny na\u0161e playbooky spou\u0161t\u00edme jednovl\u00e1knov\u011b.\n<\/li>\n<\/ul>\n<p>\nNakonec se n\u00e1m poda\u0159ilo automatizovat v\u00fdm\u011bnu p\u0159ibli\u017en\u011b 80 % disk\u016f. Celkov\u00e1 rychlost v\u00fdm\u011bny se zdvojn\u00e1sobila. Dnes administr\u00e1tor pouze sleduje incident a rozhoduje, zda je pot\u0159eba disk vym\u011bnit, a pot\u00e9 provede jeden klik.<\/p>\n<p>Ale nyn\u00ed se za\u010d\u00edn\u00e1me pot\u00fdkat s jin\u00fdm probl\u00e9mem: n\u011bkte\u0159\u00ed nov\u00ed administr\u00e1to\u0159i nev\u011bd\u00ed, jak m\u011bnit disky. \ud83d\ude42<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/452110\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432\u0435\u0434\u0443\u0449\u0438\u043c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u043e\u043c \u0432 \u041e\u041a \u0438 \u043e\u0442\u0432\u0435\u0447\u0430\u044e \u0437\u0430 \u0441\u0442\u0430\u0431\u0438\u043b\u044c\u043d\u0443\u044e \u0440\u0430\u0431\u043e\u0442\u0443 \u043f\u043e\u0440\u0442\u0430\u043b\u0430. \u0425\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u044b \u0432\u044b\u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0439 \u0437\u0430\u043c\u0435\u043d\u044b \u0434\u0438\u0441\u043a\u043e\u0432, \u0430 \u0437\u0430\u0442\u0435\u043c, \u043a\u0430\u043a \u0438\u0441\u043a\u043b\u044e\u0447\u0438\u043b\u0438 \u0438\u0437 \u044d\u0442\u043e\u0433\u043e \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0430 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u0430 \u0438 \u0437\u0430\u043c\u0435\u043d\u0438\u043b\u0438 \u0435\u0433\u043e \u0431\u043e\u0442\u043e\u043c. \u042d\u0442\u0430 \u0441\u0442\u0430\u0442\u044c\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u0432\u043e\u0435\u0433\u043e \u0440\u043e\u0434\u0430 \u0442\u0440\u0430\u043d\u0441\u043b\u0438\u0442\u0435\u0440\u0430\u0446\u0438\u0435\u0439 \u0432\u044b\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u044f \u043d\u0430 HighLoad+ 2018 \u041f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u0435 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0430 \u043f\u043e \u0437\u0430\u043c\u0435\u043d\u0435 \u0434\u0438\u0441\u043a\u043e\u0432 \u0421\u043d\u0430\u0447\u0430\u043b\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26233,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-34841","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0410\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u044f \u0437\u0430\u043c\u0435\u043d\u044b \u0434\u0438\u0441\u043a\u043e\u0432 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Ansible | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:00:43+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:00:43+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Automatizace v\u00fdm\u011bny disk\u016f pomoc\u00ed Ansible | ProHoster","description":"Ahoj v\u0161ichni.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u044f \u0437\u0430\u043c\u0435\u043d\u044b \u0434\u0438\u0441\u043a\u043e\u0432 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e Ansible | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/avtomatizatsiya-zameny-diskov-s-pomoshhyu-ansible","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:00:43+00:00","article:modified_time":"2019-10-31T19:00:43+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"34841","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 20:48:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:14:05","updated":"2026-01-21 20:48:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/34841","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=34841"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/34841\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/26233"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=34841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=34841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=34841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}