{"id":80031,"date":"2020-05-02T13:42:49","date_gmt":"2020-05-02T11:42:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/top-fakapov-czian"},"modified":"2020-05-02T13:42:49","modified_gmt":"2020-05-02T11:42:49","slug":"top-fakapov-czian","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/top-fakapov-czian","title":{"rendered":"Top chyby Cian","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Top chyby Cian\" src=\"\/wp-content\/uploads\/2020\/05\/61cf8d25f1f3e858a3b9541cb026cf3c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u0161em dobro!\u00a0<\/p>\n<p>Jmenuji se Nikita, jsem vedouc\u00ed t\u00fdmu in\u017een\u00fdr\u016f v Cian. Jednou z m\u00fdch povinnost\u00ed ve spole\u010dnosti je sn\u00ed\u017eit po\u010det incident\u016f souvisej\u00edc\u00edch s infrastrukturou v produkci na nulu.<br \/>\nT\u00e9ma, o kter\u00e9m se bude nyn\u00ed hovo\u0159it, n\u00e1m p\u0159ineslo mnoho bolesti, a c\u00edlem tohoto \u010dl\u00e1nku je zabr\u00e1nit ostatn\u00edm lidem opakovat na\u0161e chyby nebo alespo\u0148 minimalizovat jejich dopad.\u00a0<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3>\u00davod<\/h3>\n<p>\nKdy\u017e bylo Cian zalo\u017eeno jako monolit a \u017e\u00e1dn\u00e9 n\u00e1znaky mikroslu\u017eeb je\u0161t\u011b neexistovaly, m\u011b\u0159ili jsme dostupnost prost\u0159ednictv\u00edm kontroly 3\u20135 str\u00e1nek.\u00a0<\/p>\n<p>Pokud odpov\u00eddaj\u00ed \u2014 je v\u0161e v po\u0159\u00e1dku, pokud neodpov\u00eddaj\u00ed po del\u0161\u00ed dobu \u2014 spou\u0161t\u00ed se upozorn\u011bn\u00ed. Jak dlouho mus\u00ed nefungovat, aby to bylo pova\u017eov\u00e1no za incident, rozhodovali lid\u00e9 na sch\u016fzk\u00e1ch. T\u00fdm in\u017een\u00fdr\u016f v\u017edy \u00fa\u010dinkoval v vy\u0161et\u0159ov\u00e1n\u00ed incidentu. Po dokon\u010den\u00ed vy\u0161et\u0159ov\u00e1n\u00ed jsme psali postmortem \u2014 jak\u00fdsi report na e-mail ve form\u00e1tu: co se stalo, jak dlouho to trvalo, co jsme ud\u011blali v tu chv\u00edli a co ud\u011bl\u00e1me v budoucnu.\u00a0<\/p>\n<h3>Hlavn\u00ed str\u00e1nky webu nebo jak ch\u00e1peme, \u017ee jsme dos\u00e1hli dna<\/h3>\n<p>\u00a0<br \/>\nAbychom mohli n\u011bjak\u00fdm zp\u016fsobem ch\u00e1pat prioritu chyby, vymezili jsme nejkriti\u010dt\u011bj\u0161\u00ed str\u00e1nky webu pro obchodn\u00ed funkce. Podle nich po\u010d\u00edt\u00e1me po\u010det \u00fasp\u011b\u0161n\u00fdch\/neud\u011blan\u00fdch po\u017eadavk\u016f a \u010dasov\u00fdch limit\u016f. Takto m\u011b\u0159\u00edme uptime.\u00a0<\/p>\n<p>P\u0159edstavme si, \u017ee jsme zjistili, \u017ee existuje \u0159ada superd\u016fle\u017eit\u00fdch sekc\u00ed webu, kter\u00e9 zaji\u0161\u0165uj\u00ed hlavn\u00ed slu\u017ebu \u2014 vyhled\u00e1v\u00e1n\u00ed a pod\u00e1v\u00e1n\u00ed inzer\u00e1t\u016f. Pokud po\u010det po\u017eadavk\u016f, kter\u00e9 skon\u010dily chybou, p\u0159ekro\u010d\u00ed 1 %, je to kritick\u00fd incident. Pokud b\u011bhem 15 minut v hlavn\u00edm \u010dase procento chyb p\u0159ekro\u010d\u00ed 0,1 %, pova\u017euje se to tak\u00e9 za kritick\u00fd incident. Tato krit\u00e9ria pokr\u00fdvaj\u00ed v\u011bt\u0161inu incident\u016f, ostatn\u00ed p\u0159esahuj\u00ed r\u00e1mec tohoto \u010dl\u00e1nku.<\/p>\n<p><img decoding=\"async\" alt=\"Top chyby Cian\" src=\"\/wp-content\/uploads\/2020\/05\/f5d76b1784c9a52ae1dc4728f1509ccc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Top nejlep\u0161\u00edch incident\u016f Cian<\/h3>\n<p>\nTak\u017ee jsme se ur\u010dit\u011b nau\u010dili ur\u010dit, \u017ee incident se stal.\u00a0<\/p>\n<p>Nyn\u00ed je ka\u017ed\u00fd incident detailn\u011b pops\u00e1n a zaznamen\u00e1n v epiku Jira. Mimochodem: pro to jsme zalo\u017eili samostatn\u00fd projekt, nazvali jsme ho FAIL \u2014 v n\u011bm lze vytv\u00e1\u0159et pouze epiky.\u00a0<\/p>\n<p>Pokud shrom\u00e1\u017ed\u00edme v\u0161echny chyby za posledn\u00edch n\u011bkolik let, vedou:\u00a0<\/p>\n<ul>\n<li>incidenty spojen\u00e9 s mssql;<\/li>\n<li>incidenty zp\u016fsoben\u00e9 vn\u011bj\u0161\u00edmi faktory;<\/li>\n<li>chyby administr\u00e1tor\u016f.<\/li>\n<\/ul>\n<p>\nPodrobn\u011bji se zastav\u00edme u chyb administr\u00e1tor\u016f a tak\u00e9 u dal\u0161\u00edch zaj\u00edmav\u00fdch selh\u00e1n\u00ed.<\/p>\n<h4>P\u00e1t\u00e9 m\u00edsto \u2014 \u201eUkl\u00edz\u00edme v DNS\u201c<\/h4>\n<p>\nByla to de\u0161tiv\u00e1 \u00fater\u00fd. Rozhodli jsme se uklidit v DNS clusteru.\u00a0<\/p>\n<p>Chcelo se n\u00e1m p\u0159ej\u00edt intern\u00ed dns servery z bind na powerdns, p\u0159i\u010dem\u017e jsme vyhradili zcela samostatn\u00e9 servery, kde krom\u011b dns nic nen\u00ed.\u00a0<\/p>\n<p>Rozm\u00edstili jsme po jednom dns serveru v ka\u017ed\u00e9 lokalit\u011b na\u0161ich datov\u00fdch center a nastal \u010das p\u0159esunout zony z bind do powerdns a p\u0159epnout infrastrukturu na nov\u00e9 servery.\u00a0<\/p>\n<p>V nejv\u011bt\u0161\u00edm rozmachu p\u0159esunu z\u016fstala ze v\u0161ech <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/server\/\"   title=\"servery\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1484\">servery<\/a>, kter\u00e9 byly uvedeny v lok\u00e1ln\u00edch cache bind na v\u0161ech serverech, pouze jedna, kter\u00e1 byla v datov\u00e9m centru v Petrohrad\u011b. Toto datov\u00e9 centrum bylo p\u016fvodn\u011b deklarov\u00e1no jako pro n\u00e1s nekritick\u00e9, ale n\u00e1hle se stalo single point of failure.<br \/>\nPr\u00e1v\u011b v takov\u00e9m obdob\u00ed p\u0159esunu spadl kan\u00e1l mezi Moskvou a Petrohradem. Fakticky jsme z\u016fstali bez DNS na p\u011bt minut a znovu jsme se dostali online, kdy\u017e <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/\"   title=\"hoster\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1206\">hoster<\/a> prov\u00e1d\u011bl opravy.\u00a0<\/p>\n<p><b>Z\u00e1v\u011bry: <\/b><\/p>\n<p>Pokud jsme d\u0159\u00edve zanedb\u00e1vali vn\u011bj\u0161\u00ed faktory b\u011bhem p\u0159\u00edpravy na pr\u00e1ce, nyn\u00ed jsou tak\u00e9 zahrnuty do seznamu toho, na co se p\u0159ipravujeme. A nyn\u00ed se sna\u017e\u00edme, aby byly v\u0161echny komponenty rezervov\u00e1ny n-2, a na dobu prac\u00ed m\u016f\u017eeme sni\u017eovat tuto \u00farove\u0148 na n-1.<\/p>\n<ul>\n<li>P\u0159i sestavov\u00e1n\u00ed pl\u00e1nu \u010dinnost\u00ed zaznamen\u00e1vejte body, kde m\u016f\u017ee slu\u017eba selhat, a prom\u00fd\u0161lejte sc\u00e9n\u00e1\u0159, kde v\u0161e \u0161lo \"h\u016f\u0159e nen\u00ed\" p\u0159edem.<\/li>\n<li>Rozd\u011blujte intern\u00ed dns servery do r\u016fzn\u00fdch geografick\u00fdch lokalit\/datov\u00fdch center\/rack\u016f\/p\u0159ep\u00edna\u010d\u016f\/vstup\u016f.<\/li>\n<li>Na ka\u017ed\u00e9m serveru nainstalujte lok\u00e1ln\u00ed cache dns server, kter\u00fd p\u0159esm\u011brov\u00e1v\u00e1 dotazy na hlavn\u00ed dns servery, a v p\u0159\u00edpad\u011b jeho nedostupnosti odpov\u00edd\u00e1 z cache.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>\u010ctvrt\u00e9 m\u00edsto \u2014 \u201eUklid\u00edme v Nginxu\u201c<\/h4>\n<p>\nJednoho kr\u00e1sn\u00e9ho dne se n\u00e1\u0161 t\u00fdm rozhodl, \u017ee \u201edost bylo trp\u011bn\u00ed\u201c, a spustil se proces refaktorov\u00e1n\u00ed konfigurac\u00ed nginx. Hlavn\u00edm c\u00edlem je p\u0159iv\u00e9st konfigurace k intuitivn\u011b srozumiteln\u00e9 struktu\u0159e. D\u0159\u00edve bylo v\u0161e \u201ehistoricky dan\u00e9\u201c a neneslo v sob\u011b \u017e\u00e1dnou logiku. Nyn\u00ed jsme ka\u017ed\u00fd server_name vyjmuli do jednoho pojmenovan\u00e9ho souboru a rozlo\u017eili v\u0161echny konfigurace do slo\u017eek. Mimochodem \u2014 konfigurace obsahuje 253949 \u0159\u00e1dk\u016f nebo 7836520 znak\u016f a zab\u00edr\u00e1 t\u00e9m\u011b\u0159 7 megabajt\u016f. Horn\u00ed \u00farove\u0148 struktury:\u00a0<\/p>\n<p>                        <b class=\"spoiler_title\">Struktura Nginxu<\/b><\/p>\n<pre><code class=\"plaintext\">\u251c\u2500\u2500 access\n\u2502 \u00a0 \u251c\u2500\u2500 allow.list\n...\n\u2502 \u00a0 \u2514\u2500\u2500 whitelist.conf\n\u251c\u2500\u2500 geobase\n\u2502 \u00a0 \u251c\u2500\u2500 exclude.conf\n...\n\u2502 \u00a0 \u2514\u2500\u2500 geo_ip_to_region_id.conf\n\u251c\u2500\u2500 geodb\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP.dat\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP2-Country.mmdb\n\u2502 \u00a0 \u2514\u2500\u2500 GeoLiteCity.dat\n\u251c\u2500\u2500 inc\n\u2502 \u00a0 \u251c\u2500\u2500 error.inc\n...\n\u2502 \u00a0 \u2514\u2500\u2500 proxy.inc\n\u251c\u2500\u2500 lists.d\n\u2502 \u00a0 \u251c\u2500\u2500 bot.conf\n...\n\u2502 \u00a0 \u251c\u2500\u2500 dynamic\n\u2502 \u00a0 \u2514\u2500\u2500 geo.conf\n\u251c\u2500\u2500 lua\n\u2502 \u00a0 \u251c\u2500\u2500 cookie.lua\n\u2502 \u00a0 \u251c\u2500\u2500 log\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 log.lua\n\u2502 \u00a0 \u251c\u2500\u2500 logics\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 include.lua\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 utils.lua\n\u2502 \u00a0 \u2514\u2500\u2500 prom\n\u2502 \u00a0 \u00a0 \u00a0 \u251c\u2500\u2500 stats.lua\n\u2502 \u00a0 \u00a0 \u00a0 \u2514\u2500\u2500 stats_prometheus.lua\n\u251c\u2500\u2500 map.d\n\u2502 \u00a0 \u251c\u2500\u2500 access.conf\n\u2502 \u00a0 \u251c\u2500\u2500 ..\u00a0\n\u2502 \u00a0 \u2514\u2500\u2500 zones.conf\n\u251c\u2500\u2500 nginx.conf\n\u251c\u2500\u2500 robots.txt\n\u251c\u2500\u2500 server.d\n\u2502 \u00a0 \u251c\u2500\u2500 cian.ru\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 cian.ru.conf\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 my.cian.ru.conf\n\u251c\u2500\u2500 service.d\n\u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2514\u2500\u2500 status.conf\n\u2514\u2500\u2500 upstream.d\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 cian-mcs.conf\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 ...\n\u00a0\u00a0\u00a0\u00a0\u2514\u2500\u2500 wafserver.conf<\/code><\/pre>\n<p>Bylo to zna\u010dn\u011b lep\u0161\u00ed, ale b\u011bhem p\u0159ejmenov\u00e1v\u00e1n\u00ed a rozd\u011blov\u00e1n\u00ed konfigurac\u00ed m\u011bly n\u011bkter\u00e9 z nich nespr\u00e1vnou p\u0159\u00edponu a nedostaly se do direktivy include *.conf. V d\u016fsledku toho se n\u011bkter\u00e9 hosty staly nedostupn\u00fdmi a vracely 301 na hlavn\u00ed str\u00e1nku. Proto\u017ee k\u00f3d odpov\u011bdi nebyl 5xx\/4xx, nebylo to hned vid\u011bt, ale a\u017e brzy r\u00e1no. Po tomto jsme za\u010dali ps\u00e1t testy na kontrolu infrastrukturn\u00edch komponent\u016f.<\/p>\n<p><b>Z\u00e1v\u011bry:<\/b>\u00a0<\/p>\n<ul>\n<li>Spr\u00e1vn\u011b strukturovat konfigurace (nejen nginx) a prom\u00fd\u0161let strukturu u\u017e v ran\u00e9 f\u00e1zi projektu. Tak je u\u010din\u00edte srozumiteln\u011bj\u0161\u00edmi pro t\u00fdm, co\u017e zase sn\u00ed\u017e\u00ed TTM.<\/li>\n<li>Pro n\u011bkter\u00e9 infrastrukturn\u00ed komponenty pi\u0161te testy. Nap\u0159\u00edklad: kontrola, \u017ee v\u0161echny kl\u00ed\u010dov\u00e9 server_name vracej\u00ed spr\u00e1vn\u00fd status a + t\u011blo odpov\u011bdi. Sta\u010d\u00ed m\u00edt po ruce jen n\u011bkolik skript\u016f, kter\u00e9 ov\u011b\u0159uj\u00ed z\u00e1kladn\u00ed funkce komponentu, abychom si v 3 r\u00e1no nevybavovali, co je\u0161t\u011b je t\u0159eba zkontrolovat.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>T\u0159et\u00ed m\u00edsto \u2014 \u201eNajednou do\u0161el prostor v Cassandra\u201c<\/h4>\n<p>\nData postupn\u011b rostla a v\u0161e bylo v po\u0159\u00e1dku, dokud v clusteru Cassandra neza\u010daly selh\u00e1vat opravy velk\u00fdch keyspace, proto\u017ee na n\u011b nem\u016f\u017ee prob\u011bhnout compaction.\u00a0<\/p>\n<p>Jednoho de\u0161tiv\u00e9ho dne se cluster t\u00e9m\u011b\u0159 prom\u011bnil v d\u00fdni a sice:<\/p>\n<ul>\n<li>v clusteru zb\u00fdvalo p\u0159ibli\u017en\u011b 20 % celkov\u00e9ho prostoru;<\/li>\n<li>nelze pln\u011b p\u0159idat nody, proto\u017ee po p\u0159id\u00e1n\u00ed nody neprob\u011bhne cleanup pro nedostatek m\u00edsta na odd\u00edlech;<\/li>\n<li>v\u00fdkon postupn\u011b kles\u00e1, proto\u017ee nefunguje compaction;\u00a0<\/li>\n<li>cluster pracuje v havarijn\u00edm re\u017eimu.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Top chyby Cian\" src=\"\/wp-content\/uploads\/2020\/05\/1d036fbc500a49a285fa4b0014e2a70d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdstup \u2013 p\u0159idali jsme dal\u0161\u00edch 5 uzl\u016f bez \u00fadr\u017eby, po \u010dem\u017e jsme za\u010dali systematicky vy\u0159azovat z clusteru a znovu p\u0159id\u00e1vat jako pr\u00e1zdn\u00e9 uzly, na kter\u00fdch do\u0161lo m\u00edsto. \u010casu jsme str\u00e1vili mnohem v\u00edce, ne\u017e bychom si p\u0159\u00e1li. Existovalo riziko \u010d\u00e1ste\u010dn\u00e9 nebo \u00fapln\u00e9 nedostupnosti clusteru.\u00a0<\/p>\n<p><b>Z\u00e1v\u011bry:<\/b><\/p>\n<ul>\n<li>Na v\u0161ech serverech Cassandra by nem\u011blo b\u00fdt obsazeno v\u00edce ne\u017e 60 % m\u00edsta na ka\u017ed\u00e9m odd\u00edlu.\u00a0<\/li>\n<li>M\u011bly by b\u00fdt zat\u00ed\u017eeny maxim\u00e1ln\u011b na 50 % podle CPU.<\/li>\n<li>Nem\u011bli bychom zanedb\u00e1vat pl\u00e1nov\u00e1n\u00ed kapacity a je t\u0159eba ho promyslet pro ka\u017edou komponentu, vych\u00e1zej\u00edc\u00ed z jej\u00ed specifiky.<\/li>\n<li>\u010c\u00edm v\u00edce uzl\u016f v clusteru, t\u00edm l\u00e9pe. Servery obsahuj\u00edc\u00ed men\u0161\u00ed objem dat se rychleji p\u0159en\u00e1\u0161\u00ed a takov\u00fd cluster se snadn\u011bji obnovuje.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Druh\u00e9 m\u00edsto \u2013 \u201eData zmizela z \u00falo\u017ei\u0161t\u011b konzolu key-value\u201c<\/h4>\n<p>\nPro objevov\u00e1n\u00ed slu\u017eeb pou\u017e\u00edv\u00e1me consul stejn\u011b jako mnoz\u00ed ostatn\u00ed. Ale na\u0161e key-value se vyu\u017e\u00edv\u00e1 tak\u00e9 pro blue-green nasazen\u00ed monolitu. Tam se uchov\u00e1v\u00e1 informace o aktivn\u00edch a neaktivn\u00edch upstream, kter\u00e9 se b\u011bhem nasazen\u00ed m\u011bn\u00ed. Byl naps\u00e1n nasazovac\u00ed servis, kter\u00fd komunikoval s KV. V ur\u010dit\u00e9m okam\u017eiku data z KV zmizela. Obnovili jsme je podle pam\u011bti, ale s \u0159adou chyb. V d\u016fsledku toho se p\u0159i nasazen\u00ed z\u00e1t\u011b\u017e na upstreamy rozd\u011blila nerovnom\u011brn\u011b a obdr\u017eeli jsme mnoho chyb 502 z d\u016fvodu p\u0159et\u00ed\u017een\u00ed backend\u016f podle CPU. Nakonec jsme p\u0159e\u0161li z consul KV na postgres, odkud je ji\u017e nelze tak snadno odstranit.\u00a0\u00a0<\/p>\n<p><b>Z\u00e1v\u011bry:<br \/>\n<\/b><\/p>\n<ul>\n<li>Slu\u017eby bez jak\u00e9koli autorizace by nem\u011bly obsahovat kritick\u00e1 data pro fungov\u00e1n\u00ed webu. Nap\u0159\u00edklad, pokud nem\u00e1te autorizaci v ES \u2013 bylo by lep\u0161\u00ed zak\u00e1zat p\u0159\u00edstup na \u00farovni s\u00edt\u011b ze v\u0161ech m\u00edst, kde nen\u00ed pot\u0159eba, ponechat pouze nezbytn\u00e9, a tak\u00e9 nastavit action.destructive_requires_name: true.<\/li>\n<li>Vypracujte mechanismus z\u00e1lohov\u00e1n\u00ed a obnovy p\u0159edem. Nap\u0159\u00edklad si p\u0159edem vytvo\u0159te skript (nap\u0159\u00edklad na python), kter\u00fd um\u00ed z\u00e1lohovat i obnovovat.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Prvn\u00ed m\u00edsto \u2013 \u201eKapit\u00e1n o\u010dividnost\u201c\u00a0<\/h4>\n<p>\nV ur\u010dit\u00e9m okam\u017eiku jsme si v\u0161imli nerovnom\u011brn\u00e9ho rozlo\u017een\u00ed z\u00e1t\u011b\u017ee na upstreamy nginx v p\u0159\u00edpadech, kdy bylo v backendu v\u00edce ne\u017e 10 server\u016f. Proto\u017ee round-robin sm\u011broval po\u017eadavky od prvn\u00edho k posledn\u00edmu upstreamu v po\u0159ad\u00ed a ka\u017ed\u00fd reload nginx za\u010d\u00ednal znovu, na prvn\u00ed upstreamy v\u017edy p\u0159ipadalo v\u00edce po\u017eadavk\u016f ne\u017e na ostatn\u00ed. V d\u016fsledku toho pracovaly pomaleji a cel\u00fd web t\u00edm trp\u011bl. To bylo \u010d\u00edm d\u00e1l t\u00edm v\u00edce patrn\u00e9 s rostouc\u00edm mno\u017estv\u00edm provozu. Prost\u011b aktualizovat nginx pro zahrnut\u00ed random nesta\u010dilo \u2013 bylo pot\u0159eba p\u0159epracovat spoustu lua k\u00f3du, kter\u00fd na verzi 1.15 nefungoval. Museli jsme patchovat n\u00e1\u0161 nginx 1.14.2, abychom do n\u011bj implementovali podporu random. To vy\u0159e\u0161ilo probl\u00e9m. Tato chyba je v\u00edt\u011bzem v kategorii \u201ekapit\u00e1n neviditelnost\u201c.<\/p>\n<p><b>Z\u00e1v\u011bry:<\/b><\/p>\n<p>Bylo to velmi zaj\u00edmav\u00e9 a vzru\u0161uj\u00edc\u00ed zkoumat tuto chybu).\u00a0<\/p>\n<ul>\n<li>Nastavte monitorov\u00e1n\u00ed tak, aby pom\u00e1halo rychle naj\u00edt podobn\u00e9 fluktuace. Nap\u0159\u00edklad m\u016f\u017eete pou\u017e\u00edt ELK k sledov\u00e1n\u00ed rps pro ka\u017ed\u00fd backend ka\u017ed\u00e9ho upstreamu a sledovat jejich dobu odezvy z pohledu nginx. V tomto p\u0159\u00edpad\u011b n\u00e1m to pomohlo odhalit probl\u00e9m.\u00a0<\/li>\n<\/ul>\n<p>\nV\u011bt\u0161inu selh\u00e1n\u00ed by bylo mo\u017en\u00e9 p\u0159edej\u00edt p\u0159i pe\u010dliv\u011bj\u0161\u00edm p\u0159\u00edstupu k tomu, co d\u011bl\u00e1te. Mus\u00edte si v\u017edy pamatovat Murphyho z\u00e1kon:\u00a0<i>V\u0161e, co m\u016f\u017ee j\u00edt \u0161patn\u011b, p\u016fjde \u0161patn\u011b, <\/i>a stav\u011bt komponenty podle n\u011bj.\u00a0<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/cian\/blog\/499542\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430!\u00a0 \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d. \u041e\u0434\u043d\u043e\u0439 \u0438\u0437 \u043c\u043e\u0438\u0445 \u043e\u0431\u044f\u0437\u0430\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u0430 \u0438\u043d\u0446\u0438\u0434\u0435\u043d\u0442\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439 \u043d\u0430 \u043f\u0440\u043e\u0434\u0435, \u0434\u043e \u043d\u0443\u043b\u044f. \u0422\u043e, \u043e \u0447\u0435\u043c \u043f\u043e\u0439\u0434\u0435\u0442 \u0440\u0435\u0447\u044c \u0434\u0430\u043b\u0435\u0435, \u043f\u0440\u0438\u043d\u0435\u0441\u043b\u043e \u043d\u0430\u043c \u043c\u043d\u043e\u0433\u043e \u0431\u043e\u043b\u0438, \u0438 \u0446\u0435\u043b\u044c \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u2014 \u043d\u0435 \u0434\u0430\u0442\u044c \u0434\u0440\u0443\u0433\u0438\u043c \u043b\u044e\u0434\u044f\u043c \u043f\u043e\u0432\u0442\u043e\u0440\u0438\u0442\u044c \u043d\u0430\u0448\u0438\u0445 \u043e\u0448\u0438\u0431\u043e\u043a \u0438\u043b\u0438 \u0445\u043e\u0442\u044f \u0431\u044b \u043c\u0438\u043d\u0438\u043c\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438\u0445 \u0432\u043b\u0438\u044f\u043d\u0438\u0435.\u00a0 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80032,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Top chyb Cian | ProHoster","description":"V\u0161em hodn\u011b \u0161t\u011bst\u00ed! Jmenuji se Nikita, jsem t\u00fdmov\u00fd vedouc\u00ed t\u00fdmu in\u017een\u00fdr\u016f Cian.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/top-fakapov-czian","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/top-fakapov-czian","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-02T11:42:49+00:00","article:modified_time":"2020-05-02T11:42:49+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80031","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:47:44","updated":"2026-02-09 16:50:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/80031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=80031"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/80031\/revisions"}],"predecessor-version":[{"id":158728,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/80031\/revisions\/158728"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/80032"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=80031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=80031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=80031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}