{"id":80031,"date":"2020-05-02T13:42:49","date_gmt":"2020-05-02T11:42:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/top-fakapov-czian"},"modified":"2020-05-02T13:42:49","modified_gmt":"2020-05-02T11:42:49","slug":"top-fakapov-czian","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/top-fakapov-czian","title":{"rendered":"Top faux pas CIAN","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Top faux pas CIAN\" src=\"\/wp-content\/uploads\/2020\/05\/61cf8d25f1f3e858a3b9541cb026cf3c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u0161etk\u00fdm prajem dobre!\u00a0<\/p>\n<p>Vol\u00e1m sa Nikita, som t\u00edmov\u00fd ved\u00faci t\u00edmu in\u017einierov Cian. Jednou z mojich \u00faloh v spolo\u010dnosti je zn\u00ed\u017ei\u0165 po\u010det incidentov s\u00favisiacich s infra\u0161trukt\u00farou na produkcii na nulu.<br \/>\nTo, o \u010dom sa bude hovori\u0165 \u010falej, n\u00e1m prinieslo ve\u013ea bolesti a cie\u013eom tohto \u010dl\u00e1nku je zabr\u00e1ni\u0165 in\u00fdm \u013eu\u010fom opakova\u0165 na\u0161e chyby alebo aspo\u0148 minimalizova\u0165 ich vplyv.\u00a0<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3>Pre\u00e1mbula<\/h3>\n<p>\nKedysi d\u00e1vno, ke\u010f Cian pozost\u00e1val z monolitov a o mikroslu\u017eb\u00e1ch sa e\u0161te ani len neuva\u017eovalo, sme merali dostupnos\u0165 zdroja kontrolou 3\u20135 str\u00e1nok.\u00a0<\/p>\n<p>Odpovedaj\u00fa \u2014 v\u0161etko je v poriadku, neodpovedaj\u00fa dlh\u0161\u00ed \u010das \u2014 alert. Ko\u013eko \u010dasu musia nepracova\u0165, aby to bolo pova\u017eovan\u00e9 za incident, rozhodovali \u013eudia na porad\u00e1ch. T\u00edm in\u017einierov sa v\u017edy z\u00fa\u010dast\u0148oval na vy\u0161etrovan\u00ed incidentu. Ke\u010f bolo vy\u0161etrovanie ukon\u010den\u00e9, nap\u00edsali sme post mortem \u2014 ak\u00fdsi report posielan\u00fd e-mailom v form\u00e1te: \u010do sa stalo, ako dlho to trvalo, \u010do sme robili v danej chv\u00edli a \u010do urob\u00edme v bud\u00facnosti.\u00a0<\/p>\n<h3>Hlavn\u00e9 str\u00e1nky webu alebo ako vieme, \u017ee sme udreli na dno<\/h3>\n<p>\u00a0<br \/>\nAby sme mohli nejako pos\u00fadi\u0165 prioritu chyby, vy\u010dlenili sme najkritickej\u0161ie str\u00e1nky webu pre obchodn\u00e9 funkcie. Na nich po\u010d\u00edtame po\u010det \u00faspe\u0161n\u00fdch\/ne\u00faspe\u0161n\u00fdch po\u017eiadaviek a \u010dasov\u00fdch limitov. T\u00fdmto sp\u00f4sobom meriame uptime.\u00a0<\/p>\n<p>Povedzme, \u017ee sme zistili, \u017ee existuje nieko\u013eko superd\u00f4le\u017eit\u00fdch sekci\u00ed webu, ktor\u00e9 s\u00fa zodpovedn\u00e9 za hlavn\u00fd servis \u2014 vyh\u013ead\u00e1vanie a pod\u00e1vanie inzer\u00e1tov. Ak po\u010det po\u017eiadaviek, ktor\u00e9 skon\u010dili chybou, presiahne 1%, je to kritick\u00fd incident. Ak sa v priebehu 15 min\u00fat v peak \u010dase percento ch\u00fdb presiahne 0,1%, aj to sa pova\u017euje za kritick\u00fd incident. Tieto krit\u00e9ri\u00e1 pokr\u00fdvaj\u00fa v\u00e4\u010d\u0161inu incidentov, ostatn\u00e9 presahuj\u00fa r\u00e1mec tohto \u010dl\u00e1nku.<\/p>\n<p><img decoding=\"async\" alt=\"Top faux pas CIAN\" src=\"\/wp-content\/uploads\/2020\/05\/f5d76b1784c9a52ae1dc4728f1509ccc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Top najlep\u0161\u00edch incidentov Cian<\/h3>\n<p>\nTak\u017ee sme sa ur\u010dite nau\u010dili ur\u010di\u0165, \u017ee incident sa stal.\u00a0<\/p>\n<p>Teraz je ka\u017ed\u00fd incident u n\u00e1s podrobne op\u00edsan\u00fd a zazna\u010den\u00fd v epike Jira. Mimochodom, na to sme zalo\u017eili samostatn\u00fd projekt, nazvali sme ho FAIL \u2014 v \u0148om je mo\u017en\u00e9 vytv\u00e1ra\u0165 len epiky.\u00a0<\/p>\n<p>Ak by sme zhroma\u017edili v\u0161etky ne\u00faspechy za posledn\u00e9 roky, najviac by prevl\u00e1dali:\u00a0<\/p>\n<ul>\n<li>incidenty s\u00favisiace s mssql;<\/li>\n<li>incidenty sp\u00f4soben\u00e9 vonkaj\u0161\u00edmi faktormi;<\/li>\n<li>chyby administr\u00e1tora.<\/li>\n<\/ul>\n<p>\nZamerajme sa podrobnej\u0161ie na chyby administr\u00e1torov, ako aj na niektor\u00e9 \u010fal\u0161ie zauj\u00edmav\u00e9 ne\u00faspechy.<\/p>\n<h4>Piate miesto \u2014 \u201eUpratujeme v DNS\u201c<\/h4>\n<p>\nBol to pochm\u00farny utorok. Rozhodli sme sa uprata\u0165 v DNS klastri.\u00a0<\/p>\n<p>Chceli sme prejs\u0165 na intern\u00e9 DNS servery z bind na powerdns, pri\u010dom sme vyhradili \u00faplne samostatn\u00e9 servery, na ktor\u00fdch okrem DNS ni\u010d nie je.\u00a0<\/p>\n<p>Um\u00edstili jsme po jednom DNS serveru na ka\u017ed\u00e9 miesto na\u0161ich d\u00e1tov\u00fdch centier a nastal \u010das presun\u00fa\u0165 z\u00f3ny z bind do powerdns a prepn\u00fa\u0165 infra\u0161trukt\u00faru na nov\u00e9 servery.\u00a0<\/p>\n<p>V najv\u00e4\u010d\u0161ej chv\u00edli presunu zostal zo v\u0161etk\u00fdch <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sk\/server\/\"   title=\"serverov\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1484\">serverov<\/a>, ktor\u00e9 boli uveden\u00e9 v lok\u00e1lnych cache-och bind na v\u0161etk\u00fdch serveroch, len jeden, ktor\u00fd bol v d\u00e1tovom centre v Petrohrade. Toto DC bolo p\u00f4vodne deklarovan\u00e9 ako pre n\u00e1s nekritick\u00e9, no zrazu sa stalo jedin\u00fdm bodom zlyhania.<br \/>\nPr\u00e1ve v takomto obdob\u00ed presunu spadol kan\u00e1l medzi Moskvou a Petrohradom. Fakticky sme zostali bez DNS na p\u00e4\u0165 min\u00fat a obnovili sme sa, ke\u010f <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sk\/\"   title=\"hoster\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1206\">hoster<\/a> odstr\u00e1nil probl\u00e9my.\u00a0<\/p>\n<p><b>V\u00fdvody: <\/b><\/p>\n<p>Ak sme predt\u00fdm zanedb\u00e1vali vonkaj\u0161ie faktory po\u010das pr\u00edpravy na pr\u00e1ce, teraz ich tie\u017e zahrnuli do zoznamu toho, na \u010do sa pripravujeme. A teraz sa sna\u017e\u00edme o to, aby v\u0161etky komponenty boli rezervovan\u00e9 n-2, a po\u010das pr\u00e1c m\u00f4\u017eeme tento \u00farove\u0148 zn\u00ed\u017ei\u0165 na n-1.<\/p>\n<ul>\n<li>Pri vytv\u00e1ran\u00ed ak\u010dn\u00e9ho pl\u00e1nu si v\u0161imnite body, kde m\u00f4\u017ee servis zlyha\u0165, a premyslite si scen\u00e1r, kde v\u0161etko ide \"hor\u0161ie neexistuje\" vopred.<\/li>\n<li>Rozde\u013ete intern\u00e9 DNS servery do r\u00f4znych geografick\u00fdch lokal\u00edt\/d\u00e1tov\u00fdch centier\/stan\u00edc\/prep\u00edna\u010dov\/vstupov.<\/li>\n<li>Na ka\u017edom serveri nain\u0161talujte lok\u00e1lny cache-ovac\u00ed DNS server, ktor\u00fd presmerov\u00e1va po\u017eiadavky na hlavn\u00e9 DNS servery, a v pr\u00edpade jeho nedostupnosti bude odpoveda\u0165 z cache.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>\u0160tvrt\u00e9 miesto - \"Upratujeme v Nginx\"<\/h4>\n<p>\nJedn\u00e9ho kr\u00e1sneho d\u0148a sa n\u00e1\u0161 t\u00edm rozhodol, \u017ee \"sta\u010dilo to zn\u00e1\u0161a\u0165\", a za\u010dal proces refaktoringu konfigur\u00e1ci\u00ed nginx. Hlavn\u00fdm cie\u013eom je prinies\u0165 konfigur\u00e1cie do intuit\u00edvnej \u0161trukt\u00fary. Predt\u00fdm to v\u0161etko bolo \"historicky dan\u00e9\" a neneslo v sebe \u017eiadnu logiku. Teraz sme ka\u017ed\u00fd server_name vytrhli do rovnomenn\u00e9ho s\u00faboru a rozdelili v\u0161etky konf\u00edgy do prie\u010dinkov. Mimochodom, konfigur\u00e1cia obsahuje 253949 riadkov alebo 7836520 znakov a zabera takmer 7 megabajtov. Horn\u00e1 \u00farove\u0148 \u0161trukt\u00fary:\u00a0<\/p>\n<p>                        <b class=\"spoiler_title\">\u0160trukt\u00fara Nginxu<\/b><\/p>\n<pre><code class=\"plaintext\">\u251c\u2500\u2500 pr\u00edstup\n\u2502 \u00a0 \u251c\u2500\u2500 allow.list\n...\n\u2502 \u00a0 \u2514\u2500\u2500 whitelist.conf\n\u251c\u2500\u2500 geobase\n\u2502 \u00a0 \u251c\u2500\u2500 exclude.conf\n...\n\u2502 \u00a0 \u2514\u2500\u2500 geo_ip_to_region_id.conf\n\u251c\u2500\u2500 geodb\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP.dat\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP2-Country.mmdb\n\u2502 \u00a0 \u2514\u2500\u2500 GeoLiteCity.dat\n\u251c\u2500\u2500 inc\n\u2502 \u00a0 \u251c\u2500\u2500 error.inc\n...\n\u2502 \u00a0 \u2514\u2500\u2500 proxy.inc\n\u251c\u2500\u2500 lists.d\n\u2502 \u00a0 \u251c\u2500\u2500 bot.conf\n...\n\u2502 \u00a0 \u251c\u2500\u2500 dynamic\n\u2502 \u00a0 \u2514\u2500\u2500 geo.conf\n\u251c\u2500\u2500 lua\n\u2502 \u00a0 \u251c\u2500\u2500 cookie.lua\n\u2502 \u00a0 \u251c\u2500\u2500 log\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 log.lua\n\u2502 \u00a0 \u251c\u2500\u2500 logics\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 include.lua\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 utils.lua\n\u2502 \u00a0 \u2514\u2500\u2500 prom\n\u2502 \u00a0 \u00a0 \u00a0 \u251c\u2500\u2500 stats.lua\n\u2502 \u00a0 \u00a0 \u00a0 \u2514\u2500\u2500 stats_prometheus.lua\n\u251c\u2500\u2500 map.d\n\u2502 \u00a0 \u251c\u2500\u2500 access.conf\n\u2502 \u00a0 \u251c\u2500\u2500 ..\u00a0\n\u2502 \u00a0 \u2514\u2500\u2500 zones.conf\n\u251c\u2500\u2500 nginx.conf\n\u251c\u2500\u2500 robots.txt\n\u251c\u2500\u2500 server.d\n\u2502 \u00a0 \u251c\u2500\u2500 cian.ru\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 cian.ru.conf\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 my.cian.ru.conf\n\u251c\u2500\u2500 service.d\n\u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2514\u2500\u2500 status.conf\n\u2514\u2500\u2500 upstream.d\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 cian-mcs.conf\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 ...\n\u00a0\u00a0\u00a0\u00a0\u2514\u2500\u2500 wafserver.conf<\/code><\/pre>\n<p>V\u00fdrazne sa to zlep\u0161ilo, ale po\u010das premenovania a rozde\u013eovania konfigur\u00e1ci\u00ed mala \u010das\u0165 z nich nespr\u00e1vne pr\u00edpony a nedostala sa do direkt\u00edvy include *.conf. N\u00e1sledkom toho sa \u010das\u0165 hostite\u013eov stala nedostupn\u00e1 a vracala 301 na hlavn\u00fa str\u00e1nku. Preto\u017ee k\u00f3d odpovede nebol 5xx\/4xx, nev\u0161imli si to hne\u010f, ale a\u017e skoro r\u00e1no. Po tom sme za\u010dali p\u00edsa\u0165 testy na kontrolu infra\u0161trukt\u00farnych komponentov.<\/p>\n<p><b>V\u00fdvody:<\/b>\u00a0<\/p>\n<ul>\n<li>Spr\u00e1vne \u0161trukt\u00farujte konfigur\u00e1cie (nielen nginx) a premyslite \u0161trukt\u00faru v ranom \u0161t\u00e1diu projektu. T\u00fdmto sp\u00f4sobom ich sprav\u00edte zrozumite\u013enej\u0161\u00edmi pre t\u00edm, \u010do zase zn\u00ed\u017ei TTM.<\/li>\n<li>Pre niektor\u00e9 infra\u0161trukt\u00farne komponenty p\u00ed\u0161te testy. Napr\u00edklad: overenie, \u017ee v\u0161etky k\u013e\u00fa\u010dov\u00e9 server_name vracaj\u00fa spr\u00e1vny status, + telo odpovede. Sta\u010d\u00ed ma\u0165 po ruke nieko\u013eko skriptov, ktor\u00e9 kontroluj\u00fa z\u00e1kladn\u00e9 funkcie komponentu, aby ste sa v noci 3:00 nez\u00fafalo sna\u017eili spomen\u00fa\u0165, \u010do e\u0161te treba skontrolova\u0165.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Tretie miesto \u2014 \u201eZrazu sa minulo miesto v Cassandre\u201c<\/h4>\n<p>\nD\u00e1ta r\u00e1stli postupne, a v\u0161etko bolo v poriadku, k\u00fdm v klastre Cassandre neza\u010dali zlyh\u00e1va\u0165 opravy ve\u013ek\u00fdch keyspace, preto\u017ee na nich nem\u00f4\u017ee prebehn\u00fa\u0165 kompakcia.\u00a0<\/p>\n<p>Jedn\u00e9ho pochm\u00farneho d\u0148a sa klaster takmer premenil na tekvicu, a to konkr\u00e9tne:<\/p>\n<ul>\n<li>miesto zostalo pribli\u017ene 20% celkovo v klastri;<\/li>\n<li>nie je mo\u017en\u00e9 plne prida\u0165 uzly, preto\u017ee po pridan\u00ed uzla neprebieha cleanup kv\u00f4li nedostatku miesta na oddieloch;<\/li>\n<li>v\u00fdkon pomaly kles\u00e1, preto\u017ee nefunguje kompakcia;\u00a0<\/li>\n<li>klaster pracuje v havarijnom re\u017eime.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Top faux pas CIAN\" src=\"\/wp-content\/uploads\/2020\/05\/1d036fbc500a49a285fa4b0014e2a70d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOdchod \u2014 pridali sme \u010fal\u0161\u00edch 5 uzlov bez \u00fadr\u017eby, po \u010dom sme za\u010dali postupne odobera\u0165 z clustra a op\u00e4\u0165 zav\u00e1dza\u0165, ako pr\u00e1zdne uzly, na ktor\u00fdch sa minulo miesto. \u010casu bolo str\u00e1ven\u00e9ho ove\u013ea viac, ako by sme chceli. Bol tam riziko \u010diasto\u010dnej alebo \u00faplnej nedostupnosti clustra.\u00a0<\/p>\n<p><b>V\u00fdvody:<\/b><\/p>\n<ul>\n<li>Na v\u0161etk\u00fdch serveroch cassandra by nemalo by\u0165 obsaden\u00e9 viac ako 60% miesta na ka\u017edej jednotke.\u00a0<\/li>\n<li>Mali by by\u0165 za\u0165a\u017een\u00e9 maxim\u00e1lne na 50% pod\u013ea CPU.<\/li>\n<li>Nie je dobr\u00e9 zanedb\u00e1va\u0165 pl\u00e1novanie kapacity a treba ho prem\u00fd\u0161\u013ea\u0165 pre ka\u017ed\u00fa zlo\u017eku, vych\u00e1dzaj\u00fac z jej \u0161pecif\u00edk.<\/li>\n<li>\u010c\u00edm viac uzlov je v clustri \u2014 t\u00fdm lep\u0161ie. Servery, ktor\u00e9 obsahuj\u00fa mal\u00fd objem d\u00e1t, sa r\u00fdchlej\u0161ie obnovuj\u00fa a tak\u00fd clust je \u013eah\u0161ie o\u017eivi\u0165.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Druh\u00e9 miesto \u2014 \u201eZmizli d\u00e1ta z consul key-value \u00falo\u017eiska\u201c<\/h4>\n<p>\nNa zistenie slu\u017eieb pou\u017e\u00edvame consul, rovnako ako mnoh\u00ed in\u00ed. Ale n\u00e1\u0161 key-value je pou\u017e\u00edvan\u00fd aj pre blue-green nasadenie monolitu. Uchov\u00e1va tam inform\u00e1cie o akt\u00edvnych a neakt\u00edvnych upstreamoch, ktor\u00e9 sa pri nasaden\u00ed menia. Na to bol nap\u00edsan\u00fd nasadzovac\u00ed servis, ktor\u00fd komunikoval s KV. V nejakom momente d\u00e1ta z KV zmizli. Obnovili sme ich z pam\u00e4ti, ale s radom ch\u00fdb. V d\u00f4sledku toho sa pri nasaden\u00ed z\u00e1\u0165a\u017e na upstreamy rozdelila nerovnomerne, a dostali sme ve\u013ea ch\u00fdb 502 kv\u00f4li pre\u0165a\u017eeniu backendov pod\u013ea CPU. Nakoniec sme pre\u0161li z consul KV na postgres, odkia\u013e ich odstr\u00e1ni\u0165 u\u017e nie je tak\u00e9 jednoduch\u00e9.\u00a0\u00a0<\/p>\n<p><b>V\u00fdvody:<br \/>\n<\/b><\/p>\n<ul>\n<li>Slu\u017eby bez akejko\u013evek autoriz\u00e1cie by nemali obsahova\u0165 kritick\u00e9 \u00fadaje pre fungovanie webu. Napr\u00edklad, ak nem\u00e1te autoriz\u00e1ciu v ES \u2014 bolo by lep\u0161ie zak\u00e1za\u0165 pr\u00edstup na \u00farovni siete z miesta, kde nie je potrebn\u00fd, ponecha\u0165 len nevyhnutn\u00e9 a nastavi\u0165 action.destructive_requires_name: true.<\/li>\n<li>Pracujte na mechanizme z\u00e1lohovania a obnovy vopred. Napr\u00edklad si dopredu urobte skript (napr\u00edklad na python), ktor\u00fd dok\u00e1\u017ee z\u00e1lohova\u0165 a obnovova\u0165.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Prv\u00e9 miesto \u2014 \u201eKapit\u00e1n nevidite\u013enos\u0165\u201c\u00a0<\/h4>\n<p>\nV ur\u010ditom momente sme si v\u0161imli, \u017ee za\u0165a\u017eenie na upstreamy nginx nie je rovnomerne rozlo\u017een\u00e9, ke\u010f bol v backend-e 10 a viac serverov. Preto\u017ee round-robin smeroval po\u017eiadavky od prv\u00e9ho po posledn\u00fd upstream v porad\u00ed a ka\u017ed\u00fd reload nginx za\u010dal odznova, prv\u00e9 upstreamy dost\u00e1vali v\u017edy viac po\u017eiadaviek ako ostatn\u00e9. Ako d\u00f4sledok \u2014 fungovali pomal\u0161ie a cel\u00fd web trpel. Toto sa st\u00e1valo \u010doraz markantnej\u0161ie so zvy\u0161uj\u00facim sa objemom prev\u00e1dzky. Jednoduch\u00e9 aktualizovanie nginx na zapnutie random nepomohlo \u2014 bolo potrebn\u00e9 predeli\u0165 mno\u017estvo lua k\u00f3du, ktor\u00fd nefungoval vo verzii 1.15 (v tom \u010dase). Museli sme upravi\u0165 n\u00e1\u0161 nginx 1.14.2 a implementova\u0165 do neho podporu random. To vyrie\u0161ilo probl\u00e9m. Tento bug vyhr\u00e1va v kateg\u00f3rii \u201ekapit\u00e1n nevidite\u013enos\u0165\u201d.<\/p>\n<p><b>V\u00fdvody:<\/b><\/p>\n<p>Bolo ve\u013emi zauj\u00edmav\u00e9 a vzru\u0161uj\u00face sk\u00fama\u0165 tento bug).\u00a0<\/p>\n<ul>\n<li>Nastavte monitorovanie tak, aby pom\u00e1halo r\u00fdchlo n\u00e1js\u0165 podobn\u00e9 fluktu\u00e1cie. Napr\u00edklad, m\u00f4\u017eete pou\u017ei\u0165 ELK na sledovanie rps pre ka\u017ed\u00fd backend ka\u017ed\u00e9ho upstreamu, sledujte ich \u010das odpovede z poh\u013eadu nginx. V tomto pr\u00edpade n\u00e1m to pomohlo odhali\u0165 probl\u00e9m.\u00a0<\/li>\n<\/ul>\n<p>\nV\u00e4\u010d\u0161ine zlyhan\u00ed by sa dalo pred\u00eds\u0165 pri prec\u00edznej\u0161om pr\u00edstupe k tomu, \u010do rob\u00edte. Treba si v\u017edy pam\u00e4ta\u0165 z\u00e1kon Murphya:\u00a0<i>Anything that can go wrong will go wrong, <\/i>a budova\u0165 komponenty s t\u00fdm na pam\u00e4ti.\u00a0<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/cian\/blog\/499542\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430!\u00a0 \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d. \u041e\u0434\u043d\u043e\u0439 \u0438\u0437 \u043c\u043e\u0438\u0445 \u043e\u0431\u044f\u0437\u0430\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u0430 \u0438\u043d\u0446\u0438\u0434\u0435\u043d\u0442\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439 \u043d\u0430 \u043f\u0440\u043e\u0434\u0435, \u0434\u043e \u043d\u0443\u043b\u044f. \u0422\u043e, \u043e \u0447\u0435\u043c \u043f\u043e\u0439\u0434\u0435\u0442 \u0440\u0435\u0447\u044c \u0434\u0430\u043b\u0435\u0435, \u043f\u0440\u0438\u043d\u0435\u0441\u043b\u043e \u043d\u0430\u043c \u043c\u043d\u043e\u0433\u043e \u0431\u043e\u043b\u0438, \u0438 \u0446\u0435\u043b\u044c \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u2014 \u043d\u0435 \u0434\u0430\u0442\u044c \u0434\u0440\u0443\u0433\u0438\u043c \u043b\u044e\u0434\u044f\u043c \u043f\u043e\u0432\u0442\u043e\u0440\u0438\u0442\u044c \u043d\u0430\u0448\u0438\u0445 \u043e\u0448\u0438\u0431\u043e\u043a \u0438\u043b\u0438 \u0445\u043e\u0442\u044f \u0431\u044b \u043c\u0438\u043d\u0438\u043c\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438\u0445 \u0432\u043b\u0438\u044f\u043d\u0438\u0435.\u00a0 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80032,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Top faux pas Cian | ProHoster","description":"V\u0161etk\u00fdm dobr\u00e9! Vol\u00e1m sa Nikita, som t\u00edm l\u00eddrom t\u00edmu in\u017einierov Cian.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/top-fakapov-czian","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/top-fakapov-czian","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-02T11:42:49+00:00","article:modified_time":"2020-05-02T11:42:49+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80031","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:47:44","updated":"2026-02-09 16:50:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/80031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=80031"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/80031\/revisions"}],"predecessor-version":[{"id":158728,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/80031\/revisions\/158728"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/80032"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=80031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=80031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=80031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}