{"id":80031,"date":"2020-05-02T13:42:49","date_gmt":"2020-05-02T11:42:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/top-fakapov-czian"},"modified":"2020-05-02T13:42:49","modified_gmt":"2020-05-02T11:42:49","slug":"top-fakapov-czian","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/top-fakapov-czian","title":{"rendered":"Najve\u0107i propusti Ciana","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Najve\u0107i propusti Ciana\" src=\"\/wp-content\/uploads\/2020\/05\/61cf8d25f1f3e858a3b9541cb026cf3c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nVsem dobro!\u00a0<\/p>\n<p>Moje ime je Nikita, sem vodja ekipe in\u017eenirjev Ciana. Ena izmed mojih nalog v podjetju je, da zmanj\u0161am \u0161tevilo incidentov, povezanih z infrastrukturo v produkciji, na ni\u010d.<br \/>\nKar bo omenjeno v nadaljevanju, nam je povzro\u010dilo veliko bole\u010dine, cilj tega \u010dlanka pa je, da drugim prepre\u010dimo ponovitev na\u0161ih napak ali vsaj minimaliziramo njihov vpliv.\u00a0<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3>Uvod<\/h3>\n<p>\nDavno, ko je bil Cian sestavljen iz monolitov in nikakr\u0161nih namigov na mikroservise \u0161e ni bilo, smo merili dostopnost virov z preverjanjem 3\u20135 strani.\u00a0<\/p>\n<p>\u010ce odgovarjajo \u2014 je vse v redu, \u010de ne odgovarjajo dalj \u010dasa \u2014 alarm. Koliko \u010dasa morajo ne delovati, da to \u0161teje kot incident, so odlo\u010dali ljudje na sestankih. Ekipa in\u017eenirjev je vedno sodelovala pri preiskovanju incidenta. Ko je bila preiskava kon\u010dana, so napisali postmortem \u2014 svojevrsten poro\u010dilo po e-po\u0161ti v formatu: kaj se je zgodilo, kako dolgo je trajalo, kaj smo naredili v trenutku, kaj bomo naredili v prihodnosti.\u00a0<\/p>\n<h3>Glavne strani spletne strani ali kako razumemo, da smo dosegli dno<\/h3>\n<p>\u00a0<br \/>\nDa bi lahko na nek na\u010din razumeli prioritetne napake, smo izpostavili najbolj kriti\u010dne strani spletne strani za poslovne funkcionalnosti. Po njih \u0161tejejo \u0161tevilo uspe\u0161nih\/neuspe\u0161nih zahtev in \u010dasovnih omejitev. Tako merimo uptime.\u00a0<\/p>\n<p>Recimo, da smo ugotovili, da obstaja vrsta super pomembnih odsekov spletne strani, ki so odgovorni za glavno storitev \u2014 iskanje in oddajo oglasov. \u010ce \u0161tevilo zahtev, ki so se kon\u010dale z napako, presega 1%, je to kriti\u010den incident. \u010ce se v 15 minutah v konicah napak ve\u010d kot 0,1% \u2014 to se prav tako \u0161teje za kriti\u010den incident. Ti kriteriji pokrivajo ve\u010dji del incidentov, ostali pa presegajo obseg tega \u010dlanka.<\/p>\n<p><img decoding=\"async\" alt=\"Najve\u0107i propusti Ciana\" src=\"\/wp-content\/uploads\/2020\/05\/f5d76b1784c9a52ae1dc4728f1509ccc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Top najbolj\u0161ih incidentov Ciana<\/h3>\n<p>\nTorej, zagotovo smo se nau\u010dili prepoznati dejstvo, da se je incident zgodil.\u00a0<\/p>\n<p>Zdaj je vsak incident podrobno opisan in odra\u017een v epiki Jira. Mimogrede: za to smo ustanovili poseben projekt, ga poimenovali FAIL \u2014 v njem je mogo\u010de ustvariti samo epike.\u00a0<\/p>\n<p>\u010ce zberemo vse napake v zadnjih nekaj letih, so na vrhu:\u00a0<\/p>\n<ul>\n<li>incidenti, povezani z mssql;<\/li>\n<li>incidenti, povzro\u010deni z zunanjimi dejavniki;<\/li>\n<li>napake administratorja.<\/li>\n<\/ul>\n<p>\nOsredoto\u010dili se bomo podrobneje na napake administratorjev, pa tudi na nekatere druge zanimive napake.<\/p>\n<h4>Peto mesto \u2014 \u201ePostavljamo stvari v DNS\u201c<\/h4>\n<p>\nBil je turobni torek. Odlo\u010dili smo se, da bomo uredili DNS-klaster.\u00a0<\/p>\n<p>Zadali smo si, da prena\u0161amo notranje DNS-stre\u017enike iz bind na powerdns, pri \u010demer smo za to namenili popolnoma lo\u010dene stre\u017enike, na katerih ni ni\u010d drugega kot DNS.\u00a0<\/p>\n<p>Postavili smo po en DNS-stre\u017enik na vsaki lokaciji na\u0161ih podatkovnih centrov, in pri\u0161el je trenutek, da prenesemo domene iz bind v powerdns in preklopimo infrastrukturo na nove stre\u017enike.\u00a0<\/p>\n<p>V najhuj\u0161em trenutku prenosa je iz vseh <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sl\/server\/\"   title=\"stre\u017enikov\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1484\">stre\u017enikov<\/a>, ki so bili navedeni v lokalnih predpomnilnikih bind na vseh stre\u017enikih, ostal le en, ki je bil v podatkovnem centru v Sankt Peterburgu. Ta DC je prvotno bil deklariran kot nekriti\u010den za nas, a je nenadoma postal enotna to\u010dka napake.<br \/>\nPrav v tem obdobju prenosa je odpovedal kanal med Moskvo in Sankt Peterburgom. Dejansko smo ostali brez DNS-a pet minut, dokler <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sl\/\"   title=\"ponudnik gostovanja\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1206\">ponudnik gostovanja<\/a> ni odpravil te\u017eav.\u00a0<\/p>\n<p><b>Zaklju\u010dki: <\/b><\/p>\n<p>\u010ce smo prej zanemarjali zunanje dejavnike med pripravami na delo, pa smo jih zdaj vklju\u010dili v seznam stvari, na katere se pripravljamo. Zdaj stremimo k temu, da so vsi komponenti rezervirani n-2, na \u010das dela pa lahko zni\u017eamo to raven na n-1.<\/p>\n<ul>\n<li>Med sestavljanjem akcijskega na\u010drta ozna\u010dite to\u010dke, kjer bi lahko storitev odpovedala, in razmislite o scenariju, kjer se vse zgodi 'slab\u0161e ne more biti', vnaprej.<\/li>\n<li>Porazdelite notranje DNS-stre\u017enike po razli\u010dnih geolokacijah\/podatkovnih centrih\/rackih\/switchih\/priklju\u010dkih.<\/li>\n<li>Na vsakem stre\u017eniku namestite lokalni predpomnilni DNS-stre\u017enik, ki preusmerja poizvedbe na glavne DNS-stre\u017enike, v primeru njegove nedostopnosti pa odgovarja iz predpomnilnika.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>\u010cetrto mesto \u2014 'Urejanje v Nginx'<\/h4>\n<p>\nNekega lepega dne je na\u0161a ekipa sklenila, da 'je dovolj, da to prena\u0161amo', in se je za\u010del proces refaktorizacije Nginx konfiguracij. Glavni cilj je bil pripeljati konfiguracije do intuitivne strukture. Prej je bilo vse 'zgodovinsko povsem zmedeno' in ni imelo nobene logike. Zdaj smo vsak server_name prenesli v istoimensko datoteko in porazdelili vse konfiguracije po mapah. Mimogrede \u2014 konfiguracija vsebuje 253949 vrstic ali 7836520 znakov in zaseda skoraj 7 megabajtov. Vi\u0161ja raven strukture:\u00a0<\/p>\n<p>                        <b class=\"spoiler_title\">Struktura Nginx<\/b><\/p>\n<pre><code class=\"plaintext\">\u251c\u2500\u2500 access\n\u2502 \u00a0 \u251c\u2500\u2500 allow.list\n...\n\u2502 \u00a0 \u2514\u2500\u2500 whitelist.conf\n\u251c\u2500\u2500 geobase\n\u2502 \u00a0 \u251c\u2500\u2500 exclude.conf\n...\n\u2502 \u00a0 \u2514\u2500\u2500 geo_ip_to_region_id.conf\n\u251c\u2500\u2500 geodb\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP.dat\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP2-Country.mmdb\n\u2502 \u00a0 \u2514\u2500\u2500 GeoLiteCity.dat\n\u251c\u2500\u2500 inc\n\u2502 \u00a0 \u251c\u2500\u2500 error.inc\n...\n\u2502 \u00a0 \u2514\u2500\u2500 proxy.inc\n\u251c\u2500\u2500 lists.d\n\u2502 \u00a0 \u251c\u2500\u2500 bot.conf\n...\n\u2502 \u00a0 \u251c\u2500\u2500 dynamic\n\u2502 \u00a0 \u2514\u2500\u2500 geo.conf\n\u251c\u2500\u2500 lua\n\u2502 \u00a0 \u251c\u2500\u2500 cookie.lua\n\u2502 \u00a0 \u251c\u2500\u2500 log\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 log.lua\n\u2502 \u00a0 \u251c\u2500\u2500 logics\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 include.lua\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 utils.lua\n\u2502 \u00a0 \u2514\u2500\u2500 prom\n\u2502 \u00a0 \u00a0 \u00a0 \u251c\u2500\u2500 stats.lua\n\u2502 \u00a0 \u00a0 \u00a0 \u2514\u2500\u2500 stats_prometheus.lua\n\u251c\u2500\u2500 map.d\n\u2502 \u00a0 \u251c\u2500\u2500 access.conf\n\u2502 \u00a0 \u251c\u2500\u2500 ..\u00a0\n\u2502 \u00a0 \u2514\u2500\u2500 zones.conf\n\u251c\u2500\u2500 nginx.conf\n\u251c\u2500\u2500 robots.txt\n\u251c\u2500\u2500 server.d\n\u2502 \u00a0 \u251c\u2500\u2500 cian.ru\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 cian.ru.conf\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 my.cian.ru.conf\n\u251c\u2500\u2500 service.d\n\u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2514\u2500\u2500 status.conf\n\u2514\u2500\u2500 upstream.d\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 cian-mcs.conf\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 ...\n\u00a0\u00a0\u00a0\u00a0\u2514\u2500\u2500 wafserver.conf<\/code><\/pre>\n<p>\u0417\u043d\u0430\u0447\u043d\u043e \u0443\u043b\u0443\u0447\u0448\u0438\u043b\u043e\u0441\u044c, \u043d\u043e \u0432 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435 \u043f\u0435\u0440\u0435\u0438\u043c\u0435\u043d\u043e\u0432\u0430\u043d\u0438\u044f \u0438 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0439 \u0447\u0430\u0441\u0442\u044c \u0438\u0437 \u043d\u0438\u0445 \u0438\u043c\u0435\u043b\u0430 \u043d\u0435\u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e\u0435 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0438 \u043d\u0435 \u043f\u043e\u043f\u0430\u043b\u0430 \u0432 \u0434\u0438\u0440\u0435\u043a\u0442\u0438\u0432\u0443 include *.conf. \u0412 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0435 \u0447\u0430\u0441\u0442\u044c \u0445\u043e\u0441\u0442\u043e\u0432 \u043e\u043a\u0430\u0437\u0430\u043b\u0430\u0441\u044c \u043d\u0435\u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0439 \u0438 \u0432\u043e\u0437\u0432\u0440\u0430\u0449\u0430\u043b\u0430 301 \u043d\u0430 \u0433\u043b\u0430\u0432\u043d\u0443\u044e \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0443. \u041f\u043e\u0441\u043a\u043e\u043b\u044c\u043a\u0443 \u043a\u043e\u0434 \u043e\u0442\u0432\u0435\u0442\u0430 \u043d\u0435 \u0431\u044b\u043b 5xx\/4xx, \u044d\u0442\u043e \u0437\u0430\u043c\u0435\u0442\u0438\u043b\u0438 \u043d\u0435 \u0441\u0440\u0430\u0437\u0443, \u0430 \u043b\u0438\u0448\u044c \u043f\u043e\u0434 \u0443\u0442\u0440\u043e. \u041f\u043e\u0441\u043b\u0435 \u044d\u0442\u043e\u0433\u043e \u043c\u044b \u043d\u0430\u0447\u0430\u043b\u0438 \u043f\u0438\u0441\u0430\u0442\u044c \u0442\u0435\u0441\u0442\u044b \u0434\u043b\u044f \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043d\u044b\u0445 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432.<\/p>\n<p><b>Zaklju\u010dki:<\/b>\u00a0<\/p>\n<ul>\n<li>\u041f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0438\u0440\u0443\u0439\u0442\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 (\u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e nginx) \u0438 \u043f\u0440\u043e\u0434\u0443\u043c\u044b\u0432\u0430\u0439\u0442\u0435 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0443 \u043d\u0430 \u0440\u0430\u043d\u043d\u0435\u043c \u044d\u0442\u0430\u043f\u0435 \u043f\u0440\u043e\u0435\u043a\u0442\u0430. \u042d\u0442\u043e \u0441\u0434\u0435\u043b\u0430\u0435\u0442 \u0438\u0445 \u0431\u043e\u043b\u0435\u0435 \u043f\u043e\u043d\u044f\u0442\u043d\u044b\u043c\u0438 \u0434\u043b\u044f \u043a\u043e\u043c\u0430\u043d\u0434\u044b, \u0447\u0442\u043e, \u0432 \u0441\u0432\u043e\u044e \u043e\u0447\u0435\u0440\u0435\u0434\u044c, \u0441\u043e\u043a\u0440\u0430\u0442\u0438\u0442 \u0432\u0440\u0435\u043c\u044f \u043d\u0430 \u0432\u044b\u0432\u043e\u0434 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u0430 \u043d\u0430 \u0440\u044b\u043d\u043e\u043a.<\/li>\n<li>\u0414\u043b\u044f \u043d\u0435\u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043d\u044b\u0445 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u043e\u0432 \u043f\u0438\u0448\u0438\u0442\u0435 \u0442\u0435\u0441\u0442\u044b. \u041d\u0430\u043f\u0440\u0438\u043c\u0435\u0440: \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0430, \u0447\u0442\u043e \u0432\u0441\u0435 \u043a\u043b\u044e\u0447\u0435\u0432\u044b\u0435 server_name \u043e\u0442\u0434\u0430\u044e\u0442 \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u044b\u0439 \u0441\u0442\u0430\u0442\u0443\u0441, \u043f\u043b\u044e\u0441 \u0442\u0435\u043b\u043e \u043e\u0442\u0432\u0435\u0442\u0430. \u0411\u0443\u0434\u0435\u0442 \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u0438\u043c\u0435\u0442\u044c \u043f\u043e\u0434 \u0440\u0443\u043a\u043e\u0439 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0441\u043a\u0440\u0438\u043f\u0442\u043e\u0432, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u044e\u0442 \u043e\u0441\u043d\u043e\u0432\u043d\u044b\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u0438 \u043a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u0430, \u0447\u0442\u043e\u0431\u044b \u043d\u0435 \u0432\u0441\u043f\u043e\u043c\u0438\u043d\u0430\u0442\u044c \u0432 3 \u0447\u0430\u0441\u0430 \u043d\u043e\u0447\u0438, \u0447\u0442\u043e \u0435\u0449\u0435 \u043d\u0443\u0436\u043d\u043e \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u0442\u044c.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>\u0422\u0440\u0435\u0442\u044c\u0435 \u043c\u0435\u0441\u0442\u043e \u2014 \u00ab\u0412\u043d\u0435\u0437\u0430\u043f\u043d\u043e \u0437\u0430\u043a\u043e\u043d\u0447\u0438\u043b\u043e\u0441\u044c \u043c\u0435\u0441\u0442\u043e \u0432 Cassandra\u00bb<\/h4>\n<p>\n\u0414\u0430\u043d\u043d\u044b\u0435 \u043f\u043e\u0441\u0442\u0435\u043f\u0435\u043d\u043d\u043e \u0440\u043e\u0441\u043b\u0438, \u0438 \u0432\u0441\u0435 \u0431\u044b\u043b\u043e \u0445\u043e\u0440\u043e\u0448\u043e \u0434\u043e \u0442\u043e\u0433\u043e \u043c\u043e\u043c\u0435\u043d\u0442\u0430, \u043a\u043e\u0433\u0434\u0430 \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0435 Cassandra \u043d\u0430\u0447\u0430\u043b\u0438 \u043f\u0430\u0434\u0430\u0442\u044c \u0440\u0435\u043c\u043e\u043d\u0442\u044b \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u043a\u0435\u0439\u0441\u043f\u0435\u0439\u0441\u043e\u0432, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u043d\u0430 \u043d\u0438\u0445 \u043d\u0435 \u043c\u043e\u0436\u0435\u0442 \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e \u0441\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u043a\u043e\u043c\u043f\u0430\u043a\u0446\u0438\u044f.\u00a0<\/p>\n<p>\u0412 \u043e\u0434\u0438\u043d \u043d\u0435\u043d\u0430\u0441\u0442\u043d\u044b\u0439 \u0434\u0435\u043d\u044c \u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u043f\u043e\u0447\u0442\u0438 \u043f\u0440\u0435\u0432\u0440\u0430\u0442\u0438\u043b\u0441\u044f \u0432 \u0442\u044b\u043a\u0432\u0443, \u0430 \u0438\u043c\u0435\u043d\u043d\u043e:<\/p>\n<ul>\n<li>\u043e\u0441\u0442\u0430\u0432\u0430\u043b\u043e\u0441\u044c \u043e\u043a\u043e\u043b\u043e 20% \u043c\u0435\u0441\u0442\u0430 \u043f\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0443;<\/li>\n<li>\u043f\u043e\u043b\u043d\u043e\u0446\u0435\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0443\u0437\u043b\u044b \u043d\u0435\u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e, \u0442\u0430\u043a \u043a\u0430\u043a \u043d\u0435 \u043f\u0440\u043e\u0445\u043e\u0434\u0438\u0442 \u043e\u0447\u0438\u0441\u0442\u043a\u0430 \u043f\u043e\u0441\u043b\u0435 \u0434\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0443\u0437\u043b\u0430 \u0438\u0437-\u0437\u0430 \u043d\u0435\u0445\u0432\u0430\u0442\u043a\u0438 \u043c\u0435\u0441\u0442\u0430 \u043d\u0430 \u0440\u0430\u0437\u0434\u0435\u043b\u0430\u0445;<\/li>\n<li>\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u043f\u043e\u0441\u0442\u0435\u043f\u0435\u043d\u043d\u043e \u0441\u043d\u0438\u0436\u0430\u0435\u0442\u0441\u044f, \u0442\u0430\u043a \u043a\u0430\u043a \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u043e\u043c\u043f\u0430\u043a\u0446\u0438\u044f;\u00a0<\/li>\n<li>\u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u0432 \u0430\u0432\u0430\u0440\u0438\u0439\u043d\u043e\u043c \u0440\u0435\u0436\u0438\u043c\u0435.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Najve\u0107i propusti Ciana\" src=\"\/wp-content\/uploads\/2020\/05\/1d036fbc500a49a285fa4b0014e2a70d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIzlaz \u2014 dodali smo jo\u0161 5 \u010dvorova bez \u010di\u0161\u0107enja, nakon \u010dega smo po\u010deli planski povla\u010diti iz klastera i ponovo uvo\u0111enje, kao prazni \u010dvorovi, na kojima je ponestalo prostora. Vremena je potro\u0161eno mnogo vi\u0161e nego \u0161to bismo \u017eeleli. Postojala je opasnost od delimi\u010dne ili potpune nedostupnosti klastera.\u00a0<\/p>\n<p><b>Zaklju\u010dki:<\/b><\/p>\n<ul>\n<li>Na svim serverima Cassandra ne bi trebalo da bude zauzeto vi\u0161e od 60% prostora na svakom odeljenju.\u00a0<\/li>\n<li>Zadovoljstvo optere\u0107enje CPU-a ne bi smelo da prelazi 50%.<\/li>\n<li>Ne treba zanemarivati planiranje kapaciteta, i ono se mora razmi\u0161ljati za svaki komponent, uzimaju\u0107i u obzir njegovu specifi\u010dnost.<\/li>\n<li>\u0160to vi\u0161e \u010dvorova u klasteru \u2014 to bolje. Serveri koji sadr\u017ee mali obim podataka br\u017ee se prebacuju, a takav klaster je lak\u0161e o\u017eiveti.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Drugo mesto \u2014 \"Podaci su nestali iz Consul key-value skladi\u0161ta\"<\/h4>\n<p>\nZa otkrivanje usluga mi, kao i mnogi, koristimo Consul. No, na\u0161e key-value se koristi i za blue-green izbacivanje monolita. Tu se \u010duvaju informacije o aktivnim i neaktivnim upstream-ima, koje se menjaju tokom deploja. Zbog toga je napisan deploy servis koji je komunicirao sa KV-om. U nekom trenutku podaci su nestali iz KV-a. Oporavili smo se iz se\u0107anja, ali uz niz gre\u0161aka. Kao posledica \u2014 tokom izbacivanja optere\u0107enje na upstream-ima se raspodelilo nejednako, a dobili smo mnogo gre\u0161aka 502 zbog preoptere\u0107enja backend-a po CPU-u. Na kraju smo pre\u0161li sa Consul KV na Postgres, odakle ih ve\u0107 nije lako obrisati.\u00a0\u00a0<\/p>\n<p><b>Zaklju\u010dki:<br \/>\n<\/b><\/p>\n<ul>\n<li>Usluge bez ikakve autorizacije ne bi trebale sadr\u017eavati kriti\u010dne podatke za rad sajta. Na primer, ako nema autorizacije u ES \u2014 najbolje bi bilo zabraniti pristup na mre\u017enom nivou svuda gde nije potreban, ostaviti samo neophodne, kao i omogu\u0107iti action.destructive_requires_name: true.<\/li>\n<li>Isprobajte mehanizam rezervnog kopiranja i oporavka unapred. Na primer, unapred napravite skript (na primer, na Python-u) koji mo\u017ee i da pravi rezervne kopije i da obnavlja.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Prvo mesto \u2014 \"Kapetan ne o\u010diglednost\"\u00a0<\/h4>\n<p>\nNekje smo opazili nepravilno porazdelitev obremenitve na upstream nginx, ko je v ozadju delovalo ve\u010d kot 10 stre\u017enikov. Ker je round-robin usmerjal zahtevke od prvega do zadnjega upstream po vrsti, in je vsak reload nginx za\u010del znova, so prve upstream vedno prejele ve\u010d zahtevkov kot ostale. Posledi\u010dno so delovale po\u010dasneje in celotna stran je trpela. To je postajalo vse bolj o\u010ditno, ko se je pove\u010devalo \u0161tevilo prometa. Preprosto osve\u017eitev nginx, da bi vklju\u010dili random, ni uspela \u2014 treba je bilo predelati kup lua kode, ki ni delovala na verziji 1.15 (takrat). Tako smo morali patchati na\u0161 nginx 1.14.2 in vanj vgraditi podporo za random. To je re\u0161ilo problem. Ta napaka zmaga v kategoriji 'kapitan o\u010ditnost'.<\/p>\n<p><b>Zaklju\u010dki:<\/b><\/p>\n<p>Bilo je zelo zanimivo in vznemirljivo raziskovati to napako).\u00a0<\/p>\n<ul>\n<li>Postavite monitoring tako, da bo pomagal hitro odkrivati podobne fluktuacije. Na primer, lahko uporabite ELK za spremljanje rps za vsak backend vsakega upstreama in spremljate njihov \u010das odziva z vidika nginx. V tem primeru nam je to pomagalo odkriti problem.\u00a0<\/li>\n<\/ul>\n<p>\nVe\u010dji del napak bi bilo mogo\u010de prepre\u010diti z bolj skrbnim pristopom k temu, kar po\u010dnete. Vedno se morate spomniti na Murphyjev zakon:\u00a0<i>Vsaka stvar, ki lahko gre narobe, bo \u0161la narobe, <\/i>in graditi komponente, ki se ravnajo po njem.\u00a0<br \/>\n<br \/>Izvor: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/cian\/blog\/499542\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430!\u00a0 \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d. \u041e\u0434\u043d\u043e\u0439 \u0438\u0437 \u043c\u043e\u0438\u0445 \u043e\u0431\u044f\u0437\u0430\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u0430 \u0438\u043d\u0446\u0438\u0434\u0435\u043d\u0442\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439 \u043d\u0430 \u043f\u0440\u043e\u0434\u0435, \u0434\u043e \u043d\u0443\u043b\u044f. \u0422\u043e, \u043e \u0447\u0435\u043c \u043f\u043e\u0439\u0434\u0435\u0442 \u0440\u0435\u0447\u044c \u0434\u0430\u043b\u0435\u0435, \u043f\u0440\u0438\u043d\u0435\u0441\u043b\u043e \u043d\u0430\u043c \u043c\u043d\u043e\u0433\u043e \u0431\u043e\u043b\u0438, \u0438 \u0446\u0435\u043b\u044c \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u2014 \u043d\u0435 \u0434\u0430\u0442\u044c \u0434\u0440\u0443\u0433\u0438\u043c \u043b\u044e\u0434\u044f\u043c \u043f\u043e\u0432\u0442\u043e\u0440\u0438\u0442\u044c \u043d\u0430\u0448\u0438\u0445 \u043e\u0448\u0438\u0431\u043e\u043a \u0438\u043b\u0438 \u0445\u043e\u0442\u044f \u0431\u044b \u043c\u0438\u043d\u0438\u043c\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438\u0445 \u0432\u043b\u0438\u044f\u043d\u0438\u0435.\u00a0 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80032,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sl_SI\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Top fopic Cian | ProHoster","description":"Vsem dobro! Moje ime je Nikita, sem vodja ekipe in\u017eenirjev Cian.","canonical_url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/top-fakapov-czian","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sl_SI","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.","og:url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/top-fakapov-czian","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-02T11:42:49+00:00","article:modified_time":"2020-05-02T11:42:49+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80031","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:47:44","updated":"2026-02-09 16:50:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/80031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/comments?post=80031"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/80031\/revisions"}],"predecessor-version":[{"id":158728,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/80031\/revisions\/158728"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media\/80032"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media?parent=80031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/categories?post=80031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/tags?post=80031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}