{"id":80031,"date":"2020-05-02T13:42:49","date_gmt":"2020-05-02T11:42:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/top-fakapov-czian"},"modified":"2020-05-02T13:42:49","modified_gmt":"2020-05-02T11:42:49","slug":"top-fakapov-czian","status":"publish","type":"post","link":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/top-fakapov-czian","title":{"rendered":"Top gre\u0161aka na Cian-u","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Top gre\u0161aka na Cian-u\" src=\"\/wp-content\/uploads\/2020\/05\/61cf8d25f1f3e858a3b9541cb026cf3c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSvima dobro!\u00a0<\/p>\n<p>Moje ime je Nikita, ja sam tim lider in\u017eenjerskog tima Cian. Jedna od mojih odgovornosti u firmi je sni\u017eavanje broja incidenata vezanih za infrastrukturu u produkciji na nulu.<br \/>\nOno o \u010demu \u0107emo dalje govoriti nanijelo nam je mnogo boli, a cilj ovog \u010dlanka je sprije\u010diti druge da ponove na\u0161e gre\u0161ke ili barem minimizirati njihov utjecaj.\u00a0<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3>Preamvula<\/h3>\n<p>\nDavno, kada je Cian bio sastavljen od monolita, i kada nije bilo nikakvih tragova mikroservisa, mjerili smo dostupnost resursa provjerom 3-5 stranica.\u00a0<\/p>\n<p>Odgovaraju \u2014 sve je u redu, ne odgovaraju du\u017ee vrijeme \u2014 alert. Koliko vremena moraju ne raditi da bi se to smatralo incidentom, odlu\u010divali su ljudi na sastancima. In\u017eenjerski tim uvijek je sudjelovao u istragama incidenata. Kada je istraga zavr\u0161ena, pisali smo postmortem \u2014 svojevrsno izvje\u0161\u0107e na mail u formatu: \u0161to se dogodilo, koliko je trajalo, \u0161to smo u\u010dinili u tom trenutku, \u0161to \u0107emo u\u010diniti u budu\u0107nosti.\u00a0<\/p>\n<h3>Glavne stranice web sajta ili kako shva\u0107amo da smo udarili u dno<\/h3>\n<p>\u00a0<br \/>\nKako bismo mogli razumjeti prioritet gre\u0161ke, izdvojili smo najkriti\u010dnije stranice web sajta za poslovne funkcionalnosti. Na temelju njih mjerimo broj uspje\u0161nih\/neuspje\u0161nih zahtjeva i timeoutova. Tako mjerimo uptime.\u00a0<\/p>\n<p>Recimo, otkrili smo da postoji nekoliko izuzetno va\u017enih sekcija web sajta koje su odgovorne za glavnu uslugu \u2014 pretra\u017eivanje i objavljivanje oglasa. Ako broj zahtjeva koji su zavr\u0161ili gre\u0161kom prema\u0161i 1% \u2014 to je kriti\u010dni incident. Ako tijekom 15 minuta u prime time-u postotak gre\u0161aka prema\u0161i 0,1% \u2014 to se tako\u0111er smatra kriti\u010dnim incidentom. Ovi kriteriji pokrivaju ve\u0107inu incidenata, dok ostali izlaze iz okvira ovog \u010dlanka.<\/p>\n<p><img decoding=\"async\" alt=\"Top gre\u0161aka na Cian-u\" src=\"\/wp-content\/uploads\/2020\/05\/f5d76b1784c9a52ae1dc4728f1509ccc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Top najboljih incidenata Ciana<\/h3>\n<p>\nDakle, definitivno smo nau\u010dili prepoznati kada je incident nastao.\u00a0<\/p>\n<p>Sada je svaki incident detaljno opisan i odra\u017een u epiku Jira. Usput, za ovo smo otvorili poseban projekt, nazvali ga FAIL \u2014 u njemu je mogu\u0107e kreirati samo epike.\u00a0<\/p>\n<p>Ako prikupimo sve gre\u0161ke u posljednjih nekoliko godina, vode:\u00a0<\/p>\n<ul>\n<li>incidente vezane uz mssql;<\/li>\n<li>incidente nastale zbog vanjskih faktora;<\/li>\n<li>gre\u0161ke administratora.<\/li>\n<\/ul>\n<p>\nPogledajmo detaljnije gre\u0161ke administratora, kao i neke druge zanimljive gre\u0161ke.<\/p>\n<h4>Peto mjesto \u2014 'Ure\u0111ujemo DNS'<\/h4>\n<p>\nBio je to obla\u010dan utorak. Odlu\u010dili smo srediti DNS klaster.\u00a0<\/p>\n<p>\u017deljeli smo prebaciti interne DNS poslu\u017eitelje s bind-a na powerdns, dodijeliv\u0161i potpuno odvojene poslu\u017eitelje, na kojima nema ni\u0161ta osim DNS-a.\u00a0<\/p>\n<p>Postavili smo po jedan DNS poslu\u017eitelj na svaku lokaciju na\u0161ih DC-a, i do\u0161ao je trenutak preseljenja zona iz bind-a u powerdns i prebacivanja infrastrukture na nove poslu\u017eitelje.\u00a0<\/p>\n<p>U samom vrhuncu preseljenja ostao je samo jedan od svih <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/hr\/server\/\"   title=\"poslu\u017eitelja\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1484\">poslu\u017eitelja<\/a>, koji su bili navedeni u lokalnim cache-ima bind-a na svim poslu\u017eiteljima, koji se nalazio u data centru u Sankt Peterburgu. Ovaj DC je prvotno bio progla\u0161en nekriti\u010dnim za nas, ali je iznenada postao jedinstvena to\u010dka kvara.<br \/>\nBa\u0161 u tom razdoblju preseljenja pao je kanal izme\u0111u Moskve i Sankt Peterburga. U prakti\u010dki smo ostali bez DNS-a pet minuta i podigli se kada <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/hr\/\"   title=\"hosting\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1206\">hosting<\/a> otklonio kvar.\u00a0<\/p>\n<p><b>Zaklju\u010dci: <\/b><\/p>\n<p>Ako smo ranije zanemarivali vanjske \u010dimbenike tijekom pripreme za radove, sada su i oni uklju\u010deni u popis na \u0161to se pripremamo. I sada te\u017eimo tome da su svi komponente rezervirane n-2, a tijekom radova taj nivo mo\u017eemo smanjiti na n-1.<\/p>\n<ul>\n<li>Tijekom sastavljanja plana djelovanja ozna\u010dite to\u010dke gdje bi usluga mogla pasti i osmislite scenarij gdje je sve oti\u0161lo \"lo\u0161e do lo\u0161eg\", unaprijed.<\/li>\n<li>Rasporedite interne DNS poslu\u017eitelje po razli\u010ditim geo-lokacijama\/data centrima\/rakovima\/komutatorima\/povezanim to\u010dkama.<\/li>\n<li>Na svakom poslu\u017eitelju postavite lokalni caching DNS poslu\u017eitelj koji preusmjerava zahtjeve na glavne DNS poslu\u017eitelje, a u slu\u010daju njegove nedostupnosti odgovarat \u0107e iz cache-a.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>\u010cetvrto mjesto \u2014 \"Sre\u0111ivanje u Nginxu\"<\/h4>\n<p>\nJednog lijepog dana na\u0161 tim odlu\u010dio je da je \"dosta to trpljenja\" i zapo\u010deo je proces refaktoriranja nginx konfiguracija. Glavni cilj \u2014 dovesti konfiguracije u intuitivnu strukturu. Prije je sve bilo \"istorijski ranije\" i nije imalo nikakvu logiku. Sada je svaki server_name iznesen u istoimeni fajl i svi su konfiguracije raspore\u0111ene po mapama. Usput \u2014 konfiguracija sadr\u017ei 253949 redaka ili 7836520 znakova i zauzima gotovo 7 megabajta. Gornja razina strukture:\u00a0<\/p>\n<p>                        <b class=\"spoiler_title\">Struktura Nginxa<\/b><\/p>\n<pre><code class=\"plaintext\">\u251c\u2500\u2500 pristup\n\u2502 \u00a0 \u251c\u2500\u2500 allow.list\n...\n\u2502 \u00a0 \u2514\u2500\u2500 whitelist.conf\n\u251c\u2500\u2500 geobase\n\u2502 \u00a0 \u251c\u2500\u2500 exclude.conf\n...\n\u2502 \u00a0 \u2514\u2500\u2500 geo_ip_to_region_id.conf\n\u251c\u2500\u2500 geodb\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP.dat\n\u2502 \u00a0 \u251c\u2500\u2500 GeoIP2-Country.mmdb\n\u2502 \u00a0 \u2514\u2500\u2500 GeoLiteCity.dat\n\u251c\u2500\u2500 inc\n\u2502 \u00a0 \u251c\u2500\u2500 error.inc\n...\n\u2502 \u00a0 \u2514\u2500\u2500 proxy.inc\n\u251c\u2500\u2500 lists.d\n\u2502 \u00a0 \u251c\u2500\u2500 bot.conf\n...\n\u2502 \u00a0 \u251c\u2500\u2500 dynamic\n\u2502 \u00a0 \u2514\u2500\u2500 geo.conf\n\u251c\u2500\u2500 lua\n\u2502 \u00a0 \u251c\u2500\u2500 cookie.lua\n\u2502 \u00a0 \u251c\u2500\u2500 log\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 log.lua\n\u2502 \u00a0 \u251c\u2500\u2500 logics\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 include.lua\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 utils.lua\n\u2502 \u00a0 \u2514\u2500\u2500 prom\n\u2502 \u00a0 \u00a0 \u00a0 \u251c\u2500\u2500 stats.lua\n\u2502 \u00a0 \u00a0 \u00a0 \u2514\u2500\u2500 stats_prometheus.lua\n\u251c\u2500\u2500 map.d\n\u2502 \u00a0 \u251c\u2500\u2500 access.conf\n\u2502 \u00a0 \u251c\u2500\u2500 ..  \n\u2502 \u00a0 \u2514\u2500\u2500 zones.conf\n\u251c\u2500\u2500 nginx.conf\n\u251c\u2500\u2500 robots.txt\n\u251c\u2500\u2500 server.d\n\u2502 \u00a0 \u251c\u2500\u2500 cian.ru\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 cian.ru.conf\n\u2502 \u00a0 \u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2502 \u00a0 \u2514\u2500\u2500 my.cian.ru.conf\n\u251c\u2500\u2500 service.d\n\u2502 \u00a0 \u251c\u2500\u2500 ...\n\u2502 \u00a0 \u2514\u2500\u2500 status.conf\n\u2514\u2500\u2500 upstream.d\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 cian-mcs.conf\n\u00a0\u00a0\u00a0\u00a0\u251c\u2500\u2500 ...\n\u00a0\u00a0\u00a0\u00a0\u2514\u2500\u2500 wafserver.conf<\/code><\/pre>\n<p>Zna\u010dajno je bolje, ali u procesu preimenovanja i raspodjele konfiguracija, neki od njih su imali pogre\u0161nu ekstenziju i nisu do\u0161li u direktivu include *.conf. Kao posljedica \u2014 neki hostovi su postali nedostupni i vra\u0107ali 301 na glavni. Budu\u0107i da je kod odgovora bio ne 5xx\/4xx, to nije primije\u0107eno odmah, ve\u0107 tek ujutro. Nakon toga, po\u010deli smo pisati testove za provjeru infrastrukturalnih komponenti.<\/p>\n<p><b>Zaklju\u010dci:<\/b>\u00a0<\/p>\n<ul>\n<li>Ispravno strukturirajte konfiguracije (ne samo nginx) i razmislite o strukturi u ranoj fazi projekta. Tako \u0107ete ih u\u010diniti razumljivijima timu, \u0161to \u0107e smanjiti TTM.<\/li>\n<li>Za neke infrastrukturne komponente pi\u0161ite testove. Na primjer: provjera da svi klju\u010dni server_name vra\u0107aju ispravan status, + tijelo odgovora. Bit \u0107e dovoljno imati na raspolaganju samo nekoliko skripti koje provjeravaju osnovne funkcije komponente, kako ne biste nervozno prisje\u0107ali se u 3 sata ujutro \u0161to jo\u0161 treba provjeriti.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Tre\u0107e mjesto \u2014 \u201eIznenada je ponestalo mjesta u Cassandri\u201c<\/h4>\n<p>\nPodaci su se postepeno pove\u0107avali, i sve je bilo dobro do trenutka kada su u klasteru Cassandri po\u010deli padati repair veliki keyspace-ovi, jer na njima ne mo\u017ee odraditi compaction.\u00a0<\/p>\n<p>Jednog ki\u0161nog dana klaster se gotovo pretvorio u bundevu, a to\u010dno:<\/p>\n<ul>\n<li>ostalo je oko 20% ukupno u klasteru;<\/li>\n<li>ne mo\u017ee se potpuno dodati node, jer cleanup nakon dodavanja node ne prolazi zbog nedostatka mjesta na particijama;<\/li>\n<li>performanse polako padaju, jer compaction ne radi;\u00a0<\/li>\n<li>klaster radi u hitnom re\u017eimu.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Top gre\u0161aka na Cian-u\" src=\"\/wp-content\/uploads\/2020\/05\/1d036fbc500a49a285fa4b0014e2a70d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIzlaz \u2014 dodali smo jo\u0161 5 nodova bez \u010di\u0161\u0107enja, nakon \u010dega smo po\u010deli postupno povla\u010diti iz klastera i ponovo uvo\u0111enje, kao prazne nodove, na kojima je nestalo prostora. Vremena je potro\u0161eno daleko vi\u0161e nego \u0161to bismo \u017eeleli. Postojao je rizik od djelomi\u010dne ili potpune nedostupnosti klastera.\u00a0<\/p>\n<p><b>Zaklju\u010dci:<\/b><\/p>\n<ul>\n<li>Na svim serverima Cassandra ne bi smjelo biti zauzeto vi\u0161e od 60% prostora na svakom disku.\u00a0<\/li>\n<li>Treba da budu optere\u0107eni ne vi\u0161e od 50% po CPU-u.<\/li>\n<li>Ne biste trebali zapostaviti planiranje kapaciteta i trebate ga razmatrati za svaku komponentu, uzimaju\u0107i u obzir njenu specifi\u010dnost.<\/li>\n<li>\u0160to vi\u0161e nodova u klasteru, to bolje. Serveri koji sadr\u017ee mali obim podataka br\u017ee se prekonfiguriraju, i takav klaster je lak\u0161e reanimirati.\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h4>Drugo mesto \u2014 \"Nestali podaci iz Consul key-value skladi\u0161ta\"<\/h4>\n<p>\nZa otkrivanje servisa, kao i mnogi, koristimo Consul. No, na\u0161 key-value se koristi i za blue-green postavljanje monolita. Tamo se \u010duva informacija o aktivnim i neaktivnim upstreamima, koji se menjaju mesta tokom deploy-a. Za to je napisan servis za deploy koji je komunicirao s KV. U nekom trenutku, podaci iz KV su nestali. Obnovili smo ih iz pam\u0107enja, ali s nizom gre\u0161aka. Kao posledica, prilikom postavljanja, optere\u0107enje na upstreamima raspodeljeno je nerazmjerno, a mi smo dobili mnogo gre\u0161aka 502 zbog preoptere\u0107enja backendova po CPU-u. Na kraju smo pre\u0161li sa Consul KV na Postgres, odakle ih vi\u0161e nije tako lako ukloniti.\u00a0\u00a0<\/p>\n<p><b>Zaklju\u010dci:<br \/>\n<\/b><\/p>\n<ul>\n<li>Servisi bez bilo kakve autorizacije ne bi trebali sadr\u017eavati kriti\u010dne podatke za rad web stranice. Na primer, ako nemate autorizaciju u ES \u2014 bolje bi bilo zabraniti pristup na nivou mre\u017ee od svuda gde nije potreban, ostaviti samo neophodne, kao i postaviti action.destructive_requires_name: true.<\/li>\n<li>Ve\u017ebajte mehanizam rezervnog kopiranja i obnove unapred. Na primer, unapred napravite skriptu (recimo, u Pythonu) koja umije da pravi i rezervne kopije i da obnavlja.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Prvo mesto \u2014 \"Kapetan nevidljivost\"\u00a0<\/h4>\n<p>\nU nekom trenutku primijetili smo nerazmjerno optere\u0107enje na upstream nginx-u kada je u backendu bilo vi\u0161e od 10 servera. Budu\u0107i da je round-robin usmjeravao zahtjeve od prvog do posljednjeg upstream-a redoslijedom, a svaki reload nginx-a je po\u010dinjao ispo\u010detka, prvim upstream-ima uvijek je stizalo vi\u0161e zahtjeva nego ostalima. Kao posljedica, oni su radili sporije, a cijeli je sajt patio. To je postajalo sve primjetnije s pove\u0107anjem koli\u010dine prometa. Samo a\u017euriranje nginx-a za uklju\u010divanje random opcije nije bio dovoljno \u2014 trebalo je prepravljati tonu lua koda koji nije radio na verziji 1.15 (u to vrijeme). Morali smo patchirati na\u0161 nginx 1.14.2, dodav\u0161i podr\u0161ku za random. To je rije\u0161ilo problem. Ova gre\u0161ka pobje\u0111uje u nominaciji \"kapetan nejasno\u0107e.\"<\/p>\n<p><b>Zaklju\u010dci:<\/b><\/p>\n<p>Bilo je jako zanimljivo i uzbudljivo istra\u017eivati ovu gre\u0161ku).\u00a0<\/p>\n<ul>\n<li>Postavite monitoring tako da poma\u017ee brzo otkrivanje sli\u010dnih fluktuacija. Na primjer, mo\u017eete koristiti ELK za pra\u0107enje rps-a za svaki backend svakog upstream-a, prate\u0107i njihovo vrijeme odgovora s gledi\u0161ta nginx-a. U ovom slu\u010daju to nam je pomoglo da otkrijemo problem.\u00a0<\/li>\n<\/ul>\n<p>\nVe\u0107ina gre\u0161aka mogla bi se izbje\u0107i uz pomniji pristup onome \u0161to radite. Uvijek treba imati na umu Murphyjev zakon:\u00a0<i>Sve \u0161to mo\u017ee po\u0107i po zlu po\u0107i \u0107e po zlu, <\/i>i graditi komponente vode\u0107i se time.\u00a0<br \/>\n<br \/>Izvor: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/cian\/blog\/499542\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430!\u00a0 \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d. \u041e\u0434\u043d\u043e\u0439 \u0438\u0437 \u043c\u043e\u0438\u0445 \u043e\u0431\u044f\u0437\u0430\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u0430 \u0438\u043d\u0446\u0438\u0434\u0435\u043d\u0442\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439 \u043d\u0430 \u043f\u0440\u043e\u0434\u0435, \u0434\u043e \u043d\u0443\u043b\u044f. \u0422\u043e, \u043e \u0447\u0435\u043c \u043f\u043e\u0439\u0434\u0435\u0442 \u0440\u0435\u0447\u044c \u0434\u0430\u043b\u0435\u0435, \u043f\u0440\u0438\u043d\u0435\u0441\u043b\u043e \u043d\u0430\u043c \u043c\u043d\u043e\u0433\u043e \u0431\u043e\u043b\u0438, \u0438 \u0446\u0435\u043b\u044c \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u2014 \u043d\u0435 \u0434\u0430\u0442\u044c \u0434\u0440\u0443\u0433\u0438\u043c \u043b\u044e\u0434\u044f\u043c \u043f\u043e\u0432\u0442\u043e\u0440\u0438\u0442\u044c \u043d\u0430\u0448\u0438\u0445 \u043e\u0448\u0438\u0431\u043e\u043a \u0438\u043b\u0438 \u0445\u043e\u0442\u044f \u0431\u044b \u043c\u0438\u043d\u0438\u043c\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438\u0445 \u0432\u043b\u0438\u044f\u043d\u0438\u0435.\u00a0 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80032,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"hr_HR\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/top-fakapov-czian\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-02T11:42:49+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Top gre\u0161aka Ciana | ProHoster","description":"Svima dobar dan! Zovem se Nikita, tim lider tima in\u017eenjera Ciana.","canonical_url":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/top-fakapov-czian","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"hr_HR","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0422\u043e\u043f \u0444\u0430\u043a\u0430\u043f\u043e\u0432 \u0426\u0438\u0430\u043d | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u0434\u043e\u0431\u0440\u0430! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041d\u0438\u043a\u0438\u0442\u0430, \u044f \u0442\u0438\u043c\u043b\u0438\u0434 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u0426\u0438\u0430\u043d.","og:url":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/top-fakapov-czian","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-02T11:42:49+00:00","article:modified_time":"2020-05-02T11:42:49+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80031","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:47:44","updated":"2026-02-09 16:50:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts\/80031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/comments?post=80031"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts\/80031\/revisions"}],"predecessor-version":[{"id":158728,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts\/80031\/revisions\/158728"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/media\/80032"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/media?parent=80031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/categories?post=80031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/tags?post=80031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}