<\/a><\/noindex>Ne ndoq\u00ebm kalimin, u shfaq\u00ebn detyra t\u00eb tjera - t\u00eb konfiguroni backupet dhe monitorimin. N\u00eb at\u00eb moment CTO i k\u00ebsaj kompanie u largua p\u00ebr n\u00eb nj\u00eb projekt tjet\u00ebr, duke na l\u00ebn\u00eb n\u00eb komand\u00eb nj\u00eb nga t\u00eb tij\u00ebt - Leonidin. L\u00ebnda kishte nj\u00eb ndihm\u00eb t\u00eb moderate. Nj\u00eb zhvillues i thjesht\u00eb, t\u00eb cilin papritur e caktuan si kreun p\u00ebr ClickHouse. Dukej se ishte em\u00ebrimi i tij i par\u00eb p\u00ebr t\u00eb udh\u00ebhequr di\u00e7ka, dhe nga nderi i habitsh\u00ebm, i doli nj\u00eb sindrom\u00eb e famshme. <\/p>\n
Bashk\u00eb filluam t\u00eb punonim p\u00ebr backupet. Un\u00eb sugjeroja q\u00eb t\u00eb b\u00ebjm\u00eb backup t\u00eb dh\u00ebnash t\u00eb origjin\u00ebs menj\u00ebher\u00eb. Thjesht t'i merrnim, t'i ziponim dhe elegantisht t'i d\u00ebrgonim n\u00eb nj\u00eb S3. T\u00eb dh\u00ebnat e origjin\u00ebs jan\u00eb ar. Kishte edhe nj\u00eb mund\u00ebsi tjet\u00ebr - t\u00eb b\u00ebjm\u00eb backup t\u00eb tabelave vet\u00eb n\u00eb ClickHouse, p\u00ebrmes ngrirjes dhe kopjimit. Por Leonid kishte nj\u00eb zgjidhje t\u00eb tij\u00ebn.<\/p>\n
Ai shpalli se na duhej nj\u00eb grup i dyt\u00eb ClickHouse. Dhe nga tani do t\u00eb shkruanim t\u00eb dh\u00ebna n\u00eb dy grupe - kryesorin dhe at\u00eb t\u00eb backup-it. I thash\u00eb, Leonid, kjo nuk do t\u00eb jet\u00eb backup - por nj\u00eb replik\u00eb aktive. Dhe n\u00ebse t\u00eb dh\u00ebnat fillojn\u00eb t\u00eb humbasin n\u00eb prodhim, do t\u00eb ket\u00eb t\u00eb nj\u00ebjt\u00ebn gj\u00eb n\u00eb backup-in t\u00ebnd. <\/p>\n
Por Leonid ishte fort i vendosur dhe nuk pranoi t\u00eb d\u00ebgjonte argumentet e mia. Ne u grind\u00ebm gjat\u00eb n\u00eb chat, por asgj\u00eb p\u00ebr t\u00eb b\u00ebr\u00eb - Leonid ishte n\u00eb timon t\u00eb projektit, ne ishim thjesht djem t\u00eb pun\u00ebsuar nga rruga.<\/p>\n
Ne kemi monitoruar gjendjen e klasterit dhe merret vet\u00ebm nj\u00eb tarif\u00eb p\u00ebr pun\u00ebn e administratoreve. Administrim i past\u00ebr i ClickHouse pa prekur t\u00eb dh\u00ebnat. Klasteri ishte n\u00eb dispozicion, disqet ishin n\u00eb gjendje t\u00eb mir\u00eb, nodet ishin n\u00eb rregull. <\/p>\n
Ne ende nuk e dyshonim se e mor\u00ebm k\u00ebt\u00eb porosi p\u00ebr shkak t\u00eb nj\u00eb keqkuptimi t\u00eb tmerrsh\u00ebm brenda ekipit t\u00eb tyre.<\/h3>\n \nMenaxheri ishte i shqet\u00ebsuar q\u00eb ClickHouse po punonte ngadal\u00eb dhe ndonj\u00ebher\u00eb humbeshin t\u00eb dh\u00ebnat. Ai i dha CTO-s\u00eb s\u00eb tij detyr\u00ebn p\u00ebr t\u00eb zbuluar situat\u00ebn. Ai zbuloi se si mundej dhe arriti n\u00eb p\u00ebrfundimin se thjesht duhej automatizuar ClickHouse \u2014 dhe \u00e7a u b\u00eb. Por, si\u00e7 doli n\u00eb pah m\u00eb von\u00eb, ata nuk kishin nevoj\u00eb p\u00ebr nj\u00eb ekip dev ops.<\/p>\n
T\u00eb gjitha k\u00ebto u zbuluan shum\u00eb-ngadal\u00eb. Dhe m\u00eb e hidhur, kjo ndodhi n\u00eb dit\u00eblindjen time.<\/p>\n
\nE premte, mbr\u00ebmje. Kisha rezervuar nj\u00eb tavolin\u00eb n\u00eb barin tim t\u00eb preferuar t\u00eb ver\u00ebs dhe thirra shok\u00ebt. <\/p>\n
Pik\u00ebrisht para se t\u00eb dalim, na erdhi nj\u00eb detyr\u00eb p\u00ebr t\u00eb b\u00ebr\u00eb nj\u00eb alter, ne e kryem, gjith\u00e7ka ishte n\u00eb rregull. Alteri kaloi, ClickHouse e konfirmoi. Ne tashm\u00eb ishim mbledhur p\u00ebr n\u00eb bar, kur na shkruan se nuk kishte t\u00eb dh\u00ebna t\u00eb mjaftueshme. E llogarit\u00ebm \u2014 duket se kishte gjith\u00e7ka. Dhe shkuam t\u00eb festojm\u00eb.<\/p>\n
N\u00eb restorant ishte shum\u00eb e zhurmshme si zakonisht n\u00eb t\u00eb premten. Pas porosis\u00eb s\u00eb pijeve dhe ushqimit, ne u shplodh\u00ebm n\u00eb divan\u00eb. Gjat\u00eb gjith\u00eb k\u00ebsaj kohe, Slack-u im ngadal\u00eb u mbush me mesazhe. Kishin shkruar di\u00e7ka p\u00ebr munges\u00ebn e t\u00eb dh\u00ebnave. Mendoja \u2014 m\u00ebngjesi \u00ebsht\u00eb m\u00eb i men\u00e7ur se mbr\u00ebmja. Sidomos p\u00ebr dit\u00ebn e sotme.<\/p>\n
Af\u00ebrsisht n\u00eb11:00 filluan t\u00eb telefonojn\u00eb. Ishte menaxheri i kompanis\u00eb... \"Ndoshta vendosi t\u00eb m\u00eb uroj\u00eb\", mendova shum\u00eb me pasiguri dhe e mora telefonin.<\/p>\n
Dhe d\u00ebgjova di\u00e7ka n\u00eb frym\u00ebn e: \"Ju humb\u00ebt t\u00eb dh\u00ebnat tona! Po ju paguaj, por asgj\u00eb nuk funksionon! Ju ishit p\u00ebrgjegj\u00ebs p\u00ebr backup-et dhe nuk keni b\u00ebr\u00eb asgj\u00eb! Hidhini duar!\" \u2014 vet\u00ebm akoma m\u00eb ashp\u00ebr.<\/p>\n
\u2014 E di \u00e7far\u00eb, shko t\u00eb lutem! Sot \u00ebsht\u00eb dit\u00eblindja ime, dhe tani do t\u00eb pij\u00eb, e jo t\u00eb merrem me nd\u00ebrtimet tuaja t\u00eb varfra me dijenin\u00eb e djemve tuaj!<\/p>\n
K\u00ebshtu q\u00eb un\u00eb nuk e thash\u00eb k\u00ebt\u00eb. N\u00eb vend t\u00eb k\u00ebsaj, nxora laptopin dhe fillova pun\u00ebn.<\/p>\n
Jo, un\u00eb isha n\u00eb stres, isha tmerr\u00ebsisht n\u00ebn stres! D\u00ebrgova n\u00eb chat ndjenja t\u00eb hidhura \"t\u00eb thash\u00eb\" \u2014 sepse backup-i q\u00eb nuk ishte as backup, sigurisht q\u00eb nuk shp\u00ebtoi asgj\u00eb. <\/p>\n
Ne me djemt\u00eb e tjer\u00eb gjet\u00ebm nj\u00eb m\u00ebnyr\u00eb p\u00ebr t\u00eb ndaluar regjistrimin manualisht dhe p\u00ebr t\u00eb kontrolluar gjith\u00e7ka. Me t\u00eb v\u00ebrtet\u00eb u siguruam q\u00eb nj\u00eb pjes\u00eb e t\u00eb dh\u00ebnave nuk po shkruhej.<\/p>\n
Ndaluam regjistrimin, llogarita numrin e ngjarjeve q\u00eb ishin n\u00eb dispozicion p\u00ebr nj\u00eb dit\u00eb. Shtuam edhe t\u00eb dh\u00ebna, nga t\u00eb cilat vet\u00ebm nj\u00eb e treta nuk u regjistrua. Tre shard-e me 2 replika. Shton 100,000 rreshta \u2014 33,000 nuk regjistrohen.<\/p>\n
Ishte nj\u00eb kaos i plot\u00eb. \u00c7do kush ia d\u00ebrgonte nj\u00ebri-tjetrit n\u00eb ferr me radh\u00eb: i pari q\u00eb shkoi ishte L\u00ebnja, pas tij shkova un\u00eb dhe themeluesi i kompanis\u00eb. Vet\u00ebm CTO q\u00eb sapo ishte bashkuar p\u00ebrpiqej t\u00eb qet\u00ebsoj\u00eb thirrjet tona me britma dhe biseda n\u00eb k\u00ebrkim t\u00eb zgjidhjes s\u00eb problemit. <\/p>\n
Askush nuk e kuptonte \u00e7far\u00eb po ndodhte n\u00eb t\u00eb v\u00ebrtet\u00eb.<\/h3>\n \nNe me djemt\u00eb ishim t\u00eb habitur kur kuptuam se nj\u00eb e treta e t\u00eb dh\u00ebnave jo vet\u00ebm q\u00eb nuk regjistroheshin \u2014 ato humbnin! Doli se rregulli n\u00eb kompani ishte i till\u00eb: pas shtimit, t\u00eb dh\u00ebnat fshiheshin pa kthim, ngjarjet humbnin grupe. E imagjinova si Serxhio do ta shnd\u00ebrronte t\u00eb gjith\u00eb k\u00ebt\u00eb n\u00eb para t\u00eb humbura.<\/p>\n
Dit\u00eblindja ime gjithashtu p\u00ebrfundoi n\u00eb plehra. Ne ishim ulur n\u00eb bar dhe po gjeneronim ide, duke u p\u00ebrpjekur t\u00eb zgjidhim enigm\u00ebn e hedhur. Arsyet e r\u00ebnies s\u00eb ClickHouse nuk ishin t\u00eb qarta. Ndoshta ishte rrjeti, ndoshta ishte p\u00ebr arsyet e Linux-it. Nj\u00eb mori hipotezash dol\u00ebn.<\/p>\n
Nuk kisha b\u00ebr\u00eb betim si zhvillues, por ishte e pamoralshme t\u00eb lija djemt\u00eb n\u00eb cepin tjet\u00ebr t\u00eb telit \u2014 edhe n\u00ebse ata na fajsonin p\u00ebr gjith\u00e7ka. Isha 99% i sigurt se problemi nuk ishte n\u00eb zgjidhjet tona, nuk ishte nga ana jon\u00eb. 1% mund\u00ebsie q\u00eb ne t\u00eb kishim gabuar m\u00eb brente me shqet\u00ebsim. Por pavar\u00ebsisht se n\u00eb an\u00ebn e kujt ishte katastrofa \u2014 duhej ta zgjidhnim. T\u00eb l\u00eb klient\u00ebt, pavar\u00ebsisht se kush ishin, me nj\u00eb rrjedhje kaq t\u00eb tmerrshme t\u00eb t\u00eb dh\u00ebnave \u2014 \u00ebsht\u00eb shum\u00eb e ashp\u00ebr.<\/p>\n
Derisa erdhi tre pasdite, punuam n\u00eb tavolin\u00ebn e restorantit. Shtonim ngjarje, insert select \u2014 dhe filluam t\u00eb mbushnim boshll\u00ebqet. Kur humb t\u00eb dh\u00ebna, e b\u00ebn k\u00ebshtu \u2014 merr t\u00eb dh\u00ebnat mesatare nga dit\u00ebt e kaluara dhe i shton ato n\u00eb vendet e humbura.<\/p>\n
Pas tre pasdite, shkova me nj\u00eb shok n\u00eb sht\u00ebpin\u00eb time, porosita disa birra nga dyqani. Un\u00eb isha ulur me laptopin dhe problemet e ClickHouse-it, shoku m\u00eb tregonte di\u00e7ka. N\u00eb fund, pas nj\u00eb ore, ai u ofendua q\u00eb un\u00eb punoja dhe nuk pim\u00eb birr\u00eb s\u00eb bashku, dhe u largua. Klasikja \u2014 ishte mik i devops-it p\u00ebr nj\u00eb moment.<\/p>\n
N\u00eb or\u00ebn 6 t\u00eb m\u00ebngjesit, krijova p\u00ebrs\u00ebri tabel\u00ebn nga e para, dhe t\u00eb dh\u00ebnat filluan t\u00eb ngarkoheshin. \u00c7do gj\u00eb filloi t\u00eb funksionoj\u00eb pa humbje.<\/p>\n
<\/p>\n
M\u00eb pas ishte e v\u00ebshtir\u00eb. \u00c7do kush akuzonte nj\u00ebri-tjetrin p\u00ebr humbjen e t\u00eb dh\u00ebnave. Sikur t\u00eb ndodhte nj\u00eb bug i ri, jam i sigurt se do t\u00eb fillonte nj\u00eb shk\u00ebmbim zjarri.<\/h3>\n \nN\u00eb k\u00ebto p\u00ebrplasje, ne p\u00ebrfundimisht filluam t\u00eb kuptojm\u00eb \u2014 n\u00eb kompani ishin menduar se ne ishim ata djem q\u00eb punonin me t\u00eb dh\u00ebnat dhe ndihmonin me struktur\u00ebn e tabelave. Ata ngat\u00ebrruan administrator\u00ebt me DBAs. Dhe erdh\u00ebn t\u00eb na pyesnin, p\u00ebr asgj\u00eb m\u00eb pak se si administrator\u00eb.<\/p>\n
Ankesa e tyre kryesore \u2014 p\u00ebrse, o Zot, ishit p\u00ebrgjegj\u00ebs p\u00ebr back-upe dhe nuk i b\u00ebt\u00eb si\u00e7 duhet, humb\u00ebt t\u00eb dh\u00ebnat. Dhe gjith\u00eb kjo me shprehje t\u00eb ashpra.<\/p>\n
M\u00eb duhej drejt\u00ebsia. E gjeta bised\u00ebn dhe bashk\u00ebngjita t\u00eb gjitha screenshot-et, ku Leonidi me forc\u00eb k\u00ebrkonte t\u00eb b\u00ebhej ky backup, q\u00eb ishte b\u00ebr\u00eb. CTO-ja i tyre u vu n\u00eb an\u00ebn ton\u00eb pas telefonat\u00ebs sime. M\u00eb pas, Lenya pranoi fajin e tij.<\/p>\n
Kreu i kompanis\u00eb, nga ana tjet\u00ebr, nuk donte t\u00eb faj\u00ebsonte t\u00eb tijt\u00eb. Screenshot-et dhe fjal\u00ebt nuk e preknin. Ai mendonte se, meqen\u00ebse ne ishim ekspert\u00eb k\u00ebtu, duhej t\u00eb kishim bindur t\u00eb gjith\u00eb dhe insistuar p\u00ebr vendimin ton\u00eb. Dukeshin se detyra jon\u00eb ishte t\u00eb m\u00ebsonim Leny\u00ebn dhe gjithashtu n\u00eb anash t\u00eb tij, t\u00eb em\u00ebruar drejtues i projektit, t\u00eb arrinim te kryesori dhe t'i tregonim t\u00eb gjitha dyshimet tona mbi konceptin e back-upeve. <\/p>\n
Chat-i l\u00ebshonte urrejtje, agresion t\u00eb fsheht\u00eb dhe t\u00eb hapur. Nuk dija si t\u00eb sillem. Gjith\u00e7ka arriti n\u00eb nj\u00eb pik\u00eb t\u00eb verb\u00ebr. Dhe k\u00ebtu m\u00eb k\u00ebshilluan m\u00ebnyr\u00ebn m\u00eb t\u00eb thjesht\u00eb \u2014 t\u00eb shkruaja n\u00eb privat drejtorit dhe t\u00eb nd\u00ebrtoja nj\u00eb takim me t\u00eb. Vasja, njer\u00ebzit n\u00eb jet\u00ebn reale nuk jan\u00eb kaq t\u00eb guximsh\u00ebm si n\u00eb chat. N\u00eb mesazhin tim, shefi u p\u00ebrgjigj: erdha, pa problem.<\/p>\n
\nIshte takimi m\u00eb i \u00e7uditsh\u00ebm n\u00eb karrier\u00ebn time. Aleati im nga klienti \u2014 CTO \u2014 nuk mundi t\u00eb gjente koh\u00eb. N\u00eb takim po shkova te shefi dhe Leonidi.<\/p>\n
P\u00ebrs\u00ebri e p\u00ebrs\u00ebrisja n\u00eb mendjen time dialogun ton\u00eb t\u00eb mundsh\u00ebm. Arrita t\u00eb vij shum\u00eb para, gjysm\u00eb ore p\u00ebrpara. Filloi nervozizmi, kam tymosur 10 cigare. E dija, gjith\u00e7ka \u2014 isha, p\u00ebrfundimisht, vet\u00ebm. Nuk do t\u00eb mund t'i bindja ata. Dhe nisa p\u00ebr n\u00eb ashensor.<\/p>\n
Nd\u00ebrsa po ngjitesha, sa e ndeza zippo-n, e thava.<\/p>\n
N\u00eb fund, Leonidi nuk ishte n\u00eb takim. Dhe biseduam mrekullisht p\u00ebr gjith\u00e7ka me kryesori! Sergey m\u00eb tregoi p\u00ebr dhimbjen e tij. Ai nuk donte t\u00eb \"automatizonte ClickHouse\" \u2014 ai donte, \"q\u00eb k\u00ebrkesat t\u00eb funksiononin\".<\/p>\n
Nuk pash\u00eb nj\u00eb demon, por nj\u00eb djal\u00eb t\u00eb mir\u00eb, q\u00eb ishte i shqet\u00ebsuar p\u00ebr biznesin e tij, i angazhuar n\u00eb pun\u00eb 24\/7. Chat-i shpesh na paraqet ngat\u00ebrrestar\u00ebt, tradh\u00ebtar\u00ebt dhe budallenjt\u00eb. Por n\u00eb jet\u00ebn reale, ata jan\u00eb njer\u00ebz si ty. <\/p>\n
Sergjeu nuk kishte nevoj\u00eb p\u00ebr dy devops t\u00eb angazhuar. Problemi q\u00eb kishin ndodhur ishte m\u00eb i madh. <\/p>\n
I said that I could solve his problems \u2014 it's just a completely different job, and I have an acquaintance who is a DBA for it. If we had known from the beginning that this was for them, we would have avoided a lot. It\u2019s late, but we understood that the problem lay in poor data handling and not in the infrastructure. <\/p>\n
We shook hands, our fee was raised two and a half times, but on the condition \u2014 I take all the headaches with their data and ClickHouse for myself. In the elevator, I contacted that very DBA Max and brought him on board. We had to restructure the entire cluster.<\/p>\n
\nThere was a lot of junk in the accepted project. Starting with the aforementioned \u2018backup\u2019. It turned out that this \u2018backup\u2019 cluster was not isolated. They were testing everything on it, sometimes even putting things into production.<\/p>\n
The in-house developers created their own custom \u2018data inserter\u2019. It worked like this: it batched files, ran a script, and poured data into a table. But the main problem was that a huge amount of data was accepted for one simple query. The query was joining data by the second. All of this was for one number \u2014 the sum for the day.<\/p>\n
The in-house developers incorrectly used the analytics tool. They would go into Grafana and write their royal query. It exported data for 2 weeks. The beautiful graph was produced. But in reality, the data query was running every 10 seconds. All this piled up in the queue, as ClickHouse simply couldn\u2019t handle the processing. That\u2019s where the main reason lay. In Grafana, nothing worked, requests queued up, and outdated data constantly came in.<\/p>\n
We reconfigured the cluster, redesigned the insert. The in-house developers rewrote their \u2018inserter\u2019, and it began sharding data correctly.<\/p>\n
Max conducted a complete audit of the infrastructure. He outlined a plan for transitioning to a full-fledged backend. But this did not satisfy the company. They expected Max to reveal a magical secret that would allow them to work the old way, but effectively. Leonid was still responsible for the project, having learned nothing. From all that was proposed, he chose his alternative again. As always, it was the most selective... bold decision. Leonid believed that his company had a special path. A thorny one full of icebergs.<\/p>\n
Actually, that's how we parted ways \u2014 we did what we could.<\/p>\n
\nMe t\u00eb shumt\u00eb p\u00ebrvojash, t\u00eb nxitur nga ky tregim, ne hap\u00ebm biznesin ton\u00eb dhe formuam disa parime p\u00ebr veten. Tani ne kurr\u00eb nuk fillojm\u00eb pun\u00eb ashtu si\u00e7 e b\u00ebm\u00eb at\u00ebher\u00eb.<\/p>\n
Dibeyshiku Max pas k\u00ebtij projekti na u bashkua, dhe ne akoma punojm\u00eb shum\u00eb mir\u00eb s\u00eb bashku. Rastin me ClickHouse na m\u00ebsoi t\u00eb kryejm\u00eb nj\u00eb audit t\u00eb plot\u00eb dhe t\u00eb holl\u00ebsish\u00ebm t\u00eb infrastruktur\u00ebs para se t\u00eb fillojm\u00eb pun\u00ebn. Ne kuptojm\u00eb si funksionon gjith\u00e7ka dhe vet\u00ebm pastaj marrim detyrat. Dhe n\u00ebse m\u00eb par\u00eb do t\u00eb ishim hedhur direkt p\u00ebr t\u00eb sh\u00ebrbyer infrastruktur\u00ebn, tani s\u00eb pari b\u00ebjm\u00eb nj\u00eb projekt nj\u00eb her\u00eb, i cili ndihmon p\u00ebr t\u00eb kuptuar se si ta sjellim at\u00eb n\u00eb gjendje pune. <\/p>\n
Po, ne i shmangim projektet me infrastruktur\u00eb t\u00eb dob\u00ebt. Edhe n\u00ebse jan\u00eb p\u00ebr shum\u00eb para, edhe n\u00ebse p\u00ebr miq\u00ebsi. T\u00eb menaxhojm\u00eb projekte t\u00eb s\u00ebmura nuk \u00ebsht\u00eb fitimprur\u00ebse. Nd\u00ebrgjegj\u00ebsimi p\u00ebr k\u00ebt\u00eb na ndihmoi t\u00eb rritemi. Ose nj\u00eb projekt nj\u00eb her\u00eb p\u00ebr t\u00eb sjell\u00eb infrastruktur\u00ebn n\u00eb rregull dhe pastaj nj\u00eb kontrat\u00eb p\u00ebr sh\u00ebrbim, ose thjesht ikim. M\u00ebnyr\u00eb kalimi p\u00ebrpara nj\u00eb ajsbergu tjet\u00ebr.<\/p>\n
P.S. Prandaj, n\u00ebse keni pyetje p\u00ebr infrastrukturen tuaj, mos hezitoni t\u00eb lini nj\u00eb k\u00ebrkes\u00eb<\/b><\/a><\/noindex>.<\/p>\n Ne kemi 2 auditime falas n\u00eb muaj, ndoshta projekti juaj do t\u00eb jet\u00eb n\u00eb mesin e tyre.<\/i>habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u0412\u043e\u0442 \u0438\u0441\u0442\u043e\u0440\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043d\u0430\u0432\u0441\u0435\u0433\u0434\u0430 \u043f\u0435\u0440\u0435\u0432\u0435\u0440\u043d\u0443\u043b\u0430 \u043c\u043e\u0439 \u043f\u043e\u0434\u0445\u043e\u0434 \u043a \u0440\u0430\u0431\u043e\u0442\u0435 \u0434\u0435\u0432\u043e\u043f\u0441\u0430. \u0415\u0449\u0435 \u0432 \u0434\u043e\u043a\u043e\u0432\u0438\u0434\u043d\u044b\u0435 \u0432\u0440\u0435\u043c\u0435\u043d\u0430, \u0437\u0430\u0434\u043e\u043b\u0433\u043e-\u0437\u0430\u0434\u043e\u043b\u0433\u043e \u0434\u043e \u043d\u0438\u0445, \u043a\u043e\u0433\u0434\u0430 \u043c\u044b \u0441 \u043f\u0430\u0440\u043d\u044f\u043c\u0438 \u0442\u043e\u043b\u044c\u043a\u043e \u0437\u0430\u0434\u0443\u043c\u044b\u0432\u0430\u043b\u0438 \u0441\u0432\u043e\u0435 \u0434\u0435\u043b\u043e \u0438 \u0444\u0440\u0438\u043b\u0430\u043d\u0441\u0438\u043b\u0438 \u043d\u0430 \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u044b\u0445 \u0437\u0430\u043a\u0430\u0437\u0430\u0445, \u043c\u043d\u0435 \u0432 \u0442\u0435\u043b\u0435\u0433\u0443 \u0443\u043f\u0430\u043b\u043e \u043e\u0434\u043d\u043e \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0435\u043d\u0438\u0435. \u041a\u043e\u043c\u043f\u0430\u043d\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043d\u0430\u043f\u0438\u0441\u0430\u043b\u0430, \u0437\u0430\u043d\u0438\u043c\u0430\u043b\u0430\u0441\u044c \u0430\u043d\u0430\u043b\u0438\u0442\u0438\u043a\u043e\u0439 \u0434\u0430\u043d\u043d\u044b\u0445. \u0415\u0436\u0435\u0434\u043d\u0435\u0432\u043d\u043e \u043e\u043d\u0430 \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u043b\u0430 \u0442\u044b\u0441\u044f\u0447\u0438 \u0437\u0430\u043f\u0440\u043e\u0441\u043e\u0432. \u041a \u043d\u0430\u043c \u043e\u043d\u0438 \u043f\u0440\u0438\u0448\u043b\u0438 \u0441\u043e \u0441\u043b\u043e\u0432\u0430\u043c\u0438: \u0440\u0435\u0431\u044f\u0442, \u0443 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":96000,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-95999","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"Nd\u00ebrsa](\"\/wp-content\/uploads\/2020\/10\/f67882cb5b565cb2efef20f68bbef1d5.jpeg\")