{"id":53590,"date":"2019-12-05T00:00:00","date_gmt":"2019-12-04T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-my-v-tsian-ukroshhali-terabajty-logov"},"modified":"2020-02-18T14:01:30","modified_gmt":"2020-02-18T11:01:30","slug":"kak-my-v-tsian-ukroshhali-terabajty-logov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-my-v-tsian-ukroshhali-terabajty-logov","title":{"rendered":"Si e sulmuam ne n\u00eb CIAN terabajt t\u00eb logeve","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Si
\n
\nP\u00ebrsh\u00ebndetje t\u00eb gjith\u00ebve, un\u00eb quhem Aleksand\u00ebr, punoj n\u00eb CIAN si inxhinier dhe merrem me administrimin e sistemeve dhe automatizimin e proceseve infrastrukturore. N\u00eb komentet e nj\u00ebrit prej artikujve t\u00eb kaluar, na u k\u00ebrkua t\u00eb flasim mbi burimin prej 4 TB log-eve n\u00eb dit\u00eb dhe \u00e7far\u00eb b\u00ebjm\u00eb me to. Po, kemi shum\u00eb log-e dhe p\u00ebr procesimin e tyre \u00ebsht\u00eb krijuar nj\u00eb grup infrastrukture i ve\u00e7ant\u00eb q\u00eb na lejon t\u00eb zgjidhim problemet n\u00eb m\u00ebnyr\u00eb operative. N\u00eb k\u00ebt\u00eb artikull, do t\u00eb flas p\u00ebr m\u00ebnyr\u00ebn se si e kemi adaptuar at\u00eb gjat\u00eb nj\u00eb viti p\u00ebr t\u00eb punuar me fluksin n\u00eb rritje t\u00eb t\u00eb dh\u00ebnave.
\n<\/a><\/noindex><\/p>\n

Si e nis\u00ebm<\/h3>\n

\n\"Si
\n
\nGjat\u00eb disa viteve t\u00eb fundit, ngarkesa n\u00eb cian.ru u rrit shum\u00eb shpejt, dhe n\u00eb tremujorin e tret\u00eb t\u00eb vitit 2018, vizitat n\u00eb burim arrit\u00ebn 11.2 milion p\u00ebrdorues unik\u00eb n\u00eb muaj. N\u00eb ato momente kritike, ne humbnim deri n\u00eb 40% t\u00eb log-eve, duke e b\u00ebr\u00eb t\u00eb pamundur t\u00eb merreshim me incidentet n\u00eb koh\u00eb dhe duke shpenzuar shum\u00eb koh\u00eb dhe energji p\u00ebr t'i zgjidhur ato. Po ashtu, shpesh nuk mund t\u00eb gjenim shkakun e problemit dhe ai rip\u00ebrs\u00ebritej pas nj\u00eb koh\u00eb. Kjo ishte nj\u00eb ferr prej t\u00eb cilit na duhej t\u00eb b\u00ebnim di\u00e7ka.<\/p>\n

N\u00eb at\u00eb koh\u00eb, p\u00ebr ruajtjen e log-eve p\u00ebrdornim nj\u00eb grup prej 10 nodave t\u00eb dh\u00ebnash me ElasticSearch version 5.5.2 me konfigurime standarde t\u00eb indekseve. E implementuam at\u00eb m\u00eb shum\u00eb se nj\u00eb vit m\u00eb par\u00eb si nj\u00eb zgjidhje t\u00eb njohur dhe t\u00eb p\u00ebrballueshme: at\u00ebher\u00eb fluksi i log-eve nuk ishte aq i madh, prandaj nuk kishte kuptim t\u00eb krijonim konfigurime jo standarde.\u00a0<\/p>\n

P\u00ebr procesimin e log-eve t\u00eb ardhura, Logstash funksiononte n\u00eb ports t\u00eb ndryshme n\u00eb pes\u00eb koordinator\u00eb ElasticSearch. Nj\u00eb indeks, pavar\u00ebsisht nga madh\u00ebsia, ishte p\u00ebrb\u00ebr\u00eb nga pes\u00eb shard. Kemi organizuar rotacionin \u00e7do or\u00eb dhe ditor, k\u00ebshtu q\u00eb \u00e7do or\u00eb n\u00eb grupin ton\u00eb shfaqeshin rreth 100 shard t\u00eb rinj. Nd\u00ebrsa log-eve nuk ishin shum\u00eb, grupi p\u00ebrballonte situat\u00ebn dhe askush nuk e vuri re konfigurimin e tij.\u00a0<\/p>\n

Problemet e rritjes s\u00eb shpejt\u00eb<\/h3>\n

\nV\u00ebllimi i log-eve t\u00eb gjeneruara po rritej shum\u00eb shpejt, pasi dy procese sigurisht po nd\u00ebrfuten. Nga nj\u00ebra an\u00eb, numri i p\u00ebrdoruesve t\u00eb sh\u00ebrbimit po rritej gjithnj\u00eb e m\u00eb shum\u00eb. Nga ana tjet\u00ebr, filluam t\u00eb kalojm\u00eb aktivisht n\u00eb nj\u00eb arkitektur\u00eb mikrosh\u00ebrbimesh duke ndar\u00eb monolit\u00ebt tan\u00eb t\u00eb vjet\u00ebr n\u00eb C# dhe Python. Disa dhjet\u00ebra mikrosh\u00ebrbime t\u00eb reja, q\u00eb z\u00ebvend\u00ebsuan pjes\u00eb t\u00eb monolitit, gjeneronin ndjesh\u00ebm m\u00eb shum\u00eb log-e p\u00ebr grupin infrastrukturore.\u00a0<\/p>\n

Sakt\u00ebsisht shkall\u00ebzimi na \u00e7oi n\u00eb nj\u00eb situat\u00eb ku klasteri u b\u00eb praktikisht i pakontroluesh\u00ebm. Kur logat filluan t\u00eb vijn\u00eb me shpejt\u00ebsi prej 20 mij\u00eb mesazhesh n\u00eb sekund\u00eb, rotacioni i shpesht\u00eb i padobish\u00ebm e rriti numrin e shard\u00ebve n\u00eb 6 mij\u00eb, nd\u00ebrsa nj\u00eb nyje kishte m\u00eb shum\u00eb se 600 shard\u00eb.\u00a0<\/p>\n

Kjo \u00e7onte n\u00eb probleme me ndarjen e memories, nd\u00ebrsa kur nj\u00eb nyje binte, ndodhte nj\u00eb transferim i menj\u00ebhersh\u00ebm i t\u00eb gjith\u00eb shard\u00ebve, duke shum\u00ebzuar trafikun dhe ngarkuar nyjet e tjera, gj\u00eb q\u00eb e b\u00ebnte thuajse t\u00eb pamundur regjistrimin e t\u00eb dh\u00ebnave n\u00eb klaster. Gjat\u00eb k\u00ebsaj periudhe, ne ishim pa loga. Dhe kur kishte problem me server<\/a> ne humbisnim 1\/10 t\u00eb klasterit n\u00eb thelb. Numri i madh i indekseve t\u00eb vogla e shtonte kompleksitetin.<\/p>\n

Pa loga, nuk kuptonim arsyet e incidentit dhe mund t\u00eb p\u00ebrballeshim her\u00ebt ose von\u00eb me t\u00eb nj\u00ebjtat probleme p\u00ebrs\u00ebri, dhe kjo n\u00eb ideologjin\u00eb ton\u00eb ekipore ishte e papranueshme, pasi t\u00eb gjith\u00eb mekanizmat tan\u00eb t\u00eb pun\u00ebs ishin t\u00eb dizajnuar p\u00ebr t\u00eb mos i p\u00ebrs\u00ebritur ndonj\u00ebher\u00eb t\u00eb nj\u00ebjtat probleme. P\u00ebr k\u00ebt\u00eb na duhej nj\u00eb v\u00ebllim i plot\u00eb logash dhe d\u00ebrgimi i tyre praktika n\u00eb koh\u00eb reale, pasi ekipi i inxhinier\u00ebve vigilent monitoronte alarmin jo vet\u00ebm nga metrikat, por edhe nga logat. P\u00ebr t\u00eb kuptuar shkall\u00ebn e problemit - n\u00eb at\u00eb koh\u00eb, v\u00ebllimi total i log\u00ebve ishte rreth 2 TB n\u00eb dit\u00eb.\u00a0<\/p>\n

Ne vendos\u00ebm nj\u00eb detyr\u00eb - t\u00eb eliminojm\u00eb plot\u00ebsisht humbjen e log\u00ebve dhe t\u00eb shkurtom\u00eb koh\u00ebn e d\u00ebrgimit n\u00eb klasterin ELK maks 15 minuta gjat\u00eb rasteve urgjente (n\u00eb k\u00ebt\u00eb shif\u00ebr m\u00eb von\u00eb mb\u00ebshteteshim si KPI t\u00eb brendsh\u00ebm).<\/p>\n

Mekanizmi i ri i rotacionit dhe nyjet hot-warm<\/h3>\n

\n\"Si
\n
\nNe filluam transformimin e klasterit me p\u00ebrdit\u00ebsimin e versionit ElasticSearch nga 5.5.2 n\u00eb 6.4.3. Klasteri yn\u00eb i versionit 5 p\u00ebrs\u00ebri ra, dhe vendos\u00ebm ta fikim dhe ta p\u00ebrdit\u00ebsojm\u00eb plot\u00ebsisht - nuk kishte loga. Pra, k\u00ebt\u00eb kalim e b\u00ebm\u00eb p\u00ebr vet\u00ebm disa or\u00eb.<\/p>\n

Transformimi m\u00eb i madh n\u00eb k\u00ebt\u00eb faz\u00eb ishte implementimi n\u00eb tri nodet me nj\u00eb koordin\u0430\u0442\u043e\u0440 si nj\u00eb tampon nd\u00ebrmjet\u00ebs Apache Kafka. Brokeri i mesazheve na \u00e7liroi nga humbja e log\u00ebve gjat\u00eb problemeve me ElasticSearch. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, ne shtuam 2 node n\u00eb kluster dhe kaluam n\u00eb arkitektur\u00ebn hot-warm me tri nodet \"t\u00eb nxehta\", t\u00eb vendosura n\u00eb shtylla t\u00eb ndryshme n\u00eb qend\u00ebr t\u00eb t\u00eb dh\u00ebnave. Atje ne redirektuam log\u00ebt q\u00eb nuk duhet t\u00eb humben kurr\u00eb \u2014 nginx, si dhe log\u00ebt e gabimeve t\u00eb aplikacioneve. Tek nodet e tjera shkonin log\u00ebt e dor\u00ebs s\u00eb dyt\u00eb \u2014 debug, warning, etj., dhe pas 24 or\u00ebve, log\u00ebt \"e r\u00ebnd\u00ebsishme\" nga nodet \"t\u00eb nxehta\" kalonin.<\/p>\n

P\u00ebr t\u00eb mos rritur numrin e indekseve t\u00eb vogla, ne kaluam nga rotacioni sipas koh\u00ebs n\u00eb mekanizmin e rollover. N\u00eb forume kishte shum\u00eb informacione se rotacioni sipas madh\u00ebsis\u00eb s\u00eb indeksit ishte shum\u00eb i pabesuesh\u00ebm, prandaj vendos\u00ebm t\u00eb p\u00ebrdorim rotacionin sipas numrit t\u00eb dokumenteve n\u00eb indeks. Ne analizuam \u00e7do indeks dhe regjistruam numrin e dokumenteve, pas t\u00eb cilit duhej t\u00eb aktivizohej rotacioni. K\u00ebshtu arrit\u00ebm madh\u00ebsin\u00eb optimale t\u00eb shard-it \u2014 jo m\u00eb shum\u00eb se 50 GB.\u00a0<\/p>\n

Optimizimi i klasterit<\/h3>\n

\n\"Si
\n
\nMegjithat\u00eb, ne nuk u \u00e7lir\u00ebm plot\u00ebsisht nga problemet. Fatkeq\u00ebsisht, ndiheshin ende indekse t\u00eb vogla: ato nuk arrinin volumet e caktuara, nuk rotacionoheshin dhe hiqeshin me pastrimin global t\u00eb indekseve m\u00eb t\u00eb vjetra se tri dit\u00eb, pasi ne e hoq\u00ebm rotacionin sipas dat\u00ebs. Kjo \u00e7oi n\u00eb humbje t\u00eb t\u00eb dh\u00ebnave p\u00ebr shkak se indeksi zhdukej plot\u00ebsisht nga klasteri, nd\u00ebrsa p\u00ebrpjekjet p\u00ebr t\u00eb shkruar n\u00eb nj\u00eb indeks q\u00eb nuk ekzistonte prishnin logjik\u00ebn e curator-it, q\u00eb ne p\u00ebrdornim p\u00ebr menaxhim. Alias-i p\u00ebr shkruarje transformohej n\u00eb indeks dhe prishte logjik\u00ebn e rollover-it, duke shkaktuar rritjen e pakontrolluar t\u00eb disa indekseve deri n\u00eb 600 GB.\u00a0<\/p>\n

P\u00ebr shembull, p\u00ebr konfigurimin e rotacionit:<\/p>\n

curator-elk-rollover.yaml\n\n---\nactions:\n  1:\n    action: rollover\n    options:\n      name: \"nginx_write\"\n      conditions:\n        max_docs: 100000000\n  2:\n    action: rollover\n    options:\n      name: \"python_error_write\"\n      conditions:\n        max_docs: 10000000\n<\/code><\/pre>\n
N\u00eb munges\u00eb t\u00eb rollover alias, ndodhte nj\u00eb gabim:<\/p>\n
ERROR     alias \"nginx_write\" not found.\nERROR     Failed to complete action: rollover.  : Unable to perform index rollover with alias \"nginx_write\".\n<\/code><\/pre>\n
Zgjidhjen e k\u00ebtij problemi e lan\u00eb p\u00ebr iteracionin e ardhsh\u00ebm dhe u mor\u00ebm me nj\u00eb \u00e7\u00ebshtje tjet\u00ebr: kaluam n\u00eb logjik\u00ebn pull t\u00eb pun\u00ebs s\u00eb Logstash-it, i cili merret me procesimin e log\u00ebve hyr\u00ebse (ndalimin e informacionit t\u00eb tep\u00ebrt dhe pasurimin e tij). E vendos\u00ebm at\u00eb n\u00eb docker, t\u00eb cilin e ekzekutojm\u00eb p\u00ebrmes docker-compose, atje po ashtu vendos\u00ebm logstash-exporter, i cili jep metrika n\u00eb Prometheus p\u00ebr monitorimin operacional t\u00eb fluksit t\u00eb log\u00ebve. K\u00ebshtu e dham\u00eb vet\u00ebs mund\u00ebsin\u00eb t\u00eb ndryshojm\u00eb gradualisht numrin e instancave t\u00eb logstash-it, q\u00eb ishin p\u00ebrgjegj\u00ebs p\u00ebr procesimin e \u00e7do lloji logu.<\/p>\n
Nd\u00ebrkoh\u00eb q\u00eb po p\u00ebrmirsonim klasterin, vizitueshm\u00ebria e cian.ru u rrit n\u00eb 12.8 milion p\u00ebrdorues unik\u00eb n\u00eb muaj. Si rezultat, ndodhi q\u00eb transformimet tona paksa nuk arrit\u00ebn pas ndryshimeve n\u00eb prodhim, dhe u p\u00ebrball\u00ebm me faktin se nodet \"t\u00eb ngrohta\" nuk e p\u00ebrballonin ngarkes\u00ebn dhe ngadal\u00ebsuan t\u00eb gjith\u00eb shp\u00ebrndarjen e log\u00ebve. T\u00eb dh\u00ebnat \"t\u00eb nxehta\" i merrnim pa nd\u00ebrprerje, por p\u00ebr shp\u00ebrndarjen e t\u00eb tjerave duhej t\u00eb intervenonim dhe t\u00eb b\u00ebnim rollover manual p\u00ebr t\u00eb shp\u00ebrndar\u00eb n\u00eb m\u00ebnyr\u00eb t\u00eb barabart\u00eb indekset.\u00a0<\/p>\n
Megjithat\u00eb, shkall\u00ebzimi dhe ndryshimi i konfigurimeve t\u00eb instancave t\u00eb logstash-it n\u00eb klaster u komplikuar nga fakti se ishte nj\u00eb docker-compose lokal, dhe t\u00eb gjitha veprimet kryheshin manualisht (p\u00ebr t\u00eb shtuar skajet e reja kishte nevoj\u00eb t\u00eb kalonim manualisht n\u00eb t\u00eb gjitha serverat dhe t\u00eb b\u00ebnim docker-compose up -d kudo).<\/p>\n
Rredistribuimi i log\u00ebve<\/h3>\n
\nN\u00eb shtator t\u00eb k\u00ebtij vitit, ne ende po vazhdonim t\u00eb shp\u00ebrb\u00ebnim monolit, ngarkesa n\u00eb klaster po rritej, dhe fluksi i log\u00ebve po arriti n\u00eb gati 30 mij\u00eb mesazhe n\u00eb sekond\u00eb.\u00a0<\/p>\n
\"Si
\n
\nIteracionin e ardhsh\u00ebm e filluam me p\u00ebrmir\u00ebsimin e harduerit. Nga pes\u00eb koordinator\u00eb kaluam n\u00eb tre, z\u00ebvend\u00ebsuam nodet e dh\u00ebnave dhe fituam n\u00eb kosto dhe kapacitet ruajtjeje. P\u00ebr nodet p\u00ebrdorim dy konfiguracione:\u00a0<\/p>\n