{"id":53590,"date":"2019-12-05T00:00:00","date_gmt":"2019-12-04T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-my-v-tsian-ukroshhali-terabajty-logov"},"modified":"2020-02-18T14:01:30","modified_gmt":"2020-02-18T11:01:30","slug":"kak-my-v-tsian-ukroshhali-terabajty-logov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-my-v-tsian-ukroshhali-terabajty-logov","title":{"rendered":"Si i zbut\u00ebm terabyte-et e log\u00ebve n\u00eb CI\u0410\u041d","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Si
\n
\nP\u00ebrsh\u00ebndetje t\u00eb gjith\u00ebve, un\u00eb quhem Aleksand\u00ebr, punoj n\u00eb CIAN si inxhinier dhe merrem me administrimin e sistemeve dhe automatizimin e proceseve infrastrukturore. N\u00eb komentet p\u00ebr nj\u00eb nga artikujt e kaluar, na u k\u00ebrkua t\u00eb tregojm\u00eb se nga vijm\u00eb 4 TB log-e n\u00eb dit\u00eb dhe \u00e7far\u00eb b\u00ebjm\u00eb me to. Po, kemi shum\u00eb log-e, dhe p\u00ebr p\u00ebrpunimin e tyre kemi krijuar nj\u00eb klaster infrastrukture t\u00eb ve\u00e7ant\u00eb q\u00eb na lejon t\u00eb zgjidhim shpejt probleme. N\u00eb k\u00ebt\u00eb artikull do t\u00eb flas p\u00ebr m\u00ebnyr\u00ebn se si e kemi adaptuar at\u00eb p\u00ebr nj\u00eb vit p\u00ebr t\u00eb punuar me rrjedh\u00ebn n\u00eb rritje t\u00eb t\u00eb dh\u00ebnave.
\n<\/a><\/noindex><\/p>\n

Nga kemi filluar<\/h3>\n

\n\"Si
\n
\nGjat\u00eb disa viteve t\u00eb fundit, ngarkesa n\u00eb cian.ru \u00ebsht\u00eb rritur shum\u00eb shpejt, dhe deri n\u00eb tremujorin e tret\u00eb t\u00eb vitit 2018, vizitueshm\u00ebria e burimit arriti n\u00eb 11.2 milion p\u00ebrdorues unik\u00eb n\u00eb muaj. N\u00eb at\u00eb koh\u00eb, n\u00eb momente kritike humbisnim deri n\u00eb 40% t\u00eb log-eve, duke na penguar t\u00eb menaxhonim shpejt incidentet dhe duke shpenzuar shum\u00eb koh\u00eb dhe energji p\u00ebr t'i zgjidhur ato. Gjithashtu, shpesh nuk mund t\u00eb gjenim shkakun e problemit, dhe ai p\u00ebrs\u00ebritej pas nj\u00eb kohe. Kjo ishte nj\u00eb situat\u00eb e v\u00ebshtir\u00eb q\u00eb duhej t\u00eb zgjidhnim.<\/p>\n

N\u00eb at\u00eb koh\u00eb, p\u00ebr ruajtjen e log-eve p\u00ebrdornim nj\u00eb klaster prej 10 nodash t\u00eb dh\u00ebnash me ElasticSearch version 5.5.2 me parametra standard t\u00eb indekseve. E implementuam at\u00eb m\u00eb shum\u00eb se nj\u00eb vit m\u00eb par\u00eb si nj\u00eb zgjidhje t\u00eb njohur dhe t\u00eb p\u00ebrballueshme: at\u00ebher\u00eb fluksi i log-eve nuk ishte aq i madh, dhe nuk kishte kuptim t\u00eb shpiknim konfiguracione jo standarde.\u00a0<\/p>\n

P\u00ebrpunimin e log-eve t\u00eb ardhshme e siguronim me Logstash n\u00eb porte t\u00eb ndryshme n\u00eb pes\u00eb koordinator\u00eb t\u00eb ElasticSearch. Nj\u00eb indeks, pavar\u00ebsisht nga madh\u00ebsia, p\u00ebrb\u00ebhej prej pes\u00eb shardesh. U organizua rotacioni \u00e7do or\u00eb dhe \u00e7do dit\u00eb, si rezultat \u00e7do or\u00eb n\u00eb klaster ndodhnin rreth 100 sharde t\u00eb reja. Nd\u00ebrsa log-eve nuk ishin aq shum\u00eb, klasteri p\u00ebrballonte dhe askush nuk kishte v\u00ebn\u00eb re parametrat e tij.\u00a0<\/p>\n

Problemet nga rritja e shpejt\u00eb<\/h3>\n

\nVolumi i log-eve t\u00eb gjeneruara rritej shum\u00eb shpejt, sepse dy procese p\u00ebrplasnin nj\u00ebri-tjetrin. Nga nj\u00ebra an\u00eb, numri i p\u00ebrdoruesve t\u00eb sh\u00ebrbimit po rritej. Nga ana tjet\u00ebr, ne filluam t\u00eb kalojm\u00eb aktivisht n\u00eb nj\u00eb arkitektur\u00eb mikrosh\u00ebrbimesh, duke ndar\u00eb monolit\u00ebt tan\u00eb t\u00eb vjet\u00ebr n\u00eb C# dhe Python. Disa dhjet\u00ebra mikrosh\u00ebrbime t\u00eb reja, q\u00eb z\u00ebvend\u00ebsuan pjes\u00eb t\u00eb monolitit, gjeneronin ndjesh\u00ebm m\u00eb shum\u00eb log-e p\u00ebr klasterin infrastruktural.\u00a0<\/p>\n

Pik\u00ebrisht shkall\u00ebzimi na \u00e7oi n\u00eb nj\u00eb situat\u00eb ku klasteri u b\u00eb praktikisht i pakontrolluar. Kur log-et filluan t\u00eb vinin me shpejt\u00ebsi 20,000 mesazhe n\u00eb sekond\u00eb, rotacionet e shpeshta t\u00eb padobishme rrit\u00ebn numrin e shardeve n\u00eb 6,000, nd\u00ebrsa p\u00ebr \u00e7do nod kishte m\u00eb shum\u00eb se 600 sharde.\u00a0<\/p>\n

Kjo e \u00e7onte n\u00eb probleme me ndarjen e memorie operativ dhe n\u00eb momentin e r\u00ebnies s\u00eb nj\u00eb nodi fillonte nj\u00eb kalim i menj\u00ebhersh\u00ebm i t\u00eb gjith\u00eb shardeve, duke shum\u00ebfishuar trafikun dhe ngarkuar nodet e tjera, gj\u00eb q\u00eb e b\u00ebnte praktikisht t\u00eb pamundur regjistrimin e t\u00eb dh\u00ebnave n\u00eb klaster. Dhe n\u00eb at\u00eb periudh\u00eb mbet\u00ebm pa log-e. N\u00eb rastin e nj\u00eb problemi me serverin<\/a> humbnim 1\/10 e klasterit n\u00eb p\u00ebrgjith\u00ebsi. Nj\u00eb num\u00ebr i madh indekseve t\u00eb vogla shtonte v\u00ebshtir\u00ebsi.<\/p>\n

Pa log-e nuk kuptonim arsyet e incidentit dhe mund t\u00eb p\u00ebrballeshim p\u00ebrs\u00ebri me t\u00eb nj\u00ebjt\u00ebn problematik\u00eb, ndon\u00ebse ne n\u00eb ideologjin\u00eb ton\u00eb nuk pranojm\u00eb q\u00eb ndodhin t\u00eb nj\u00ebjtat probleme. Prandaj na duhej volume t\u00eb plot\u00eb log-esh dhe dor\u00ebzimi i tyre pothuajse n\u00eb koh\u00eb reale, pasi ekipi i inxhinier\u00ebve p\u00ebrgjonte sinjalizimet jo vet\u00ebm nga metrikat, por edhe nga log-\u00ebt. P\u00ebr t\u00eb kuptuar sasin\u00eb e problemit \u2014 n\u00eb at\u00eb koh\u00eb volumi total i log-eve arrinte rreth 2 TB n\u00eb dit\u00eb.\u00a0<\/p>\n

Ne vendos\u00ebm nj\u00eb q\u00ebllim \u2014 t\u00eb eliminojm\u00eb plot\u00ebsisht humbjen e log-eve dhe t\u00eb shkurtuam koh\u00ebn e dor\u00ebzimit t\u00eb tyre n\u00eb klasterin ELK deri n\u00eb maksimum 15 minuta gjat\u00eb forcon e jasht\u00ebzakonshme (kjo ishte numri n\u00eb t\u00eb cilin ne u mb\u00ebshtet\u00ebm si KPI t\u00eb brendsh\u00ebm m\u00eb von\u00eb).<\/p>\n

Mekanizmi i ri i rotacionit dhe nodet hot-warm<\/h3>\n

\n\"Si
\n
\nTransformimi i klasterit e filluam me p\u00ebrdit\u00ebsimin e versionit t\u00eb ElasticSearch nga 5.5.2 n\u00eb 6.4.3. Klasteri yn\u00eb i versionit 5 ra p\u00ebrs\u00ebri, dhe ne vendos\u00ebm ta fiknim dhe ta azhurnonim plot\u00ebsisht \u2014 gjithsesi nuk kishim log-e. Pra, k\u00ebt\u00eb kalim e realizuam brenda disa or\u00ebve.<\/p>\n

Transformimi m\u00eb i madh n\u00eb k\u00ebt\u00eb faz\u00eb ishte implementimi n\u00eb tre nodet me nj\u00eb koordinatator si nj\u00eb tampon t\u00eb p\u00ebrkohsh\u00ebm Apache Kafka. Brokeri i mesazheve na shp\u00ebtoi nga humbja e log-eve gjat\u00eb problemeve me ElasticSearch. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, ne shtuam 2 nod\u00eb n\u00eb klaster dhe kaluam n\u00eb arkitektur\u00ebn hot-warm me tre nod\u00eb \"t\u00eb nxehta\", t\u00eb vendosura n\u00eb rafte t\u00eb ndryshme n\u00eb qendrat e t\u00eb dh\u00ebnave. N\u00eb to dirigjuam log-et q\u00eb nuk mund t\u00eb humbasim n\u00ebn asnj\u00eb rrethan\u00eb \u2014 nginx, si dhe log-et e gabimeve t\u00eb aplikacioneve. N\u00eb nodet e tjera kalonin log\u00ebt minor\u00eb \u2014 debug, warning, etj., nd\u00ebrsa pas 24 or\u00ebsh kalonin \"log\u00ebt e r\u00ebnd\u00ebsish\u00ebm\" nga nodet \"e nxehta\".<\/p>\n

P\u00ebr t\u00eb shmangur rritjen e numrit t\u00eb indekseve t\u00eb vegj\u00ebl, ne kaluam nga rotacioni n\u00eb koh\u00eb n\u00eb mekanizmin rollover. N\u00eb forume kishte shum\u00eb informacion q\u00eb rotacioni sipas madh\u00ebsis\u00eb s\u00eb indeksit ishte shum\u00eb i pasigurt, prandaj ne vendos\u00ebm t\u00eb p\u00ebrdorim rotacionin sipas numrit t\u00eb dokumenteve n\u00eb indeks. Anlizuam \u00e7do indeks dhe regjistruam numrin e dokumenteve, pas t\u00eb cilit duhet t\u00eb aktivizohej rotacioni. N\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb arrit\u00ebm nj\u00eb madh\u00ebsi optimale sharde \u2014 jo m\u00eb shum\u00eb se 50 GB.\u00a0<\/p>\n

Optimizimi i klasterit<\/h3>\n

\n\"Si
\n
\nMegjithat\u00eb, ne nuk u shp\u00ebtuam plot\u00ebsisht problemeve. Fatkeq\u00ebsisht, akoma shfaqeshin indekse t\u00eb vogla: ato nuk arrinin volumin e caktuar, nuk rotatoeshin dhe hiqeshin me pastrimin global t\u00eb indekseve mbi tre dit\u00eb, sepse ne hoq\u00ebm rotacionin sipas dat\u00ebs. Kjo \u00e7onte n\u00eb humbje t\u00eb t\u00eb dh\u00ebnave, pasi indeksi nga klasteri zhdukej plot\u00ebsisht, dhe tentimi p\u00ebr t\u00eb shkruar n\u00eb nj\u00eb indeks q\u00eb nuk ekzistonte prishte logjik\u00ebn e curator-it, t\u00eb cilin e p\u00ebrdornim p\u00ebr menaxhim. Alias p\u00ebr t\u00eb shkruar shnd\u00ebrrohej n\u00eb indeks dhe prishte logjik\u00ebn e rollover-it, duke shkaktuar nj\u00eb rritje t\u00eb pakontrolluar t\u00eb disa indekseve deri n\u00eb 600 GB.\u00a0<\/p>\n

P\u00ebr shembull, p\u00ebr konfigurimin e rotacionit:<\/p>\n

curator-elk-rollover.yaml\n\n---\nactions:\n\u00a0\u00a01:\n\u00a0\u00a0\u00a0\u00a0action: rollover\n\u00a0\u00a0\u00a0\u00a0options:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0name: \"nginx_write\"\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0conditions:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0max_docs: 100000000\n\u00a0\u00a02:\n\u00a0\u00a0\u00a0\u00a0action: rollover\n\u00a0\u00a0\u00a0\u00a0options:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0name: \"python_error_write\"\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0conditions:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0max_docs: 10000000\n<\/code><\/pre>\n
N\u00eb munges\u00eb t\u00eb rollover alias, shfaqej nj\u00eb gabim:<\/p>\n
ERROR \u00a0 \u00a0 alias \"nginx_write\" not found.\nERROR \u00a0 \u00a0 Failed to complete action: rollover.\u00a0 <type 'exceptions.ValueError'>: Unable to perform index rollover with alias \"nginx_write\".\n<\/code><\/pre>\n
Zgjidhjen e k\u00ebsaj probleme e lam\u00eb p\u00ebr iterat\u00ebn e ardhshme dhe u angazhuam me nj\u00eb \u00e7\u00ebshtje tjet\u00ebr: kaluam n\u00eb logjik\u00ebn pull t\u00eb funksionimit t\u00eb Logstash, q\u00eb merret me p\u00ebrpunimin e logeve t\u00eb ardhshme (heqja e informacionit t\u00eb tep\u00ebrt dhe pasurimi). E vendos\u00ebm at\u00eb n\u00eb docker, t\u00eb cilin e aktivizojm\u00eb p\u00ebrmes docker-compose, po aty e vendos\u00ebm logstash-exporter, i cili jep metrika n\u00eb Prometheus p\u00ebr monitorimin operativ t\u00eb fluksit t\u00eb logeve. K\u00ebshtu i dham\u00eb vetes mund\u00ebsin\u00eb t\u00eb ndryshonim me leht\u00ebsi numrin e instancave t\u00eb logstash q\u00eb ishin p\u00ebrgjegj\u00ebse p\u00ebr p\u00ebrpunimin e \u00e7do lloji logu.<\/p>\n
Nd\u00ebrkoh\u00eb q\u00eb po p\u00ebrmir\u00ebsonim klasterin, vizita n\u00eb cian.ru u rrit n\u00eb 12,8 milion p\u00ebrdorues unik\u00eb n\u00eb muaj. Si rezultat, transformimet tona paksa nuk po p\u00ebrputheshin me ndryshimet n\u00eb prodhim, dhe p\u00ebrballeshim me faktin q\u00eb nodet \"t\u00eb ngrohta\" nuk po p\u00ebrballonin ngarkes\u00ebn dhe ngadal\u00ebsonin t\u00eb gjith\u00eb d\u00ebrgimin e logeve. T\u00eb dh\u00ebnat \"t\u00eb nxehta\" i merrnim pa u prishur, por p\u00ebr d\u00ebrgimin e t\u00eb tjerave duhej t\u00eb nd\u00ebrhyhej dhe t\u00eb b\u00ebhej rollover manual, p\u00ebr t\u00eb shp\u00ebrndar\u00eb n\u00eb m\u00ebnyr\u00eb t\u00eb barabart\u00eb indekset.\u00a0<\/p>\n
N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, shkall\u00ebzimi dhe ndryshimi i cil\u00ebsimeve t\u00eb instancave t\u00eb logstash n\u00eb klaster u v\u00ebshtir\u00ebsuan nga fakti se kjo ishte nj\u00eb docker-compose vendor, dhe t\u00eb gjitha veprimet kryheshin me duar (p\u00ebr t\u00eb shtuar p\u00ebrfundimet e reja duhej t\u00eb kalonim me duar n\u00eb t\u00eb gjitha server\u00ebt dhe t\u00eb b\u00ebnim docker-compose up -d kudo).<\/p>\n
Rredistribuimi i logeve<\/h3>\n
\nN\u00eb shtator t\u00eb k\u00ebtij viti, ne ende vazhdojm\u00eb t\u00eb ndajm\u00eb monolit, ngarkesa n\u00eb klaster po rritej, dhe fluksi i logeve po afrohej n\u00eb 30 mij\u00eb mesazhe n\u00eb sekond\u00eb.\u00a0<\/p>\n
\"Si
\n
\nIterat\u00ebn e ardhshme e filluam me p\u00ebrdit\u00ebsimin e harduerit. Nga pes\u00eb koordinator\u00eb kaluam n\u00eb tre, z\u00ebvend\u00ebsuam nodet e t\u00eb dh\u00ebnave dhe fituam si n\u00eb para ashtu edhe n\u00eb kapacitetin e ruajtjes. P\u00ebr nodet p\u00ebrdorim dy konfiguracione:\u00a0<\/p>\n