{"id":53590,"date":"2019-12-05T00:00:00","date_gmt":"2019-12-04T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-my-v-tsian-ukroshhali-terabajty-logov"},"modified":"2020-02-18T14:01:30","modified_gmt":"2020-02-18T11:01:30","slug":"kak-my-v-tsian-ukroshhali-terabajty-logov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/az\/blog\/administrirovanie\/kak-my-v-tsian-ukroshhali-terabajty-logov","title":{"rendered":"Biz C\u0130AN-\u0131 terabayt loq d\u00fcz\u0259ld\u0259r\u0259k nec\u0259 bo\u011fduq","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Biz
\n
\nH\u0259r k\u0259s\u0259 salam, m\u0259nim ad\u0131m Aleksandr, m\u0259n CIAN-da m\u00fch\u0259ndis olaraq \u00e7al\u0131\u015f\u0131ram v\u0259 sistem administrasiyas\u0131 il\u0259 infrastruktur prosesl\u0259rinin avtomatla\u015fd\u0131r\u0131lmas\u0131 il\u0259 m\u0259\u015f\u011ful oluram. Ke\u00e7\u0259n yaz\u0131lardan birinin \u015f\u0259rhl\u0259rind\u0259 bizd\u0259n soru\u015fdular ki, g\u00fcnd\u0259 4 TB-l\u0131q loglar\u0131 haradan al\u0131r\u0131q v\u0259 onlarla n\u0259 edirik. B\u0259li, loglar\u0131m\u0131z \u00e7oxdur v\u0259 onlar\u0131n emal\u0131 \u00fc\u00e7\u00fcn ayr\u0131 bir infrastruktur klasteri yarad\u0131lm\u0131\u015fd\u0131r ki, bu da biz\u0259 probleml\u0259ri tez h\u0259ll etm\u0259y\u0259 imkan tan\u0131y\u0131r. Bu m\u0259qal\u0259d\u0259 m\u0259n bir il \u0259rzind\u0259 onu daima artan m\u0259lumat ax\u0131n\u0131 il\u0259 i\u015fl\u0259m\u0259k \u00fc\u00e7\u00fcn nec\u0259 adaptasiya etdiyimizi izah ed\u0259c\u0259y\u0259m.
\n<\/a><\/noindex><\/p>\n

Ba\u015flam\u0131\u015f oldu\u011fumuz yer<\/h3>\n

\n\"Biz
\n
\nSon bir ne\u00e7\u0259 ild\u0259 cian.ru sayt\u0131na olan y\u00fck \u00e7ox s\u00fcr\u0259tl\u0259 artd\u0131 v\u0259 2018-ci ilin \u00fc\u00e7\u00fcnc\u00fc r\u00fcb\u00fcnd\u0259 resursun ziyar\u0259t\u00e7i say\u0131 ayda 11.2 milyon unikal istifad\u0259\u00e7iy\u0259 \u00e7atd\u0131. O vaxt kritik anlarda loglar\u0131n 40%-d\u0259kini itirirdik, bu da biz\u0259 hadis\u0259l\u0259rl\u0259 tez m\u0259\u015f\u011ful olma\u011fa imkan vermirdi v\u0259 onlar\u0131n h\u0259llin\u0259 \u00e7ox vaxt v\u0259 enerji s\u0259rf edirdik. H\u0259m\u00e7inin, problemi tapmaqda tez-tez \u00e7\u0259tinlik \u00e7\u0259kirdik v\u0259 o, m\u00fc\u0259yy\u0259n bir vaxtdan sonra t\u0259krarlan\u0131rd\u0131. Bu, n\u0259s\u0259 etm\u0259li oldu\u011fumuz bir v\u0259ziyy\u0259t idi.<\/p>\n

O d\u00f6vrd\u0259 loglar\u0131 saxlamaq \u00fc\u00e7\u00fcn 5.5.2 versiyal\u0131 ElasticSearch il\u0259 10 m\u0259lumat noddan ibar\u0259t bir klaster istifad\u0259 edirdik. Onu daha bir il \u0259vv\u0259l populyar v\u0259 \u0259lveri\u015fli bir h\u0259ll olaraq \u0259lav\u0259 etmi\u015fdik: o vaxt log ax\u0131n\u0131 bu q\u0259d\u0259r b\u00f6y\u00fck deyildi, qeyri-standart konfiqurasiyalar\u0131 d\u00fc\u015f\u00fcnm\u0259y\u0259 d\u0259ym\u0259zdi.\u00a0<\/p>\n

Giri\u015f loglar\u0131n\u0131n emal\u0131 Logstash t\u0259r\u0259find\u0259n be\u015f ElasticSearch koordinatorunda m\u00fcxt\u0259lif portlarda t\u0259min edilirdi. Bir indeks, \u00f6l\u00e7\u00fcs\u00fcnd\u0259n as\u0131l\u0131 olmayaraq, be\u015f sharddan ibar\u0259t idi. Saatl\u0131q v\u0259 g\u00fcnd\u0259lik rotasiya t\u0259\u015fkil olunmu\u015fdu, n\u0259tic\u0259d\u0259 klasterd\u0259 h\u0259r saat t\u0259xmin\u0259n 100 yeni shard yaran\u0131rd\u0131. Loglar \u00e7ox olmad\u0131qda, klaster i\u015fini g\u00f6r\u00fcrd\u00fc v\u0259 onun parametrl\u0259ri \u00fcz\u0259rind\u0259 he\u00e7 kim \u00e7ox dayanmad\u0131.\u00a0<\/p>\n

S\u00fcr\u0259tli art\u0131m probleml\u0259ri<\/h3>\n

\nYarad\u0131lan loglar\u0131n h\u0259cmi \u00e7ox s\u00fcr\u0259tl\u0259 artd\u0131, \u00e7\u00fcnki iki proses bir-birini tamamlad\u0131. Bir t\u0259r\u0259fd\u0259n, xidm\u0259tin istifad\u0259\u00e7il\u0259ri artd\u0131. Dig\u0259r t\u0259r\u0259fd\u0259n, biz k\u00f6hn\u0259 monolitl\u0259ri C# v\u0259 Python dilind\u0259 paralel mikrosistem memarl\u0131\u011f\u0131na aktiv ke\u00e7m\u0259y\u0259 ba\u015flad\u0131q. Bir ne\u00e7\u0259 on yeni mikrosistem, monolitin b\u0259zi hiss\u0259l\u0259rini \u0259v\u0259z ed\u0259r\u0259k, infrastruktur klasteri \u00fc\u00e7\u00fcn daha \u00e7ox loglar yaradma\u011fa ba\u015flad\u0131.\u00a0<\/p>\n

M\u0259hz bu miqyaslanma klasterin \u00e7oxlu idar\u0259 olunmaz hala g\u0259lm\u0259sin\u0259 s\u0259b\u0259b oldu. Loglar saniy\u0259y\u0259 20 min mesaj s\u00fcr\u0259ti il\u0259 g\u0259lm\u0259y\u0259 ba\u015flad\u0131qda, tez-tez m\u0259nas\u0131z rotasiya shardlar\u0131n say\u0131n\u0131 6 min\u0259 \u00e7atd\u0131rd\u0131, bir n\u00f3d \u00fc\u00e7\u00fcn 600-d\u0259n \u00e7ox shard d\u00fc\u015fd\u00fc.\u00a0<\/p>\n

Bu, \u0259m\u0259li yadda\u015f\u0131n ayr\u0131lmas\u0131 il\u0259 ba\u011fl\u0131 probleml\u0259r\u0259 s\u0259b\u0259b oldu, n\u00f3dun \u00e7\u00f6km\u0259si zaman\u0131 b\u00fct\u00fcn shardlar\u0131n eyni anda k\u00f6\u00e7\u00fcr\u00fclm\u0259si ba\u015flay\u0131r, bu da trafikl\u0259ri art\u0131r\u0131r v\u0259 dig\u0259r n\u00f3dlar\u0131 y\u00fckl\u0259yirdi, bu da klasterd\u0259 m\u0259lumat yaz\u0131lmas\u0131n\u0131 dem\u0259k olar ki, m\u00fcmk\u00fcn etmirdi. V\u0259 bu d\u00f6vrd\u0259 loglar\u0131m\u0131z olmurdu. server<\/a> Biz 1\/10 klasteri itiraf edirdik. Ki\u00e7ik \u00f6l\u00e7\u00fcl\u00fc \u00e7ox sayda indeks, \u00e7\u0259tinlikl\u0259rimizi art\u0131r\u0131rd\u0131.<\/p>\n

G\u00f6zl\u0259m\u0259 siyah\u0131s\u0131 olmadan hadis\u0259nin s\u0259b\u0259bl\u0259rini anlam\u0131rd\u0131q v\u0259 bir g\u00fcn eyni s\u0259hvl\u0259ri t\u0259krarlaya bil\u0259rdik, bu is\u0259 komandam\u0131z\u0131n ideologiyas\u0131 \u00fc\u00e7\u00fcn yolverilm\u0259z idi, \u00e7\u00fcnki i\u015f mexanizml\u0259rimiz bunun \u0259ksin\u0259, eyni probleml\u0259ri he\u00e7 vaxt t\u0259krarlamamaq \u00fcz\u0259rind\u0259 qurulmu\u015fdur. Buna g\u00f6r\u0259 tam log h\u0259cmin\u0259 v\u0259 onlar\u0131n dem\u0259k olar ki, real vaxt rejimind\u0259 g\u00f6nd\u0259rilm\u0259sin\u0259 ehtiyac\u0131m\u0131z var idi, \u00e7\u00fcnki n\u00f6vb\u0259t\u00e7i m\u00fch\u0259ndis komandas\u0131 yaln\u0131z metrikl\u0259rd\u0259n deyil, loglardan da alerta bax\u0131rd\u0131. Problemin miqyas\u0131n\u0131 anlamaq \u00fc\u00e7\u00fcn \u2014 o d\u00f6vrd\u0259 g\u00fcnd\u0259lik log h\u0259cmi t\u0259xmin\u0259n 2 TB idi.\u00a0<\/p>\n

Bizim m\u0259qs\u0259dimiz - log itkisinin tamamil\u0259 qar\u015f\u0131s\u0131n\u0131 almaq v\u0259 f\u00f6vq\u0259lad\u0259 hallar zaman\u0131 loglar\u0131n ELK klasterin\u0259 g\u00f6nd\u0259rilm\u0259 m\u00fcdd\u0259tini maksimum 15 d\u0259qiq\u0259y\u0259 endirm\u0259k idi (bu r\u0259q\u0259m\u0259 g\u0259l\u0259c\u0259kd\u0259 daxili KPI kimi istinad ed\u0259c\u0259yik).<\/p>\n

Yeni d\u00f6vriyy\u0259 mexanizmi v\u0259 hot-warm d\u00fcy\u00fcnl\u0259ri<\/h3>\n

\n\"Biz
\n
\nKlasteri \u00e7evirm\u0259k \u00fc\u00e7\u00fcn ElasticSearch versiyas\u0131n\u0131 5.5.2-d\u0259n 6.4.3-\u0259 yenil\u0259m\u0259kl\u0259 ba\u015flad\u0131q. 5-ci versiyan\u0131n klasteri bir daha qald\u0131, onu s\u00f6nd\u00fcrm\u0259yi v\u0259 tamamil\u0259 yenil\u0259m\u0259yi q\u0259rara ald\u0131q - loglar bel\u0259 yoxdur. Bel\u0259likl\u0259, bu ke\u00e7idi yaln\u0131z bir ne\u00e7\u0259 saat \u0259rzind\u0259 etdik.<\/p>\n

Bu m\u0259rh\u0259l\u0259d\u0259 \u0259n \u0259h\u0259miyy\u0259tli d\u0259yi\u015fiklik \u00fc\u00e7 d\u00fcy\u00fcn\u0259 koordinat\u00f6r kimi Apache Kafka-n\u0131n vasit\u0259\u00e7i buffer olaraq t\u0259tbiq edilm\u0259si oldu. Mesaj brokeri ElasticSearch il\u0259 probleml\u0259r zaman\u0131 log itkisind\u0259n qurtulma\u011f\u0131m\u0131za k\u00f6m\u0259k etdi. Eyni zamanda, klasterimiz\u0259 2 d\u00fcy\u00fcn \u0259lav\u0259 etdik v\u0259 f\u0259rqli serverl\u0259rd\u0259 yerl\u0259\u015f\u0259n \u00fc\u00e7 \"isti\" d\u00fcy\u00fcnl\u0259 hot-warm arxitekturas\u0131na ke\u00e7dik. Onlara itirilm\u0259m\u0259si laz\u0131m olan loglar\u0131 - nginx v\u0259 t\u0259tbiq s\u0259hvl\u0259ri loglar\u0131n\u0131 y\u00f6nl\u0259ndirdik. Dig\u0259r d\u00fcy\u00fcnl\u0259r\u0259 is\u0259 ki\u00e7ik loglar - debug, warning v\u0259 s. g\u00f6nd\u0259rildi, 24 saatdan sonra is\u0259 \"isti\" d\u00fcy\u00fcnl\u0259rd\u0259n \"\u00f6n\u0259mli\" loglar k\u00f6\u00e7\u00fcr\u00fcld\u00fc.<\/p>\n

Ki\u00e7ik \u00f6l\u00e7\u00fcl\u00fc indeksl\u0259rin say\u0131n\u0131 art\u0131rmamaq \u00fc\u00e7\u00fcn zamanla d\u00f6vriyy\u0259d\u0259n \u00f6l\u00e7\u00fc \u0259sasl\u0131 mexanizm\u0259 ke\u00e7dik. Forumlarda indeks \u00f6l\u00e7\u00fcs\u00fcn\u0259 g\u00f6r\u0259 d\u00f6vriyy\u0259nin \u00e7ox etibars\u0131z oldu\u011fu haqq\u0131nda \u00e7oxlu m\u0259lumat var idi, buna g\u00f6r\u0259 d\u0259 biz indeksd\u0259ki s\u0259n\u0259d say\u0131na g\u00f6r\u0259 d\u00f6vriyy\u0259 t\u0259tbiq etm\u0259y\u0259 q\u0259rar verdik. H\u0259r indeksin say\u0131n\u0131 analiz etdik v\u0259 d\u00f6vriyy\u0259nin ba\u015flamas\u0131 laz\u0131m oldu\u011fu s\u0259n\u0259d say\u0131n\u0131 qeyd\u0259 ald\u0131q. Bel\u0259likl\u0259, optimal shard \u00f6l\u00e7\u00fcs\u00fcn\u0259 - 50 GB-dan \u00e7ox olmadan - nail olduq.\u00a0<\/p>\n

Klasterin optimalla\u015fd\u0131r\u0131lmas\u0131<\/h3>\n

\n\"Biz
\n
\nAmma tamamil\u0259 probleml\u0259rd\u0259n azad ola bilm\u0259dik. T\u0259\u0259ss\u00fcf ki, h\u0259l\u0259 d\u0259 ki\u00e7ik indeksl\u0259r meydana g\u0259lirdi: onlar t\u0259l\u0259b olunan h\u0259cmi \u00e7atd\u0131ra bilmirdi, d\u00f6vriyy\u0259y\u0259 sal\u0131nm\u0131rd\u0131 v\u0259 \u00fc\u00e7 g\u00fcnd\u0259n k\u00f6hn\u0259 indeksl\u0259rin qlobal t\u0259mizl\u0259nm\u0259si il\u0259 silinirdi, \u00e7\u00fcnki tarix\u0259 g\u00f6r\u0259 d\u00f6vriyy\u0259ni aradan qald\u0131rd\u0131q. Bu, m\u0259lumatlar\u0131n itirilm\u0259sin\u0259 s\u0259b\u0259b oldu, \u00e7\u00fcnki klusterd\u0259n bir indeks tamamil\u0259 yox olurdu v\u0259 m\u00f6vcud olmayan bir indeks\u0259 yazma c\u0259hdi istifad\u0259 etdiyimiz curator logikas\u0131n\u0131 pozurdu. Yazma \u00fc\u00e7\u00fcn alias indeks\u0259 \u00e7evrilirdi v\u0259 rollover logikas\u0131n\u0131 pozaraq b\u0259zi indeksl\u0259rin 600 Gb-a q\u0259d\u0259r n\u0259zar\u0259tsiz artmas\u0131na s\u0259b\u0259b olurdu.\u00a0<\/p>\n

M\u0259s\u0259l\u0259n, d\u00f6vriyy\u0259 konfiqurasiyas\u0131 \u00fc\u00e7\u00fcn:<\/p>\n

curator-elk-rollover.yaml\n\n---\nactions:\n  1:\n    action: rollover\n    options:\n      name: \"nginx_write\"\n      conditions:\n        max_docs: 100000000\n  2:\n    action: rollover\n    options:\n      name: \"python_error_write\"\n      conditions:\n        max_docs: 10000000\n<\/code><\/pre>\n
Rollover alias olmadan bir x\u0259ta ba\u015f verdi:<\/p>\n
ERROR     alias \"nginx_write\" not found.\nERROR     Failed to complete action: rollover.  : Unable to perform index rollover with alias \"nginx_write\".\n<\/code><\/pre>\n
Bu problemin h\u0259llini n\u00f6vb\u0259ti iterasiya \u00fc\u00e7\u00fcn saxlad\u0131q v\u0259 ba\u015fqa bir m\u0259s\u0259l\u0259y\u0259 ke\u00e7dik: Logstash-\u0131n daxil olan loglar\u0131 emal ed\u0259n pull logika il\u0259 i\u015fl\u0259m\u0259y\u0259 ba\u015flad\u0131q (art\u0131q m\u0259lumatlar\u0131n silinm\u0259si v\u0259 z\u0259nginl\u0259\u015fdirm\u0259). Onu docker-da yerl\u0259\u015fdirdik, docker-compose il\u0259 ba\u015flad\u0131q, burada logstash-exporter yerl\u0259\u015fdirdik, o da Prometheus-a log ax\u0131n\u0131 \u00fc\u00e7\u00fcn metrikl\u0259ri g\u00f6nd\u0259rir. Bel\u0259likl\u0259, loglar\u0131n emal\u0131ndan m\u0259sul olan logstash instansiyalar\u0131n\u0131n say\u0131n\u0131 rahatca d\u0259yi\u015fdirm\u0259k imkan\u0131n\u0131 verdik.<\/p>\n
Biz klosteri inki\u015faf etdir\u0259rk\u0259n, cian.ru-nun ayl\u0131q unikal istifad\u0259\u00e7i say\u0131 12.8 milyon artd\u0131. N\u0259tic\u0259d\u0259, g\u00f6rd\u00fcy\u00fcm\u00fcz d\u0259yi\u015fiklikl\u0259r istehsalda ba\u015f ver\u0259n d\u0259yi\u015fiklikl\u0259r\u0259 tam uy\u011fun g\u0259lm\u0259di v\u0259 \"isti\" nodel\u0259rin y\u00fck\u00fc idar\u0259 ed\u0259 bilm\u0259m\u0259si il\u0259 qar\u015f\u0131la\u015fd\u0131q, bu da loglar\u0131n \u00e7atd\u0131r\u0131lmas\u0131na gecikm\u0259l\u0259r\u0259 s\u0259b\u0259b oldu. \"\u0130sti\" m\u0259lumatlar\u0131 problemsiz ald\u0131q, amma qalanlar\u0131n\u0131n \u00e7atd\u0131r\u0131lmas\u0131nda m\u00fcdaxil\u0259 etm\u0259li olduq v\u0259 indeksl\u0259ri b\u0259rab\u0259r paylamaq \u00fc\u00e7\u00fcn \u0259l manevrl\u0259ri il\u0259 rollover etm\u0259li olduq.\u00a0<\/p>\n
Bu arada, klosterd\u0259 logstash instansiyalar\u0131n\u0131n miqyasland\u0131r\u0131lmas\u0131 v\u0259 t\u0259nziml\u0259nm\u0259si \u00e7\u0259tinl\u0259\u015fdi, \u00e7\u00fcnki bu, yerli docker-compose idi v\u0259 b\u00fct\u00fcn \u0259m\u0259liyyatlar \u0259l il\u0259 h\u0259yata ke\u00e7irilirdi (yeni u\u00e7 n\u00f6qt\u0259l\u0259r \u0259lav\u0259 etm\u0259k \u00fc\u00e7\u00fcn b\u00fct\u00fcn serverl\u0259rd\u0259 docker-compose up -d yerin\u0259 yetirm\u0259k laz\u0131m idi).<\/p>\n
Loglar\u0131n yenid\u0259n paylanmas\u0131<\/h3>\n
\nBu ilin sentyabr ay\u0131nda h\u0259l\u0259 d\u0259 monolit par\u00e7alama\u011fa davam edirdik, klasterd\u0259 y\u00fck art\u0131rd\u0131 v\u0259 log ax\u0131n\u0131 saniy\u0259d\u0259 30 min mesajla yax\u0131nla\u015f\u0131rd\u0131.\u00a0<\/p>\n
\"Biz
\n
\nN\u00f6vb\u0259ti iterasiyaya avadanl\u0131qlar\u0131 yenil\u0259m\u0259kl\u0259 ba\u015flad\u0131q. Be\u015f koordinatordan \u00fc\u00e7 n\u0259f\u0259r\u0259 ke\u00e7dik, m\u0259lumat nodlar\u0131n\u0131 d\u0259yi\u015fdik v\u0259 h\u0259m maliyy\u0259, h\u0259m d\u0259 saxlama h\u0259cmi qazanm\u0131\u015f olduq. Nodlar \u00fc\u00e7\u00fcn iki konfiqurasiya istifad\u0259 edirik:\u00a0<\/p>\n