{"id":32723,"date":"2019-10-31T21:48:35","date_gmt":"2019-10-31T18:48:35","guid":{"rendered":"https:\/\/prohoster.info\/blog\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada\/"},"modified":"2019-10-31T21:48:35","modified_gmt":"2019-10-31T18:48:35","slug":"yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada","title":{"rendered":"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Logy s\u00fa d\u00f4le\u017eitou s\u00fa\u010das\u0165ou syst\u00e9mu, ktor\u00e1 umo\u017e\u0148uje pochopi\u0165, \u010di funguje (alebo nefunguje) tak, ako sa o\u010dak\u00e1valo. V podmienkach mikroslu\u017eobnej architekt\u00fary sa pr\u00e1ca s logmi st\u00e1va samostatnou discipl\u00ednou \u0161peci\u00e1lnej olympi\u00e1dy. Je potrebn\u00e9 vyrie\u0161i\u0165 mno\u017estvo ot\u00e1zok:<\/p>\n<p><\/p>\n<ul>\n<li>ako zapisova\u0165 logy z aplik\u00e1cie;<\/li>\n<li>kam zapisova\u0165 logy;<\/li>\n<li>ako doru\u010dova\u0165 logy na ulo\u017eenie a spracovanie;<\/li>\n<li>ako spracov\u00e1va\u0165 a uklada\u0165 logy.<\/li>\n<\/ul>\n<p><\/p>\n<p>Pou\u017eitie popul\u00e1rnych technol\u00f3gii kontajneriz\u00e1cie prid\u00e1va piesok na hrable na poli mo\u017enost\u00ed rie\u0161enia \u00falohy.<\/p>\n<p><\/p>\n<p>Pr\u00e1ve o tom hovor\u00ed v\u00fdklad spr\u00e1vy Jurija Bu\u0161me\u013eeva \u201eMapa probl\u00e9moch na poli zberu a dod\u00e1vky logov\u201c <\/p>\n<p>\n<center><div class=\"youtube-placeholder\" data-id=\"NAeedJv-S3I\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/NAeedJv-S3I\/hqdefault.jpg\" alt=\"Prehrajte video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<p>Kto m\u00e1 z\u00e1ujem, pros\u00edm, pod kat.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Vol\u00e1m sa Jurij Bu\u0161melev. Pracujem v Lazada. Dnes budem rozpr\u00e1va\u0165 o tom, ako sme robili na\u0161e logy, ako sme ich zbierali a \u010do do nich zapisujeme. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/ece6053b1027ab3681022049ddcd79ea.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Odkia\u013e sme? Kto sme? Lazada je internetov\u00fd obchod \u010d\u00edslo 1 v \u0161iestich krajin\u00e1ch juhov\u00fdchodnej \u00c1zie. V\u0161etky tieto krajiny m\u00e1me rozdelen\u00e9 pod\u013ea d\u00e1tov\u00fdch centier. Moment\u00e1lne m\u00e1me 4 d\u00e1tov\u00e9 centr\u00e1. Pre\u010do je to d\u00f4le\u017eit\u00e9? Preto\u017ee niektor\u00e9 rie\u0161enia boli podmienen\u00e9 t\u00fdm, \u017ee medzi centrami je ve\u013emi slab\u00e9 prepojenie. M\u00e1me mikroslu\u017eobn\u00fa architekt\u00faru. Bol som prekvapen\u00fd, ke\u010f som zistil, \u017ee m\u00e1me u\u017e 80 mikroslu\u017eieb. Ke\u010f som za\u010d\u00ednal s \u00falohou oh\u013eadom logov, bolo ich len 20. Navy\u0161e m\u00e1me pomerne ve\u013ek\u00fd kus PHP legacy, s ktor\u00fdm tie\u017e mus\u00edme \u017ei\u0165 a zmierova\u0165 sa. To v\u0161etko n\u00e1m v s\u00fa\u010dasnosti generuje viac ako 6 mili\u00f3nov spr\u00e1v za min\u00fatu v celom syst\u00e9me. \u010ealej budem ukazova\u0165, ako sa s t\u00fdm sna\u017e\u00edme \u017ei\u0165 a pre\u010do je to tak\u00e9.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/0ba5fcee5064865e836cefd47c40d246.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S t\u00fdmito 6 mili\u00f3nmi spr\u00e1v mus\u00edme nejak\u00fdm sp\u00f4sobom \u017ei\u0165. \u010co s nimi mus\u00edme urobi\u0165? 6 mili\u00f3nov spr\u00e1v, ktor\u00e9 mus\u00edme:<\/p>\n<p><\/p>\n<ul>\n<li>odosla\u0165 z aplik\u00e1cie<\/li>\n<li>prija\u0165 na doru\u010denie<\/li>\n<li>doru\u010di\u0165 na anal\u00fdzu a ulo\u017eenie.<\/li>\n<li>analyzova\u0165<\/li>\n<li>nejako uklada\u0165.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/a449062193c4b1757a61f5c92e270a74.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ke\u010f sa objavilo tri mili\u00f3ny spr\u00e1v, vyzeral som pribli\u017ene takto. Preto\u017ee sme za\u010d\u00ednali s nejak\u00fdmi centami. Je jasn\u00e9, \u017ee sa tam zapisuj\u00fa logy aplik\u00e1ci\u00ed. Napr\u00edklad, nepodarilo sa pripoji\u0165 k datab\u00e1ze, podarilo sa pripoji\u0165 k datab\u00e1ze, ale nepodarilo sa nie\u010do pre\u010d\u00edta\u0165. Ale okrem toho ka\u017ed\u00fd n\u00e1\u0161 mikroslu\u017eba zapisuje aj access-log. Ka\u017ed\u00fd po\u017eiadavok, ktor\u00fd pristane na mikroslu\u017ebe, padne do logu. Pre\u010do to rob\u00edme? V\u00fdvoj\u00e1ri chc\u00fa ma\u0165 mo\u017enos\u0165 sledovania. V ka\u017edom access-logu le\u017e\u00ed pole traceid, pod\u013ea ktor\u00e9ho \u0161peci\u00e1lne rozhranie odkr\u00fdva cel\u00fa re\u0165az a pekne zobrazuje sledovanie. Sledovanie ukazuje, ako prebiehala po\u017eiadavka, a to pom\u00e1ha na\u0161im v\u00fdvoj\u00e1rom r\u00fdchlej\u0161ie sa vysporiada\u0165 so v\u0161etkou neidentifikovanou anom\u00e1liou.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/e13a2d37ce57e894d023ad405a2b5200.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako s t\u00fdm \u017ei\u0165? Teraz v skratke op\u00ed\u0161em mo\u017enosti \u2014 ako sa t\u00e1to probl\u00e9matika v\u00f4bec rie\u0161i. Ako rie\u0161i\u0165 \u00falohu zbierania, prenosu a ukladania logov. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/60af5e5db804770295741995adb37d90.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako zapisova\u0165 z aplik\u00e1cie? Je jasn\u00e9, \u017ee existuj\u00fa r\u00f4zne sp\u00f4soby. Predov\u0161etk\u00fdm existuj\u00fa best practices, ako hovoria m\u00f3dni odborn\u00edci. Existuje aj star\u00e1 \u0161kola v dvoch variantoch, ako rozpr\u00e1vali na\u0161i predkovia. Existuj\u00fa aj in\u00e9 sp\u00f4soby. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/deab57e4ac0472a8591c0fad9e9f2910.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>So zbieran\u00edm logov je situ\u00e1cia pribli\u017ene rovnak\u00e1. Mo\u017enost\u00ed rie\u0161enia tejto konkr\u00e9tnej \u010dasti nie je a\u017e tak ve\u013ea. Je ich u\u017e viac, ale e\u0161te st\u00e1le nie tak ve\u013ea. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/8a6004aaeb931332983a975d0fa5c1d6.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S dod\u00e1van\u00edm a n\u00e1slednou anal\u00fdzou sa v\u0161ak po\u010det vari\u00e1ci\u00ed za\u010d\u00edna explodova\u0165. Ka\u017ed\u00fd variant teraz nebudem opisova\u0165. Mysl\u00edm, \u017ee hlavn\u00e9 mo\u017enosti s\u00fa zn\u00e1me ka\u017ed\u00e9mu, kto sa zauj\u00edmal o t\u00fato t\u00e9mu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/6a0b64c7e58bc59f56aaddd27112ec51.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Uk\u00e1\u017eem, ako sme to robili v Lazada, a ako vlastne v\u0161etko za\u010dalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/20f3aa5ceea8b28733ce5399bc30f5b5.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pred rokom som pri\u0161iel do Lazady a poslali ma na projekt o logoch. Bolo to pribli\u017ene takto. Log z aplik\u00e1cie sa zapisoval do stdout a stderr. V\u0161etko bolo uroben\u00e9 \u0161t\u00fdlovo. Ale potom to v\u00fdvoj\u00e1ri vyhodili zo \u0161tandardn\u00fdch pr\u00fadov, a potom sa s t\u00fdm nejak\u00fdm sp\u00f4sobom vysporiadaj\u00fa odborn\u00edci na infra\u0161trukt\u00faru. Medzi odborn\u00edkmi na infra\u0161trukt\u00faru a v\u00fdvoj\u00e1rmi s\u00fa e\u0161te release mana\u017e\u00e9ri, ktor\u00ed povedali: \u201eehm\u2026 no dobre, zabali\u0165 ich do s\u00faboru jednoducho shellom a koniec\u201c. A ke\u010f\u017ee to v\u0161etko bolo v kontejnery, zabali\u0165 to priamo do samotn\u00e9ho kontajnera, mapovali sme dovn\u00fatra prie\u010dinok a polo\u017eili to tam. Mysl\u00edm, \u017ee je v\u0161etk\u00fdm pribli\u017ene jasn\u00e9, \u010do z toho vzniklo.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/d92e02c18a4c683c9389321e4d5412df.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pozrime sa zatia\u013e trochu \u0161ir\u0161ie. Ako sme tieto logy doru\u010dovali. Niekto si vybral td-agent, ktor\u00fd je v skuto\u010dnosti fluentd, ale nie \u00faplne fluentd. Ja som nikdy nepochopil vz\u0165ah t\u00fdchto dvoch projektov, ale zd\u00e1 sa, \u017ee s\u00fa o tom istom. A tento fluentd, nap\u00edsan\u00fd v Ruby, \u010d\u00edtal s\u00fabory logov, parsel ich do JSON pod\u013ea nejak\u00fdch regul\u00e1rnych v\u00fdrazov. Potom ich odosielal do Kafka. A v Kafka sme mali pre ka\u017ed\u00e9 API \u0161tyri samostatn\u00e9 topic-y. Pre\u010do \u0161tyri? Preto\u017ee m\u00e1me live, staging a preto existuje stdout a stderr. V\u00fdvoj\u00e1ri ich vytv\u00e1raj\u00fa, a infra\u0161trukt\u00farni odborn\u00edci ich musia vytv\u00e1ra\u0165 v Kafka. A Kafka bola kontrolovan\u00e1 in\u00fdm oddelen\u00edm. Preto bolo potrebn\u00e9 vytvori\u0165 tiket, aby vytvorili \u0161tyri topic-y na ka\u017ed\u00e9 API. V\u0161etci na to zab\u00fadali. Celkovo to bola chaotick\u00e1 situ\u00e1cia. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/3130f9ebfda22713cffacf10db45ca89.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010co sme s t\u00fdm \u010falej robili? Odosielali sme to do Kafka. Potom z Kafka polovica logov odletela do Logstash. Druh\u00e1 polovica logov sa rozdelila. \u010cas\u0165 odletela do jedn\u00e9ho Graylogu, \u010das\u0165 \u2013 do druh\u00e9ho Graylogu. Nakoniec to v\u0161etko skon\u010dilo v jednom Elasticsearch klastri. Teda, cel\u00fd tento chaos nakoniec padol tam. Takto sa to nerob\u00ed!<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/98bf4811c102fc12fcd1114c25886b69.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Toto je takto vyzer\u00e1, ak sa na to pozrieme z dia\u013eky. Takto to nerobte! Tu s\u00fa \u010d\u00edslicami okam\u017eite ozna\u010den\u00e9 problematick\u00e9 miesta. V skuto\u010dnosti ich je viac, ale 6 \u2014 to s\u00fa naozaj ve\u013emi problematick\u00e9, s ktor\u00fdmi je potrebn\u00e9 nie\u010do urobi\u0165. O nich teraz zvl\u00e1\u0161\u0165 poviem. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/69f7682e6677bdda3068d6dea4db7732.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu (1,2,3) sa p\u00ed\u0161u s\u00fabory, a teda tu s\u00fa hne\u010f tri prek\u00e1\u017eky. <\/p>\n<p><\/p>\n<p>Prv\u00e9 (1) \u2014 mus\u00edme ich niekam zapisova\u0165. Nie v\u017edy by sme chceli poskytn\u00fa\u0165 API mo\u017enos\u0165 zapisova\u0165 priamo do s\u00faboru. Je dobr\u00e9, aby API bolo izolovan\u00e9 v kontajneri, a e\u0161te lep\u0161ie \u2013 aby bolo v re\u017eime iba na \u010d\u00edtanie. Ja som sysadmin, tak\u017ee m\u00e1m trochu alternat\u00edvny poh\u013ead na tieto veci.<\/p>\n<p><\/p>\n<p>Druh\u00fd bod (2,3) \u2014 prich\u00e1dza k n\u00e1m mno\u017estvo po\u017eiadaviek do API. API zapisuje ve\u013ea \u00fadajov do s\u00faboru. S\u00fabory rast\u00fa. Mus\u00edme ich rotova\u0165. Lebo inak na disku nezostane miesto. Rot\u00e1cia je zlo\u017eit\u00e1, preto\u017ee s\u00fa vytvoren\u00e9 preposielan\u00edm cez shell do adres\u00e1ra. Nem\u00f4\u017eeme to rotova\u0165. Nemo\u017eno poveda\u0165 aplik\u00e1cii, aby znovu otvorila deskriptory. Preto\u017ee v\u00fdvoj\u00e1ri sa na v\u00e1s pozr\u00fa ako na bl\u00e1zna: \u201eAk\u00e9 deskriptory? My p\u00ed\u0161eme rovno do stdout.\u201c Infrastrukt\u00farny t\u00edm urobil copytruncate v logrotate, ktor\u00fd jednoducho skop\u00edruje s\u00fabor a zmen\u0161\u00ed p\u00f4vodn\u00fd. Tak\u017ee medzi t\u00fdmito procesmi kop\u00edrovania v\u00e4\u010d\u0161inou d\u00f4jde k zaplneniu miesta na disku.<\/p>\n<p><\/p>\n<p>(4) Mali sme r\u00f4zne form\u00e1ty v r\u00f4znych API. Trochu sa l\u00ed\u0161ili, ale bolo potrebn\u00e9 p\u00edsa\u0165 r\u00f4zne regexp. Ke\u010f\u017ee to v\u0161etko spravoval Puppet, bola tam ve\u013ek\u00e1 k\u00f4pka tried so svojimi probl\u00e9mami. Navy\u0161e, td-agent mohol v\u00e4\u010d\u0161inu \u010dasu spotrebov\u00e1va\u0165 pam\u00e4\u0165 a spoma\u013eova\u0165, mohol sa tv\u00e1ri\u0165, \u017ee pracuje, a ni\u010d nerobi\u0165. Z vonku bolo nemo\u017en\u00e9 pochopi\u0165, \u017ee ni\u010d nerob\u00ed. V tom lep\u0161om pr\u00edpade spadol, a niekto ho nesk\u00f4r znovu na\u0161tartoval. Presnej\u0161ie, pri\u0161iel alert a niekto ho \u0161iel manu\u00e1lne re\u0161tartova\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/424cda2b60e9e5669b7ef2b34ca74e48.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>(6) A najhor\u0161ie a naj\u0161ialenej\u0161ie bolo elasticsearch. Preto\u017ee to bola star\u00e1 verzia. Nemali sme vtedy vyhraden\u00fdch majstrov. Mali sme r\u00f4znorod\u00e9 protokoly, ktor\u00fdch polia sa mohli prekry\u0165. R\u00f4zne protokoly r\u00f4znych aplik\u00e1ci\u00ed mohli by\u0165 p\u00edsan\u00e9 so rovnak\u00fdmi n\u00e1zvami pol\u00ed, ale vn\u00fatri mohli ma\u0165 r\u00f4zne \u00fadaje. Teda, jeden protokol prich\u00e1dza s Integer v poli, napr\u00edklad, level. Druh\u00fd protokol prich\u00e1dza so String v poli level. Bez statick\u00e9ho mapovania dostaneme tak\u00fd \u00fa\u017easn\u00fd probl\u00e9m. Ak sa po rot\u00e1cii indexu v elasticsearch ako prv\u00e9 dostane spr\u00e1va so stringom, v\u0161etko je v poriadku. Ak sa prv\u00e9 dostane s Integer, v\u0161etky nasleduj\u00face spr\u00e1vy, ktor\u00e9 pr\u00eddu so String, sa jednoducho odmietaj\u00fa. Preto\u017ee typ po\u013ea sa nezhoduje.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/bc869f827bed326792857113191bb028.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Za\u010dali sme sa p\u00fdta\u0165 na tieto ot\u00e1zky. Rozhodli sme sa neh\u013eada\u0165 vinn\u00edkov.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/a071a4b38462f249339c988c35463639.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale nie\u010do mus\u00edme robi\u0165! Je to jasn\u00e1 vec \u2014 mus\u00edme zavies\u0165 \u0161tandardy. Niektor\u00e9 \u0161tandardy sme u\u017e mali. Niektor\u00e9 sme zaviedli nesk\u00f4r. Na\u0161\u0165astie, jednotn\u00fd form\u00e1t logov pre v\u0161etky API bol u\u017e v tom \u010dase schv\u00e1len\u00fd. Je zap\u00edsan\u00fd priamo v \u0161tandardoch interakcie slu\u017eieb. T\u00ed, ktor\u00ed chc\u00fa prij\u00edma\u0165 logy, ich musia p\u00edsa\u0165 v tomto form\u00e1te. Ak niekto nep\u00ed\u0161e logy v tomto form\u00e1te, znamen\u00e1 to, \u017ee ni\u010d negarantujeme. <\/p>\n<p><\/p>\n<p>\u010ealej by sme chceli zavies\u0165 jednotn\u00fd \u0161tandard na sp\u00f4soby z\u00e1pisu, doru\u010denia a zbierania logov. V podstate, kam ich p\u00edsa\u0165 a \u010d\u00edm ich doru\u010dova\u0165. Ide\u00e1lna situ\u00e1cia je, ke\u010f sa v projektoch pou\u017e\u00edva rovnak\u00e1 kni\u017enica. Existuje samostatn\u00e1 kni\u017enica logovania pre Go, existuje osobitn\u00e1 kni\u017enica pre PHP. V\u0161etci, ktor\u00ed u n\u00e1s s\u00fa \u2014 musia ich pou\u017e\u00edva\u0165. V s\u00fa\u010dasnosti by som povedal, \u017ee to m\u00e1me na 80 %. Ale niektor\u00ed st\u00e1le jedia kaktusy.<\/p>\n<p><\/p>\n<p>A tam (na sn\u00edmke) len ve\u013emi pomaly za\u010d\u00edna vystupova\u0165 \u201eSLA na doru\u010denie logov\u201c. Zatia\u013e to neexistuje, ale pracujeme na tom. Preto\u017ee je ve\u013emi pohodln\u00e9, ke\u010f infra\u0161trukt\u00fara hovor\u00ed, \u017ee ak p\u00ed\u0161ete v takomto form\u00e1te na tak\u00e9to miesto a nie viac ako N spr\u00e1v za sekundu, potom to s takou pravdepodobnos\u0165ou doru\u010d\u00edme tam. To odb\u00farava mno\u017estvo bolesti hlavy. Ak SLA existuje, je to skvel\u00e9!<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/a887aceac13135afe90bb5f5a5ec1d26.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako sme za\u010dali rie\u0161i\u0165 probl\u00e9m? Hlavn\u00fd probl\u00e9m bol s td-agent. Nevedeli sme, kam sa na\u0161e logy dost\u00e1vaj\u00fa. Doru\u010duj\u00fa sa? Zbieraj\u00fa sa? Kde s\u00fa vlastne? Preto sa rozhodlo v prvom bode nahradi\u0165 td-agent. Mo\u017enosti, \u010d\u00edm ho nahradi\u0165, som tu stru\u010dne na\u010drtol.<\/p>\n<p><\/p>\n<p>Fluentd. Po prv\u00e9, stretol som sa s n\u00edm pri predch\u00e1dzaj\u00facej pr\u00e1ci a tam tie\u017e ob\u010das zlyhal. Po druh\u00e9, je to to ist\u00e9, len v profile. <\/p>\n<p><\/p>\n<p>Filebeat. \u010c\u00edm bol pre n\u00e1s pohodln\u00fd? T\u00fdm, \u017ee je na Go, a my m\u00e1me ve\u013ek\u00fa expert\u00edzu v Go. Preto ak by bolo treba, mohli sme ho prisp\u00f4sobi\u0165. Preto sme ho nezobrali. Aby sa ani nenaskytla my\u0161lienka za\u010da\u0165 ho prep\u00edsa\u0165. <\/p>\n<p><\/p>\n<p>Evidentn\u00fdm rie\u0161en\u00edm pre sysadmina zost\u00e1vaj\u00fa r\u00f4zne syslogy v takomto mno\u017estve (syslog-ng\/rsyslog\/nxlog).<\/p>\n<p><\/p>\n<p>Alebo nap\u00edsa\u0165 nie\u010do vlastn\u00e9, ale to sme odmietli, rovnako ako filebeat. Ak nie\u010do p\u00edsa\u0165, tak je lep\u0161ie p\u00edsa\u0165 nie\u010do u\u017eito\u010dn\u00e9 pre biznis. Na doru\u010denie logov je lep\u0161ie zobra\u0165 nie\u010do hotov\u00e9.<\/p>\n<p><\/p>\n<p>Preto sa v\u00fdber vlastne z\u00fa\u017eil na v\u00fdber medzi syslog-ng a rsyslog. Naklonil som sa k rsyslog len preto, \u017ee sme u\u017e mali v Puppet triedy pre rsyslog a nena\u0161iel som medzi nimi o\u010dividn\u00fd rozdiel. \u010co tam syslog, to tu syslog. \u00c1no, niekto m\u00e1 zl\u00fa dokument\u00e1ciu, niekto lep\u0161iu. Ten vie to takto, a ten \u2014 inak.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/a88116711b547fa5b2b34daf72cc4220.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A trochu o rsyslog. Po prv\u00e9, je skvel\u00fd, preto\u017ee m\u00e1 ve\u013ea modulov. M\u00e1 \u010dloveku zrozumite\u013en\u00fd RainerScript (modern\u00fd jazyk konfigur\u00e1cie). \u00da\u017easn\u00fd bonus je, \u017ee sme mohli jeho \u0161tandardn\u00fdmi prostriedkami simulova\u0165 spr\u00e1vanie td-agent, a pre aplik\u00e1cie sa ni\u010d nezmenilo. To znamen\u00e1, \u017ee vymen\u00edme td-agent za rsyslog a v\u0161etko ostatn\u00e9 zatia\u013e nemen\u00edme. A okam\u017eite dostaneme funguj\u00face doru\u010denie. \u010ealej, mmnormalize \u2014 to je \u00fa\u017easn\u00e1 vec v rsyslog. Umo\u017e\u0148uje analyzova\u0165 logy, ale nie pomocou Grok a regexp. Vytv\u00e1ra abstraktn\u00fd syntaktick\u00fd strom. Analyzuje logy podobne, ako kompil\u00e1tor analyzuje zdrojov\u00e9 k\u00f3dy. To umo\u017e\u0148uje pracova\u0165 ve\u013emi r\u00fdchlo, spotrebova\u0165 m\u00e1lo CPU, a v\u00f4bec, je to naozaj skvel\u00e1 vec. Je tu mno\u017estvo \u010fal\u0161\u00edch bonusov. Na t\u00fdch sa nebudem zastavova\u0165.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/ca811be9f9ff318fb8c8bbf3dc45a038.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rsyslog m\u00e1 e\u0161te mno\u017estvo nedostatkov. S\u00fa pribli\u017ene tak\u00e9 ist\u00e9 ako bonusy. Hlavn\u00e9 probl\u00e9my \u2014 je potrebn\u00e9 vedie\u0165 ho nastavi\u0165 a je potrebn\u00e9 vybera\u0165 verziu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/034f588e0834b1d4e4873d9c3dbd0873.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rozhodli sme sa, \u017ee budeme zapisova\u0165 logy do unixov\u00e9ho socketu. A to nie do \/dev\/log, preto\u017ee tam m\u00e1me chaos zo syst\u00e9mov\u00fdch logov, tam je journald v tomto pipeline. Preto p\u00ed\u0161me do vlastn\u00e9ho socketu. Pripneme ho k samostatn\u00e9mu rulesetu. Ni\u010d nebudeme mie\u0161a\u0165. V\u0161etko bude transparentn\u00e9 a jasn\u00e9. Tak sme vlastne urobili. Katal\u00f3g s t\u00fdmito socketmi je \u0161tandardizovan\u00fd a pren\u00e1\u0161a sa do v\u0161etk\u00fdch kontajnerov. Kontajnery m\u00f4\u017eu vidie\u0165 potrebn\u00fd socket, otvori\u0165 ho a zapisova\u0165 do neho. <\/p>\n<p><\/p>\n<p>Pre\u010do nie s\u00fabor? Preto\u017ee v\u0161etci \u010d\u00edtali <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/badoo\/blog\/280606\/\">\u010dl\u00e1nok o Badu\u0161ke<\/a><\/noindex>, ktor\u00e1 sa sna\u017eila prenies\u0165 s\u00fabor do dockeru, a zistilo sa, \u017ee po re\u0161tarte rsyslog sa men\u00ed deskriptor s\u00faboru, a docker str\u00e1ca tento s\u00fabor. Dr\u017e\u00ed otvoren\u00fd nie\u010do in\u00e9, ale u\u017e nie ten socket, do ktor\u00e9ho sa p\u00ed\u0161e. Rozhodli sme sa, \u017ee t\u00fato problematiku ob\u00eddeme a z\u00e1rove\u0148 ob\u00eddeme probl\u00e9m blokovania.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/78db15581a68ce16d11f5ebe7e6fe04e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rsyslog vykon\u00e1va akcie uveden\u00e9 na sn\u00edmke a odosiela logy bu\u010f do relay, alebo do Kafka. Kafka zodpoved\u00e1 star\u00e9mu sp\u00f4sobu. Relay \u2014 to som sa pok\u00fasil pou\u017ei\u0165 \u010disto rsyslog pre doru\u010denie logov. Bez Message Queue, \u0161tandardn\u00fdmi prostriedkami rsyslog. V podstate to funguje.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/e61a25005ecbdc86d9de3215ee6fb97e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuj\u00fa v\u0161ak \u0161pecifik\u00e1, pokia\u013e ide o to, ako ich n\u00e1sledne umiestni\u0165 do tejto \u010dasti (Logstash\/Graylog\/ES). T\u00e1to \u010das\u0165 (rsyslog-rsyslog) sa pou\u017e\u00edva medzi d\u00e1tov\u00fdmi centrami. Je tu skomprimovan\u00e9 tcp spojenie, ktor\u00e9 \u0161etr\u00ed \u0161\u00edrku p\u00e1sma a t\u00fdm zvy\u0161uje pravdepodobnos\u0165, \u017ee z\u00edskame nejak\u00e9 logy z in\u00e9ho d\u00e1tov\u00e9ho centra, ke\u010f je kan\u00e1l pre\u0165a\u017een\u00fd. Pre\u010do? Preto\u017ee m\u00e1me Indon\u00e9ziu, kde je situ\u00e1cia zl\u00e1. Tam je to neust\u00e1ly probl\u00e9m.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/7509ba360be2db4dec0c8679225d285f.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zamysleli sme sa nad t\u00fdm, ako v skuto\u010dnosti monitorova\u0165 pravdepodobnos\u0165, \u017ee logy, ktor\u00e9 zapisujeme z aplik\u00e1cie, sa dostan\u00fa na druh\u00fd koniec? Rozhodli sme sa zavies\u0165 metriky. Rsyslog m\u00e1 svoj modul na zber \u0161tatist\u00edk, ktor\u00fd obsahuje nie\u010do ako po\u010d\u00edtadl\u00e1. M\u00f4\u017ee napr\u00edklad zobrazi\u0165 ve\u013ekos\u0165 fronty alebo po\u010det spr\u00e1v, ktor\u00e9 pri\u0161li v konkr\u00e9tnej akcii. Z toho u\u017e m\u00f4\u017eeme nie\u010do z\u00edska\u0165. Navy\u0161e m\u00e1 aj vlastn\u00e9 po\u010d\u00edtadl\u00e1, ktor\u00e9 m\u00f4\u017eeme nastavi\u0165, a tie n\u00e1m ukazuj\u00fa napr\u00edklad po\u010det spr\u00e1v, ktor\u00e9 zaznamenalo nejak\u00e9 API. Potom som nap\u00edsal rsyslog_exporter v Pythone a v\u0161etko sme to poslali do Prometheus a vytvorili grafy. Ve\u013emi sme t\u00fa\u017eili po metrik\u00e1ch Graylog, ale zatia\u013e sme ich nestihli nastavi\u0165.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/ef9bd61fc43e056b4a56240c85e7fd20.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S ak\u00fdmi probl\u00e9mami sme sa stretli? Probl\u00e9my vznikli, ke\u010f sme zistili (\u00faplne ne\u010dakane!), \u017ee na\u0161e Live API generuje 50 tis\u00edc spr\u00e1v za sekundu. To je len Live API bez stagingu. A Graylog n\u00e1m zobrazuje len 12 tis\u00edc spr\u00e1v za sekundu. A objavil sa opr\u00e1vnen\u00fd dotaz, kde s\u00fa zvy\u0161n\u00e9 spr\u00e1vy? Z \u010doho sme us\u00fadili, \u017ee Graylog jednoducho nezvl\u00e1da. Pozreli sme sa a naozaj, Graylog s Elasticsearch nezvl\u00e1daj\u00fa tento tok.<\/p>\n<p><\/p>\n<p>\u010ealej, in\u00e9 objavy, ktor\u00e9 sme urobili v procese.<\/p>\n<p><\/p>\n<p>Z\u00e1znam do socketu je pozastaven\u00fd. Ako sa to stalo? Ke\u010f som pou\u017e\u00edval rsyslog na doru\u010dovanie, v ur\u010ditom momente sa n\u00e1m zlomil kan\u00e1l medzi d\u00e1tov\u00fdmi centrami. Doru\u010dovanie sa zastavilo na jednom mieste, zastavilo sa na druhom mieste. To v\u0161etko sa prenieslo na stroj s API, ktor\u00e9 zapisovali do sockets rsyslog. Tam sa zaplnila fronta. Potom sa zaplnila fronta na z\u00e1pis do unix socketu, ktor\u00e1 je predvolene 128 paketov. A nasleduj\u00faci write() v aplik\u00e1cii je zablokovan\u00fd. Ke\u010f sme sa pozreli do kni\u017enice, ktor\u00fa pou\u017e\u00edvame v aplik\u00e1ci\u00e1ch na Go, bolo tam nap\u00edsan\u00e9, \u017ee z\u00e1pis do socketu prebieha v neblokuj\u00facom re\u017eime. Boli sme presved\u010den\u00ed, \u017ee ni\u010d nie je zablokovan\u00e9. Preto\u017ee sme \u010d\u00edtali. <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/badoo\/blog\/280606\/\">\u010dl\u00e1nok o Badu\u0161ke<\/a><\/noindex>, ktor\u00e1 o tom nap\u00edsala. Ale je tu e\u0161te jeden moment. Okolo tohto volania bol e\u0161te nekone\u010dn\u00fd cyklus, v ktorom sa neust\u00e1le sna\u017eilo zasun\u00fa\u0165 spr\u00e1vu do soketu. To sme si nev\u0161imli. Museli sme prepisova\u0165 kni\u017enicu. Odvtedy sa p\u00e1rkr\u00e1t zmenila, ale teraz sme sa zbavili blokovania vo v\u0161etk\u00fdch subsystemoch. Preto m\u00f4\u017eeme zastavi\u0165 rsyslog a ni\u010d nespadne.<\/p>\n<p><\/p>\n<p>Mus\u00edme monitorova\u0165 ve\u013ekos\u0165 frontov, \u010do pom\u00e1ha vyhn\u00fa\u0165 sa t\u00fdmto probl\u00e9mom. Po prv\u00e9, m\u00f4\u017eeme monitorova\u0165, kedy za\u010d\u00edname str\u00e1ca\u0165 spr\u00e1vy. Po druh\u00e9, m\u00f4\u017eeme sledova\u0165, \u010di m\u00e1me v\u0161eobecne probl\u00e9my s doru\u010dovan\u00edm.<\/p>\n<p><\/p>\n<p>A e\u0161te nepr\u00edjemn\u00fd moment \u2014 amplifik\u00e1cia desa\u0165kr\u00e1t v mikroservisnej architekt\u00fare \u2014 to je naozaj ve\u013emi jednoduch\u00e9. M\u00e1me nie ve\u013ea prich\u00e1dzaj\u00facich po\u017eiadaviek, ale kv\u00f4li grafu, ktor\u00fdm tieto spr\u00e1vy n\u00e1sledne prech\u00e1dzaj\u00fa, kv\u00f4li access-logom, naozaj zvy\u0161ujeme z\u00e1\u0165a\u017e na logy pribli\u017ene desa\u0165kr\u00e1t. Bohu\u017eia\u013e, nestihol som vypo\u010d\u00edta\u0165 presn\u00e9 \u010d\u00edsla, ale mikroservisy \u2014 s\u00fa tak\u00e9. Mus\u00ed sa to ma\u0165 na pam\u00e4ti. V s\u00fa\u010dasnosti je teda subsyst\u00e9m zbierania logov najviac za\u0165a\u017een\u00fd v Lazada. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/d76c6d91613ff77c41b8985cee83218f.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako vyrie\u0161i\u0165 probl\u00e9m s elasticsearch? Ak potrebujete r\u00fdchlo z\u00edska\u0165 logy na jednom mieste, aby ste nemali probl\u00e9m behan\u00edm po v\u0161etk\u00fdch strojoch, a nesbiera\u0165 ich tam, pou\u017eite s\u00faborov\u00e9 \u00falo\u017eisko. To zaru\u010dene funguje. M\u00f4\u017ee sa vytvori\u0165 z ak\u00e9hoko\u013evek servera. Sta\u010d\u00ed prida\u0165 disky a nastavi\u0165 syslog. Potom budete ma\u0165 v\u0161etky logy zaru\u010dene na jednom mieste. A \u010falej m\u00f4\u017eete pokojne nastavova\u0165 elasticsearch, graylog, alebo nie\u010do in\u00e9. Ale u\u017e budete ma\u0165 v\u0161etky logy, a m\u00f4\u017eete ich uchov\u00e1va\u0165, ako dlh\u00e9 s\u00fa va\u0161e diskov\u00e9 array.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/571297f25de76958769f3a4b9bfbf5b9.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>K\u00fdm som mal svoju prezent\u00e1ciu, sch\u00e9ma vyzerala takto. Na s\u00fabor sme prakticky prestali zapisova\u0165. Teraz pravdepodobne vypneme zostatky. Na lok\u00e1lnych strojoch, na ktor\u00fdch s\u00fa spusten\u00e9 API, prestaneme zapisova\u0165 do s\u00faborov. Po prv\u00e9, m\u00e1me s\u00faborov\u00e9 \u00falo\u017eisko, ktor\u00e9 funguje ve\u013emi dobre. Po druh\u00e9, na t\u00fdchto strojoch sa neust\u00e1le m\u00ed\u0148a miesto, a je potrebn\u00e9 ho neust\u00e1le monitorova\u0165.<\/p>\n<p><\/p>\n<p>T\u00e1to \u010das\u0165 s Logstash a Graylog naozaj ob\u0165a\u017euje. Preto sa jej treba zbavi\u0165. Je potrebn\u00e9 vybra\u0165 nie\u010do jedno.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/4a285577708aa1c55c16a72364fa8072.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rozhodli sme sa zbavi\u0165 Logstash a Kibana. Pre\u010do? Preto\u017ee m\u00e1me bezpe\u010dnostn\u00fd oddelenie. Ak\u00fd to m\u00e1 v\u00fdznam? To, \u017ee Kibana bez X-Pack a Shield neumo\u017e\u0148uje obmedzi\u0165 pr\u00edstupov\u00e9 pr\u00e1va k logom. Tak\u017ee sme si zvolili Graylog. Nie je to m\u00f4j ob\u013e\u00faben\u00fd, ale funguje. K\u00fapili sme nov\u00fd hardv\u00e9r, nain\u0161talovali sme tam \u010derstv\u00fd Graylog a presunuli v\u0161etky logy so striktne definovan\u00fdmi form\u00e1tmi do samostatn\u00e9ho Graylogu. Rie\u0161ili sme probl\u00e9m s r\u00f4znymi typmi rovnak\u00fdch pol\u00ed organiza\u010dne.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/91b9fdada2648ae1caec05169b11926a.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010co presne zah\u0155\u0148a nov\u00fd Graylog? Jednoducho sme v\u0161etko nahrali do Dockera. Zobrali sme nieko\u013eko serverov, rozlo\u017eili tri in\u0161tancie Kafka, 7 serverov Graylog verzie 2.3 (preto\u017ee sme chceli Elasticsearch verziu 5). To v\u0161etko sme spustili na RAIDoch z HDD. Uvideli sme r\u00fdchlos\u0165 indexovania do 100 tis\u00edc spr\u00e1v za sekundu. Uvideli sme \u010d\u00edslo, \u017ee 140 terabajtov \u00fadajov za t\u00fd\u017ede\u0148. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/05f9b3c06739548f092bbe271bf8fc54.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A znova probl\u00e9my! Chystaj\u00fa sa n\u00e1m dve v\u00fdpredaje. Presunuli sme sa k 6 mili\u00f3nom spr\u00e1v. Na\u0161e Graylog nest\u00edha spracov\u00e1va\u0165. Mus\u00edme sa znova nejako prisp\u00f4sobi\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/424aa2febc2fd903aabee4aeabb64010.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Takto sme pre\u017eili. Pridali sme e\u0161te p\u00e1r serverov a SSD. Moment\u00e1lne \u017eijeme t\u00fdmto sp\u00f4sobom. Teraz spracov\u00e1vame u\u017e 160 tis\u00edc spr\u00e1v za sekundu. E\u0161te sme nedosiahli limit, tak\u017ee zatia\u013e nie je jasn\u00e9, ko\u013eko naozaj zvl\u00e1dneme. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/17d84fa7de5403551e55f42345e62a1a.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Toto s\u00fa na\u0161e pl\u00e1ny do bud\u00facnosti. Z nich je, pravdepodobne, najd\u00f4le\u017eitej\u0161ia vysok\u00e1 dostupnos\u0165. E\u0161te ju nem\u00e1me. Nieko\u013eko strojov je nastaven\u00fdch rovnako, ale v\u0161etko putuje zatia\u013e cez jeden server. Mus\u00edme str\u00e1vi\u0165 \u010das nastaven\u00edm failoveru medzi nimi.<\/p>\n<p><\/p>\n<p>Zbiera\u0165 metriky z Graylogu.<\/p>\n<p><\/p>\n<p>Vytvori\u0165 rate limit, aby n\u00e1m jedna, \u0161ialen\u00e1 API, nezni\u010dila \u0161\u00edrku p\u00e1sma a v\u0161etko ostatn\u00e9.<\/p>\n<p><\/p>\n<p>A nakoniec podp\u00edsa\u0165 nejak\u00fd SLA s v\u00fdvoj\u00e1rmi, \u017ee m\u00f4\u017eeme obsl\u00fa\u017ei\u0165 toto\u013eko. Ak p\u00ed\u0161ete viac, tak prep\u00e1\u010dte.<\/p>\n<p><\/p>\n<p>A nap\u00edsa\u0165 dokument\u00e1ciu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zbierania a doru\u010dovania logov\u201c \u2014 prepis predn\u00e1\u0161ky\" src=\"\/wp-content\/uploads\/2019\/04\/4a78e2834d5877f8bf17d77a44000c22.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kr\u00e1tko, z\u00e1very v\u0161etk\u00e9ho, \u010do sme pre\u017eili. Po prv\u00e9, \u0161tandardy. Po druh\u00e9, syslog \u2014 torta. Po tretie, rsyslog funguje presne tak, ako je nap\u00edsan\u00e9 na sn\u00edmke. A po\u010fme k ot\u00e1zkam.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zky<\/strong>.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pre\u010do ste sa nakoniec rozhodli nezobra\u0165\u2026 (filebeat?)<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Mus\u00ed sa p\u00edsa\u0165 do s\u00faboru. Ve\u013emi sa mi to nechcelo. Ke\u010f m\u00e1\u0161 API, ktor\u00e9 p\u00ed\u0161e tis\u00edce spr\u00e1v za sekundu, aj ke\u010f bude\u0161 rotova\u0165 raz za hodinu, nie je to st\u00e1le cesta. M\u00f4\u017ee\u0161 p\u00edsa\u0165 do potrubia. Na \u010do sa ma v\u00fdvoj\u00e1ri sp\u00fdtali: \u00ab\u010co sa stane, ak proces, do ktor\u00e9ho p\u00ed\u0161eme, padne\u00bb? Proste som nena\u0161iel, \u010do im odpoveda\u0165, a povedal som: \u00abNo dobre, tak to nerobme\u00bb.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pre\u010do nezapisujete logy priamo do HDFS?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Toto je \u010fal\u0161\u00ed krok. Prem\u00fd\u0161\u013eali sme o \u0148om na za\u010diatku, ale moment\u00e1lne na to nem\u00e1me zdroje, tak\u017ee to m\u00e1me v dlhodobom rie\u0161en\u00ed.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: St\u013apcov\u00fd form\u00e1t by bol vhodnej\u0161\u00ed.<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: V\u0161etko ch\u00e1pem. Sme \u201eza\u201c oboma rukami. <\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Zapisujete do rsyslog. Tam m\u00f4\u017ee by\u0165 TCP aj UDP. Ale ak je to UDP, ako zabezpe\u010d\u00edte dod\u00e1vku?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: S\u00fa tu dva aspekty. Po prv\u00e9, hne\u010f v\u0161etk\u00fdm hovor\u00edm, \u017ee nezaru\u010dujeme dod\u00e1vku logov. Preto\u017ee, ke\u010f pr\u00eddu v\u00fdvoj\u00e1ri a povedia: \u201ePo\u010fme tam za\u010da\u0165 zapisova\u0165 finan\u010dn\u00e9 \u00fadaje, a vy ich niekam ulo\u017e\u00edte pre pr\u00edpad, \u017ee by sa nie\u010do stalo\u201c, odpoved\u00e1me im \u201eSkvel\u00e9! Po\u010fte najprv za\u010da\u0165 blokova\u0165 z\u00e1pis do socketu a robi\u0165 to v transakci\u00e1ch, aby ste n\u00e1m to tam naozaj poslali a uistili sa, \u017ee sme to prijali\u201c A v tom momente sa v\u0161etk\u00fdm hne\u010f prestane chcie\u0165. A ke\u010f sa nechc\u00fa, tak ak\u00e9 k n\u00e1m ot\u00e1zky? Ak nechcete zaru\u010di\u0165 z\u00e1pis do socketu, pre\u010do by sme mali zaru\u010di\u0165 dod\u00e1vku? Rob\u00edme best effort. Naozaj sa sna\u017e\u00edme doda\u0165 \u010do najviac a \u010do najlep\u0161ie, ale ned\u00e1vame 100% z\u00e1ruku. Preto netreba zapisova\u0165 finan\u010dn\u00e9 \u00fadaje tam. Na to s\u00fa datab\u00e1zy s transakciami.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Ke\u010f API generuje nejak\u00fa spr\u00e1vu do logu a pred\u00e1va riadenie mikroservisom, nenastali v\u00e1m probl\u00e9my s t\u00fdm, \u017ee spr\u00e1vy od r\u00f4znych mikroservisov prich\u00e1dzaj\u00fa v nespr\u00e1vnom porad\u00ed? To sp\u00f4sobuje zm\u00e4tok.<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Je norm\u00e1lne, \u017ee prich\u00e1dzaj\u00fa v r\u00f4znom porad\u00ed. Treba na to by\u0165 pripraven\u00fd. Preto\u017ee ak\u00e1ko\u013evek sie\u0165ov\u00e1 dod\u00e1vka v\u00e1m nezaru\u010duje poradie, alebo mus\u00edte na to investova\u0165 \u0161peci\u00e1lne prostriedky. Ak vezmeme s\u00faborov\u00e9 \u00falo\u017eisk\u00e1, tak ka\u017ed\u00e9 API uklad\u00e1 logy do svojho s\u00faboru. Presnej\u0161ie, rsyslog ich triedi do prie\u010dinkov. Ka\u017ed\u00e9 API m\u00e1 svoje logy, do ktor\u00fdch sa d\u00e1 \u00eds\u0165 pozrie\u0165, a potom ich pod\u013ea timestampu v t\u00fdchto logoch m\u00f4\u017eeme porovna\u0165. Ak sa id\u00fa pozrie\u0165 do Graylogu, tam bud\u00fa zoraden\u00e9 pod\u013ea timestampu. Tam bude v\u0161etko v poriadku.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Timestamp m\u00f4\u017ee by\u0165 odli\u0161n\u00fd o milisekundy.<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Timestamp generuje samotn\u00e9 API. V tom je vlastne cel\u00e9 tajomstvo. M\u00e1me NTP. API generuje timestamp u\u017e v samotnej spr\u00e1ve. Nie je to prid\u00e1van\u00e9 rsyslogom.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Nie je ve\u013emi jasn\u00e9, ako prebieha interakcia medzi d\u00e1tov\u00fdmi centrami. V r\u00e1mci d\u00e1tov\u00e9ho centra je jasn\u00e9, ako sa logy zbieraj\u00fa a spracov\u00e1vaj\u00fa. Ako prebieha interakcia medzi d\u00e1tov\u00fdmi centrami? Alebo ka\u017ed\u00e9 d\u00e1tov\u00e9 centrum \u017eije svoj\u00edm vlastn\u00fdm \u017eivotom?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Takmer. Ka\u017ed\u00e1 krajina sa nach\u00e1dza v jednom d\u00e1tovom centre. Moment\u00e1lne nem\u00e1me rozprestret\u00e9 umiestnenie, aby jedna krajina bola umiestnen\u00e1 v r\u00f4znych d\u00e1tov\u00fdch centr\u00e1ch. Preto ich nie je potrebn\u00e9 sp\u00e1ja\u0165. V ka\u017edom centre je Log Relay. To je Rsyslog server. V skuto\u010dnosti s\u00fa to dve mana\u017e\u00e9rske stroje. S\u00fa nastaven\u00e9 rovnako. Ale zatia\u013e priebe\u017ene prech\u00e1dza prev\u00e1dzka cez jeden z nich. Agreguje v\u0161etky logy. M\u00e1 diskov\u00fa frontu pre ka\u017ed\u00fd pr\u00edpad. Komprimuje logy a odosiela ich do centr\u00e1ln\u00e9ho d\u00e1tov\u00e9ho centra (v Singapure), odkia\u013e u\u017e pokra\u010duj\u00fa do Graylog. A v ka\u017edom d\u00e1tovom centre je svoj \u00falo\u017en\u00fd priestor. V pr\u00edpade, ak strat\u00edme spojenie, m\u00e1me tam v\u0161etky logy. Tie tam zostan\u00fa. Bud\u00fa tam uchovan\u00e9.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pri mimoriadnych situ\u00e1ci\u00e1ch dost\u00e1vate logy odtia\u013e?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: M\u00f4\u017eete tam \u00eds\u0165 (do s\u00faborov\u00e9ho \u00falo\u017eiska) a pozrie\u0165 sa.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Ako monitorujete, \u017ee nestr\u00e1cate logy?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: V skuto\u010dnosti ich str\u00e1came a monitorujeme to. Monitoring sme spustili pred mesiacom. V kni\u017enici, ktor\u00fa pou\u017e\u00edva Go API, s\u00fa metriky. Vie po\u010d\u00edta\u0165, ko\u013ekokr\u00e1t sa jej nepodarilo zap\u00edsa\u0165 do socketu. Moment\u00e1lne je tam \u0161ikovn\u00e1 heuristika. Je tam buffer. Sna\u017e\u00ed sa zapisova\u0165 z neho spr\u00e1vy do socketu. Ak sa buffer napln\u00ed, za\u010dne ich zahadzova\u0165. A po\u010d\u00edta, ko\u013eko ich zahodil. Ak sa za\u010dn\u00fa nap\u013a\u0148a\u0165 po\u010d\u00edtadl\u00e1, dozvieme sa o tom. Moment\u00e1lne prich\u00e1dzaj\u00fa aj do Prometheus, a v Grafane m\u00f4\u017eete vidie\u0165 grafy. Je mo\u017en\u00e9 nastavi\u0165 upozornenia. Ale zatia\u013e nie je jasn\u00e9, komu ich posiela\u0165.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: V Elasticsearch skladujete logy s replikovan\u00edm. Ko\u013eko m\u00e1te replik?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Jedna replik\u00e1cia.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: To je len jedna replik\u00e1cia?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: To je master a replik\u00e1cia. D\u00e1ta s\u00fa uchov\u00e1van\u00e9 v dvoch k\u00f3pi\u00e1ch.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Ve\u013ekos\u0165 bufferu rsyslog ste nejak\u00fdm sp\u00f4sobom upravovali? <\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: P\u00ed\u0161eme datagramy do vlastn\u00e9ho unixov\u00e9ho socketu. To n\u00e1m hne\u010f kladie obmedzenie 128 kilobajtov. Nem\u00f4\u017eeme do neho zapisova\u0165 viac. Toto sme zap\u00edsali do normy. Kto chce prech\u00e1dza\u0165 do \u00falo\u017eiska, ten p\u00ed\u0161e 128 kilobajtov. Kni\u017enice navy\u0161e truncuj\u00fa a nastavuj\u00fa flag, \u017ee spr\u00e1va bola orezan\u00e1. V na\u0161ej norme samotnej spr\u00e1vy je \u0161peci\u00e1lne pole, ktor\u00e9 ukazuje, \u010di bola spr\u00e1va orezan\u00e1 pri z\u00e1pise alebo nie. Tak\u017ee m\u00e1me mo\u017enos\u0165 sledova\u0165 aj tento moment.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: P\u00ed\u0161ete po\u0161koden\u00fd JSON? <\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Po\u0161koden\u00fd JSON bude odmietnut\u00fd bu\u010f po\u010das relay, preto\u017ee je bal\u00edk pr\u00edli\u0161 ve\u013ek\u00fd. Alebo bude odmietnut\u00fd Graylogom, preto\u017ee nebude m\u00f4c\u0165 JSON analyzova\u0165. Ale s\u00fa tu nuansy, ktor\u00e9 je potrebn\u00e9 vyrie\u0161i\u0165 a tie s\u00fa v\u00e4\u010d\u0161inou spojen\u00e9 s rsyslogom. U\u017e som tam zadal nieko\u013eko probl\u00e9mov, na ktor\u00fdch je potrebn\u00e9 e\u0161te pracova\u0165.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pre\u010do Kafka? Sk\u00fa\u0161ali ste RabbitMQ? Neprekon\u00e1va Graylog pri tak\u00fdchto z\u00e1\u0165a\u017eiach?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: S Graylogom sa n\u00e1m to nedar\u00ed. A Graylog sa n\u00e1m dar\u00ed. Je s n\u00edm naozaj problematick\u00e9. Je to skuto\u010dne zvl\u00e1\u0161tna vec. A v skuto\u010dnosti nie je potrebn\u00fd. Rad\u0161ej by som p\u00edsal priamo z rsyslogu do Elasticsearch a potom si pozrel Kibanu. Ale mus\u00edm dohodn\u00fa\u0165 ot\u00e1zku s bezpe\u010dnostn\u00fdmi \u013eu\u010fmi. Toto je mo\u017en\u00e1 varianta n\u00e1\u0161ho rozvoja, ke\u010f vyhod\u00edme Graylog a budeme pou\u017e\u00edva\u0165 Kibanu. Pou\u017e\u00edvanie Logstash bude bezpredmetn\u00e9. Preto\u017ee to v\u0161etko m\u00f4\u017eem urobi\u0165 pomocou rsyslogu. A m\u00e1 modul na z\u00e1pis do Elasticsearch. S Graylogom sa pok\u00fa\u0161ame nejako \u017ei\u0165. Trochu sme ho aj vyladili. Ale st\u00e1le je tam priestor na zlep\u0161enie.<\/p>\n<p><\/p>\n<p>Pokia\u013e ide o Kafka. Historicky to tak dopadlo. Ke\u010f som pri\u0161iel, u\u017e tu bola a u\u017e sa do nej zapisovali logy. My sme len spustili n\u00e1\u0161 klaster a presunuli sme tam logy. Spravujeme ho, vieme, ako sa m\u00e1. \u010co sa t\u00fdka RabbitMQ\u2026 nem\u00e1me s RabbitMQ dobr\u00e9 sk\u00fasenosti. Ale s RabbitMQ je to in\u00e9. M\u00e1me ho v produkcii a mali sme s n\u00edm probl\u00e9my. Teraz, pred v\u00fdpredajom, ho upravili a za\u010dal norm\u00e1lne fungova\u0165. Ale predt\u00fdm som nebol pripraven\u00fd ho pusti\u0165 do produkcie. Je tu e\u0161te jeden aspekt. Graylog dok\u00e1\u017ee \u010d\u00edta\u0165 verziu AMQP 0.9, ale rsyslog dok\u00e1\u017ee p\u00edsa\u0165 verziu AMQP 1.0. A neexistuje \u017eiadne rie\u0161enie, ktor\u00e9 by vedelo oboje. Bu\u010f to, alebo to. Preto moment\u00e1lne pou\u017e\u00edvame len Kafka. Ale aj tam s\u00fa nejak\u00e9 nuansy. Preto\u017ee omkafka tej verzie rsyslog, ktor\u00fa pou\u017e\u00edvame, m\u00f4\u017ee strati\u0165 cel\u00fd buffer spr\u00e1v, ktor\u00e9 z\u00edskala z rsyslog. Zatia\u013e sa s t\u00fdm zmierujeme. <\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pou\u017e\u00edvate Kafka len preto, \u017ee u\u017e bola vo va\u0161ich syst\u00e9mom? Na in\u00e9 \u00fa\u010dely sa nepou\u017e\u00edva?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: Kafka, ktor\u00e1 tam bola, sa pou\u017e\u00edva t\u00edmom Data Science. To je \u00faplne samostatn\u00fd projekt, o ktorom, \u017eia\u013e, nem\u00f4\u017eem poveda\u0165 ni\u010d. Neviem o \u0148om. Bola v spr\u00e1ve t\u00edmu Data Science. Ke\u010f za\u010dali zapisova\u0165 logy, rozhodli sa ju pou\u017ei\u0165, aby nemuseli in\u0161talova\u0165 vlastn\u00fa. Teraz sme aktualizovali Graylog a stratili sme kompatibilitu, preto\u017ee tam je star\u00e1 verzia Kafka. Museli sme zavies\u0165 vlastn\u00fa. Z\u00e1rove\u0148 sme sa zbavili t\u00fdchto \u0161tyroch t\u00e9m pre ka\u017ed\u00fd API. Urobili sme jednu \u0161irok\u00fa t\u00e9mu pre v\u0161etky live, jednu \u0161irok\u00fa t\u00e9mu pre v\u0161etky staging a jednoducho v\u0161etko tam posielame. Graylog to v\u0161etko paralelne spracov\u00e1va.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Pre\u010do je potrebn\u00e9 robi\u0165 toto \u010darovanie so socketmi? Sk\u00fa\u0161ali ste pou\u017ei\u0165 log-dr\u00e1ver syslog pre kontejnery?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: V \u010dase, ke\u010f sme sa touto ot\u00e1zkou zaoberali, sme mali s dockerom nap\u00e4t\u00e9 vz\u0165ahy. Bol to docker 1.0 alebo 0.9. Docker bol s\u00e1m o sebe zvl\u00e1\u0161tny. Po druh\u00e9, ak do\u0148 e\u0161te aj logy pch\u00e1me... M\u00e1m neoveren\u00e9 podozrenie, \u017ee v\u0161etky logy prech\u00e1dzaj\u00fa cez neho, cez docker demon. Ak jedno API bl\u00e1zni, tak ostatn\u00e9 API nar\u00e1\u017eaj\u00fa na to, \u017ee nem\u00f4\u017eu posla\u0165 stdout a stderr. Neviem, kam to povedie. M\u00e1m pocit, \u017ee syslog driver dockera nie je na tomto mieste vhodn\u00fd. N\u00e1\u0161 oddelenie funk\u010dn\u00e9ho testovania m\u00e1 svoj vlastn\u00fd Graylog klaster s logmi. Pou\u017e\u00edvaj\u00fa docker log drivery a vraj im to tam funguje celkom dobre. Ale GELF okam\u017eite p\u00ed\u0161u do Graylogu. V tom \u010dase, ke\u010f sme to v\u0161etko pl\u00e1novali, sme potrebovali, aby to jednoducho fungovalo. Mo\u017eno nesk\u00f4r, ke\u010f niekto pr\u00edde a povie, \u017ee to u\u017e dlh\u00fa dobu funguje norm\u00e1lne, to sk\u00fasime.<\/p>\n<p><\/p>\n<p><strong>Ot\u00e1zka<\/strong>: Dod\u00e1vku medzi d\u00e1tov\u00fdmi centrami realizujete cez rsyslog. Pre\u010do nie cez Kafka?<\/p>\n<p><\/p>\n<p><strong>Odpove\u010f<\/strong>: V skuto\u010dnosti rob\u00edme oboje. Z dvoch d\u00f4vodov. Ak je kan\u00e1l \u00faplne zlomen\u00fd, tak n\u00e1m ani skomprimovan\u00e9 logy neprejd\u00fa. A Kafka umo\u017e\u0148uje ich jednoducho strati\u0165 po\u010das procesu. T\u00fdmto sp\u00f4sobom sa zbavujeme zadr\u017eiavania t\u00fdchto logov. Tento sp\u00f4sob priamo pou\u017e\u00edvame na Kafku. Ak m\u00e1me dobr\u00fd kan\u00e1l a chceme ho uvo\u013eni\u0165, pou\u017e\u00edvame rsyslog. Ale v skuto\u010dnosti sa d\u00e1 nastavi\u0165 tak, aby s\u00e1m odmietal to, \u010do nepre\u0161lo. Moment\u00e1lne niekde pou\u017e\u00edvame dod\u00e1vku rsyslog priamo, niekde Kafka.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/450098\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b. \u041d\u0443\u0436\u043d\u043e \u0440\u0435\u0448\u0438\u0442\u044c \u0441\u0440\u0430\u0437\u0443 \u043a\u0443\u0447\u0443 \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432: \u043a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438 \u0438\u0437 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f; \u043a\u0443\u0434\u0430 \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438; \u043a\u0430\u043a \u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u043b\u043e\u0433\u0438 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438; \u043a\u0430\u043a \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u0438 \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438. \u041f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u043d\u044b\u043d\u0435 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24507,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u042e\u0440\u0438\u0439 \u0411\u0443\u0448\u043c\u0435\u043b\u0435\u0432 \u00ab\u041a\u0430\u0440\u0442\u0430 \u0433\u0440\u0430\u0431\u043b\u0435\u0439 \u043d\u0430 \u043f\u043e\u043b\u0435 \u0441\u0431\u043e\u0440\u0430 \u0438 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u043b\u043e\u0433\u043e\u0432\u00bb \u2014 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u0430 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:48:35+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:48:35+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Jurij Bu\u0161melev \u201eMapa hrabl\u00ed na poli zberu a dod\u00e1vky logov\u201c \u2013 prepis predn\u00e1\u0161ky | ProHoster","description":"Logy s\u00fa d\u00f4le\u017eitou s\u00fa\u010das\u0165ou syst\u00e9mu, ktor\u00e1 umo\u017e\u0148uje pochopi\u0165, \u010di funguje (alebo nefunguje) tak, ako sa o\u010dak\u00e1va. V podmienkach mikroslu\u017ebovej architekt\u00fary sa pr\u00e1ca s logmi st\u00e1va samostatnou discipl\u00ednou \u0161peci\u00e1lnej olympi\u00e1dy.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u042e\u0440\u0438\u0439 \u0411\u0443\u0448\u043c\u0435\u043b\u0435\u0432 \u00ab\u041a\u0430\u0440\u0442\u0430 \u0433\u0440\u0430\u0431\u043b\u0435\u0439 \u043d\u0430 \u043f\u043e\u043b\u0435 \u0441\u0431\u043e\u0440\u0430 \u0438 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u043b\u043e\u0433\u043e\u0432\u00bb \u2014 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u0430 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 | ProHoster","og:description":"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/yurij-bushmelev-karta-grablej-na-pole-sbora-i-dostavki-logov-rasshifrovka-doklada","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:48:35+00:00","article:modified_time":"2019-10-31T18:48:35+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"32723","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 12:16:55","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:53:25","updated":"2026-01-21 12:16:55","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/32723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=32723"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/32723\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/24507"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=32723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=32723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=32723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}