artikullin mbi Badushk\u00ebn<\/a><\/noindex>, q\u00eb shkroi p\u00ebr k\u00ebt\u00eb. Por ka nj\u00eb moment. Rreth k\u00ebtij thirrjeje kishte nj\u00eb cik\u00ebl t\u00eb pafund, ku vazhdimisht po p\u00ebrpiqeshim t\u00eb futnim mesazhin n\u00eb soket. K\u00ebt\u00eb nuk e v\u00ebm\u00eb re. Na duhej t\u00eb rishkruanim bibliotek\u00ebn. Q\u00eb at\u00ebher\u00eb ajo \u00ebsht\u00eb ndryshuar disa her\u00eb, por tani ne e kemi eliminuar bllokimin n\u00eb t\u00eb gjitha n\u00ebn sistemet. Prandaj, mund t\u00eb ndalojm\u00eb rsyslog-un dhe asgj\u00eb nuk do t\u00eb d\u00ebshtoj\u00eb.<\/p>\n<\/p>\n
Duhet t\u00eb monitorojm\u00eb madh\u00ebsin\u00eb e radh\u00ebve, q\u00eb ndihmon p\u00ebr t\u00eb mos shkelur n\u00eb k\u00ebto gracka. S\u00eb pari, ne mund t\u00eb monitorojm\u00eb kur fillojm\u00eb t\u00eb humbasim mesazhe. S\u00eb dyti, mund t\u00eb monitorojm\u00eb n\u00ebse kemi probleme me d\u00ebrgesat n\u00eb p\u00ebrgjith\u00ebsi.<\/p>\n
<\/p>\n
Dhe nj\u00eb moment tjet\u00ebr i pak\u00ebndsh\u00ebm \u2014 amplifikimi i 10-fish n\u00eb arkitektur\u00ebn mikro-sh\u00ebrbimore \u00ebsht\u00eb shum\u00eb i leht\u00eb. Ne nuk kemi aq shum\u00eb k\u00ebrkesa hyr\u00ebse, por p\u00ebr shkak t\u00eb grafit, p\u00ebrmes t\u00eb cilit kalojn\u00eb k\u00ebto mesazhe m\u00eb tej, p\u00ebr shkak t\u00eb logjeve t\u00eb aksesit, ne realisht rrisim ngarkes\u00ebn e logeve p\u00ebraf\u00ebrsisht 10 her\u00eb. Fatkeq\u00ebsisht nuk e kam pasur koh\u00eb t\u00eb llogaris numrat e sakt\u00eb, por mikro-sh\u00ebrbimet \u2014 jan\u00eb t\u00eb till\u00eb. Duhet ta kemi k\u00ebt\u00eb parasysh. Pra, p\u00ebr momentin, n\u00ebn sistemi i mbledhjes s\u00eb logeve \u00ebsht\u00eb m\u00eb i ngarkuari n\u00eb Lazada. <\/p>\n
<\/p>\n
<\/p>\n
Si duhet t\u00eb zgjidhni problemin e elasticsearch? N\u00ebse keni nevoj\u00eb p\u00ebr t\u00eb marr\u00eb shpejt log-et n\u00eb nj\u00eb vend, q\u00eb t\u00eb mos b\u00ebni sh\u00ebtitje n\u00ebp\u00ebr t\u00eb gjitha makinat dhe t\u00eb mos i mbledhni atje, p\u00ebrdorni nj\u00eb depo t\u00eb skedar\u00ebve. Kjo funksionon me siguri. Mund ta b\u00ebni nga \u00e7do server. Thjesht vendosni disqet atje dhe instaloni syslog. Pas k\u00ebsaj, do t\u00eb keni me siguri t\u00eb gjitha log-et n\u00eb nj\u00eb vend. M\u00eb pas, do t\u00eb jeni n\u00eb gjendje t\u00eb konfiguroheni me qet\u00ebsi p\u00ebr elasticsearch, graylog, di\u00e7ka tjet\u00ebr. Por do t\u00eb keni tashm\u00eb t\u00eb gjitha log-et, dhe madje, mund t\u2019i ruani sa mjafton kapaciteti i disqeve.<\/p>\n
<\/p>\n
<\/p>\n
N\u00eb momentin q\u00eb po flisja, skema kishte marr\u00eb pamjen e k\u00ebsaj. Ne praktikisht kemi ndaluar shkrimin n\u00eb skedar\u00eb. Tani, p\u00ebr momentin, do t\u00eb ndalojm\u00eb dhe mbeturinat. N\u00eb makinat lokale, ku jan\u00eb aktivizuar API-t\u00eb, ne do t\u00eb ndalojm\u00eb shkrimin n\u00eb skedar\u00eb. S\u00eb pari, ekziston nj\u00eb depo skedar\u00ebsh q\u00eb funksionon shum\u00eb mir\u00eb. S\u00eb dyti, n\u00eb k\u00ebto makina vazhdimisht po i mbaron hap\u00ebsira, dhe duhet ta monitorojm\u00eb vazhdimisht.<\/p>\n
<\/p>\n
Kjo pjes\u00eb me Logstash dhe Graylog, v\u00ebrtet \u00ebsht\u00eb problematike. Pra, duhet ta heqim at\u00eb. Duhet t\u00eb zgjedhim vet\u00ebm nj\u00eb.<\/p>\n
<\/p>\n
<\/p>\n
Ne vendos\u00ebm t\u00eb heqim Logstash dhe Kibana. Kjo ndodhi sepse kemi nj\u00eb departament sigurie. Cila \u00ebsht\u00eb lidhja? Lidhja \u00ebsht\u00eb se Kibana pa X-Pack dhe pa Shield nuk lejon ndarjen e t\u00eb drejtave t\u00eb aksesit n\u00eb log-e. Prandaj zgjodh\u00ebm Graylog. Ai nuk m\u00eb p\u00eblqen, por funksionon. Blejm\u00eb harduer t\u00eb ri, vendosim aty nj\u00eb Graylog t\u00eb ri dhe transferojm\u00eb t\u00eb gjitha log-et me formate strikte n\u00eb nj\u00eb Graylog t\u00eb ve\u00e7ant\u00eb. Zgjidh\u00ebm problemin me lloje t\u00eb ndryshme t\u00eb t\u00eb nj\u00ebjtave fusha organizativisht.<\/p>\n
<\/p>\n
<\/p>\n
\u00c7far\u00eb p\u00ebrfshin n\u00eb t\u00eb v\u00ebrtet\u00eb Graylog i ri. Thjesht e regjistruam gjith\u00e7ka n\u00eb Docker. Mor\u00ebm shum\u00eb servera, vendos\u00ebm tre instance t\u00eb Kafka dhe 7 servera Graylog versioni 2.3 (sepse donim Elasticsearch versionin 5). Gjith\u00e7ka u ngrit n\u00eb RAID nga HDD. V\u00ebrejt\u00ebm nj\u00eb indeksim deri n\u00eb 100 mij\u00eb mesazhe n\u00eb sekond\u00eb. V\u00ebrejt\u00ebm nj\u00eb shif\u00ebr se 140 terabajt t\u00eb dh\u00ebnash n\u00eb jav\u00eb. <\/p>\n
<\/p>\n
<\/p>\n
Dhe p\u00ebrs\u00ebri gabime! Na presin dy shfaqje. U transferuam me 6 milion mesazhe. Graylog-i yn\u00eb nuk po arrin t\u00eb p\u00ebrpunoj\u00eb. Duhet t\u00eb gjejm\u00eb p\u00ebrs\u00ebri nj\u00eb m\u00ebnyr\u00eb p\u00ebr t\u00eb mbijetuar. <\/p>\n
<\/p>\n
<\/p>\n
Ne kemi mbijetuar k\u00ebshtu. Shtuam edhe disa server\u00eb dhe SSD. Aktualisht jetojm\u00eb n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb. Tani jemi duke procesuar 160,000 mesazhe n\u00eb sekond\u00eb. Ende nuk kemi arritur kufirin, prandaj nuk dihet se sa do t\u00eb mund t\u00eb nxjerrim nga kjo. <\/p>\n
<\/p>\n
<\/p>\n
K\u00ebto jan\u00eb planet tona p\u00ebr t\u00eb ardhmen. Nga to, ndoshta m\u00eb e r\u00ebnd\u00ebsishmja \u00ebsht\u00eb disponueshm\u00ebria e lart\u00eb. Aktualisht nuk e kemi at\u00eb. Disa makina jan\u00eb t\u00eb konfiguruara nj\u00ebsoj, por gjith\u00e7ka po kalon p\u00ebrmes nj\u00eb makine. Duhet t\u00eb kalojm\u00eb koh\u00eb p\u00ebr t\u00eb konfiguruar failover nd\u00ebrmjet tyre.<\/p>\n
<\/p>\n
T\u00eb mbledhim metrike nga Graylog.<\/p>\n
<\/p>\n
T\u00eb b\u00ebjm\u00eb rate limit q\u00eb nj\u00eb API e \u00e7mendur t\u00eb mos na shkat\u00ebrroj\u00eb bandwidth-in dhe gjith\u00e7ka tjet\u00ebr.<\/p>\n
<\/p>\n
Dhe p\u00ebrfundimisht, t\u00eb n\u00ebnshkruajm\u00eb nj\u00eb SLA me zhvilluesit, q\u00eb ne mund t\u00eb sh\u00ebrbejm\u00eb deri n\u00eb k\u00ebt\u00eb mas\u00eb. N\u00ebse shkruani m\u00eb shum\u00eb, at\u00ebher\u00eb na vjen keq.<\/p>\n
<\/p>\n
Dhe t\u00eb shkruajm\u00eb dokumentacion.<\/p>\n
<\/p>\n
<\/p>\n
P\u00ebrmbledhje e shkurt\u00ebr e gjith\u00e7kaje q\u00eb kemi p\u00ebrjetuar. S\u00eb pari, standardet. S\u00eb dyti, syslog \u2014 tort\u00eb. S\u00eb treti, rsyslog funksionon pik\u00ebrisht ashtu si\u00e7 \u00ebsht\u00eb shkruar n\u00eb slajd. Tani le t\u00eb kalojm\u00eb te pyetjet.<\/p>\n
<\/p>\n
Pyetje<\/strong>.<\/p>\n<\/p>\n
Pyetja<\/strong>: Pse vendos\u00ebt t\u00eb mos merrni\u2026 (filebeat?)<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Duhet t\u00eb shkruajm\u00eb n\u00eb skedar. S'm\u00eb p\u00eblqeu aspak. Kur API juaj shkruan mij\u00ebra mesazhe n\u00eb sekond\u00eb, madje edhe n\u00ebse e rotuloni nj\u00eb her\u00eb n\u00eb or\u00eb, kjo s'\u00ebsht\u00eb akoma nj\u00eb zgjidhje. Mund t\u00eb shkruhet n\u00eb pipe. N\u00eb \u00e7far\u00ebdo q\u00eb m\u00eb pyet\u00ebn zhvilluesit: \u00ab\u00c7far\u00eb do ndodh\u00eb n\u00ebse procesi n\u00eb t\u00eb cilin shkruajm\u00eb bie?\u00bb Un\u00eb thjesht nuk gjeta \u00e7far\u00eb t'u p\u00ebrgjigjem dhe thash\u00eb: \u00abS'nuk do ta b\u00ebjm\u00eb ashtu\u00bb.<\/p>\n<\/p>\n
Pyetja<\/strong>: Pse nuk shkruani log-\u00ebt thjesht n\u00eb HDFS?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Ky \u00ebsht\u00eb hapi tjet\u00ebr. Ne e menduam k\u00ebt\u00eb n\u00eb fillim, por, tani p\u00ebr tani, p\u00ebr shkak t\u00eb munges\u00ebs s\u00eb burimeve p\u00ebr ta trajtuar, ai mbetet si zgjidhje afatgjat\u00eb.<\/p>\n<\/p>\n
Pyetja<\/strong>: Formati kolonor do t\u00eb ishte m\u00eb i p\u00ebrshtatsh\u00ebm.<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: \u042f \u0432\u0441\u0435 \u043f\u043e\u043d\u0438\u043c\u0430\u044e. \u041c\u044b «\u0437\u0430» \u043e\u0431\u0435\u0438\u043c\u0438 \u0440\u0443\u043a\u0430\u043c\u0438. <\/p>\n<\/p>\n
Pyetja<\/strong>: Ju shkruani n\u00eb rsyslog. Aty mund t\u00eb keni edhe TCP, edhe UDP. Por n\u00ebse p\u00ebrdorni UDP, si e garantoni dor\u00ebzimin?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Kant\u00f6het dy aspekte. E para, un\u00eb gjithmon\u00eb iu them t\u00eb gjith\u00ebve se ne nuk garantojm\u00eb d\u00ebrgimin e logeve. Sepse, kur zhvilluesit vijn\u00eb dhe thon\u00eb: \u201cPse nuk fillojm\u00eb t\u00eb shkruajm\u00eb t\u00eb dh\u00ebnat financiare atje, dhe ju do t'i ruani ato diku n\u00eb rast se ndodhin ndodhi,\u201d ne iu p\u00ebrgjigjemi: \u201cShk\u00eblqyesh\u00ebm! Le t\u00eb filloni t\u00eb bllokoni regjistrimin n\u00eb socket dhe ta b\u00ebni k\u00ebt\u00eb n\u00eb transaksione, p\u00ebr t\u00eb garantuar q\u00eb na e d\u00ebrgoni at\u00eb n\u00eb socket dhe t\u00eb siguroheni se ne e mor\u00ebm nga ana tjet\u00ebr.\u201d Dhe n\u00eb at\u00eb moment, t\u00eb gjith\u00eb e kuptojn\u00eb se nuk \u00ebsht\u00eb e nevojshme. Tani, n\u00ebse nuk \u00ebsht\u00eb e nevojshme, \u00e7far\u00eb pyetjesh kemi p\u00ebr ju? N\u00ebse nuk d\u00ebshironi t\u00eb garantoni regjistrimin n\u00eb socket, pse duhet t\u00eb garantojm\u00eb d\u00ebrgimin? Ne b\u00ebjm\u00eb p\u00ebrpjekje maksimale. Ne v\u00ebrtet p\u00ebrpiqemi t\u00eb d\u00ebrgojm\u00eb sa m\u00eb shum\u00eb dhe sa m\u00eb mir\u00eb, por ne nuk japim garanci 100%. K\u00ebshtu, mos shkruani atje t\u00eb dh\u00ebnat financiare. P\u00ebr k\u00ebt\u00eb ekzistojn\u00eb bazat e t\u00eb dh\u00ebnave me transaksione.<\/p>\n<\/p>\n
Pyetja<\/strong>: Kur API gjeneron nj\u00eb mesazh n\u00eb log dhe i kalon menaxhimin mikrosh\u00ebrbimeve, a keni hasur ndonj\u00ebher\u00eb me problemin q\u00eb mesazhet nga mikrosh\u00ebrbime t\u00eb ndryshme vijn\u00eb n\u00eb rend t\u00eb gabuar? Kjo shkakton konfuzion.<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: \u00cbsht\u00eb normale q\u00eb ato t\u00eb vijn\u00eb n\u00eb rend t\u00eb ndrysh\u00ebm. Duhet t\u00eb jemi t\u00eb gatsh\u00ebm p\u00ebr k\u00ebt\u00eb. Sepse \u00e7do d\u00ebrges\u00eb n\u00eb rrjet nuk garanton rendin, ose duhet t\u00eb shpenzoni burime specifike p\u00ebr k\u00ebt\u00eb. N\u00ebse marrim depozit\u00eb skedar\u00ebsh, \u00e7do API ruan logjet n\u00eb skedarin e saj. N\u00eb t\u00eb v\u00ebrtet\u00eb, atje rsyslog i rendit ato n\u00eb kataloge. \u00c7do API aty ka logjet e saj, ku mund t\u00eb shkoni dhe t\u00eb shihni, dhe m\u00eb pas sipas timestamp n\u00eb k\u00ebt\u00eb log mund t'i p\u00ebrputhni ato. N\u00ebse shkojn\u00eb t\u00eb shikojn\u00eb n\u00eb Graylog, atje ato do t\u00eb renditen sipas timestamp. Atje gjith\u00e7ka do t\u00eb jet\u00eb mir\u00eb.<\/p>\n<\/p>\n
Pyetja<\/strong>: Timestamp mund t\u00eb ndryshoj\u00eb n\u00eb milisekonda.<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Timestamp i gjeneron vet\u00eb API. Kjo, n\u00eb thelb, \u00ebsht\u00eb gjith\u00eb shaka. Ne kemi NTP. API gjeneron timestamp tashm\u00eb n\u00eb mesazhin e vet. Ai nuk shtohet nga rsyslog.<\/p>\n<\/p>\n
Pyetja<\/strong>: Nuk \u00ebsht\u00eb shum\u00eb e qart\u00eb nd\u00ebrveprimi midis qendrave t\u00eb t\u00eb dh\u00ebnave. Brenda qendr\u00ebs s\u00eb t\u00eb dh\u00ebnave \u00ebsht\u00eb e qart\u00eb si u grumbulluan dhe u p\u00ebrpunuan logjet. Si ndodh nd\u00ebrveprimi midis qendrave t\u00eb t\u00eb dh\u00ebnave? Ose \u00e7do qend\u00ebr t\u00eb dh\u00ebnash jeton jet\u00ebn e saj?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Gati. \u00c7do vend ndodhet n\u00eb nj\u00eb data qend\u00ebr t\u00eb caktuar. Aktualisht nuk kemi nj\u00eb shp\u00ebrndarje, q\u00eb do t\u00eb thot\u00eb se nj\u00eb vend \u00ebsht\u00eb i vendosur n\u00eb data qendra t\u00eb ndryshme. Pra, nuk duhet t'i gruponi. Brenda \u00e7do qendre ka Log Relay. Ky \u00ebsht\u00eb nj\u00eb server Rsyslog. N\u00eb fakt, ka dy makina menaxher. Ato jan\u00eb t\u00eb konfigurara nj\u00ebsoj. Por aktualisht, vet\u00ebm nj\u00ebra prej tyre e merr trafikun. Ajo agregon t\u00eb gjitha log-et. Ka nj\u00eb rresht diskash p\u00ebr \u00e7do rast. Ajo kompreson log-et dhe i d\u00ebrgon n\u00eb data qendr\u00ebn qendrore (n\u00eb Singapor), ku m\u00eb pas ato d\u00ebrgohen n\u00eb Graylog. \u00c7do data qend\u00ebr ka nj\u00eb storage t\u00eb saj p\u00ebr skedar\u00eb. N\u00eb rast se humbasim lidhjen, ne i kemi t\u00eb gjitha log-et atje. Ato do t\u00eb mbeten aty. Ato do t\u00eb ruhen.<\/p>\n<\/p>\n
Pyetja<\/strong>: N\u00eb situata t\u00eb paplanifikuara, a merrni log-et nga atje?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: A mund t\u00eb shkoni atje (n\u00eb storage-in p\u00ebr skedar\u00eb) dhe t\u00eb shikoni?<\/p>\n<\/p>\n
Pyetja<\/strong>: Si e monitoroni q\u00eb mos humbni log-et?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: N\u00eb t\u00eb v\u00ebrtet\u00eb i humbasim ata, dhe ne e monitorojm\u00eb k\u00ebt\u00eb. Monitorimi filloi nj\u00eb muaj m\u00eb par\u00eb. N\u00eb bibliotek\u00ebn q\u00eb p\u00ebrdor API-n\u00eb Go, ka metrika. Ajo di t\u00eb num\u00ebroj\u00eb sa her\u00eb nuk ka mundur t\u00eb shkruaj\u00eb n\u00eb socket. Atje aktualisht ka nj\u00eb heuristik\u00eb t\u00eb zgjuar. Ka nj\u00eb bufer. Ai p\u00ebrpiqet t\u00eb d\u00ebrgoj\u00eb mesazhin nga ai n\u00eb socket. N\u00ebse buferi mbushet, fillon t'i braktis\u00eb. Dhe num\u00ebron sa i ka braktisur. N\u00ebse fillojn\u00eb t\u00eb mbushen num\u00ebruesit aty, ne do ta dim\u00eb k\u00ebt\u00eb. Tani ata mb\u00ebrrijn\u00eb gjithashtu n\u00eb prometheus, dhe n\u00eb Grafana mund t\u00eb shihni grafikat. Mund t\u00eb konfigurohen njoftimet. Por tani nuk \u00ebsht\u00eb e qart\u00eb se kujt t'i d\u00ebrgohen ato.<\/p>\n<\/p>\n
Pyetja<\/strong>: N\u00eb elasticsearch ruani log-et me rezervim. Sa replikat keni?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Nj\u00eb replik\u00eb.<\/p>\n<\/p>\n
Pyetja<\/strong>: A \u00ebsht\u00eb k\u00ebt\u00eb vet\u00ebm nj\u00eb replik\u00eb?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Ky \u00ebsht\u00eb master dhe nj\u00eb replik\u00eb. T\u00eb dh\u00ebnat ruhet n\u00eb dy ekzemplar\u00eb.<\/p>\n<\/p>\n
Pyetja<\/strong>: A e keni rregulluar ndonj\u00ebher\u00eb madh\u00ebsin\u00eb e buferit rsyslog? <\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Ne shkruajm\u00eb datagramet n\u00eb nj\u00eb socket t\u00eb personalizuar UNIX. Kjo na vendos menj\u00ebher\u00eb nj\u00eb kufizim prej 128 kilobajt\u00ebsh. Nuk mund t\u00eb shkruajm\u00eb m\u00eb shum\u00eb se kaq. E kemi p\u00ebrfshir\u00eb k\u00ebt\u00eb n\u00eb standard. Ata q\u00eb duan t\u00eb hyjn\u00eb n\u00eb magazinat, shkruajn\u00eb 128 kilobajt\u00eb. Bibliotekat, p\u00ebr m\u00eb tep\u00ebr, e presin, duke vendosur nj\u00eb flamur q\u00eb mesazhi \u00ebsht\u00eb prishur. N\u00eb standardin e vet t\u00eb mesazhit, ka nj\u00eb fush\u00eb speciale q\u00eb tregon n\u00ebse \u00ebsht\u00eb prishur gjat\u00eb shkruarjes apo jo. Pra, kemi mund\u00ebsin\u00eb t\u00eb ndjekim k\u00ebt\u00eb moment.<\/p>\n<\/p>\n
Pyetje<\/strong>: A shkruani JSON t\u00eb prishur? <\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: JSON i prishur do t\u00eb hidhet ose gjat\u00eb relay-t, sepse paketa \u00ebsht\u00eb shum\u00eb e madhe. Ose do t\u00eb hidhet nga Graylog, sepse nuk mund t\u00eb parse-oj\u00eb JSON-in. Por ka nuansa q\u00eb duhet t\u00eb rregullohen, dhe ato n\u00eb shumic\u00ebn e rasteve lidhen me rsyslog-un. Kam plot\u00ebsuar disa \u00e7\u00ebshtje aty q\u00eb duhet t\u00eb punojm\u00eb ende mbi to.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse Kafka? Keni provuar RabbitMQ? Nuk funksionon Graylog n\u00ebn k\u00ebto ngarkesa?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Ne kemi probleme me Graylog. Por Graylog \u00ebsht\u00eb nj\u00eb ndihm\u00eb. Ka v\u00ebrtet\u00eb probleme. \u00cbsht\u00eb nj\u00eb gj\u00eb e \u00e7uditshme. N\u00eb t\u00eb v\u00ebrtet\u00eb, nuk \u00ebsht\u00eb e nevojshme. Un\u00eb do t\u00eb preferoja t\u00eb shkruaja direkt nga rsyslog n\u00eb elasticsearch dhe m\u00eb pas t\u00eb shihja n\u00eb Kibana. Por duhet t\u00eb zgjidhim \u00e7\u00ebshtjen me sigurin\u00eb. Ky \u00ebsht\u00eb nj\u00eb opsion i mundsh\u00ebm p\u00ebr zhvillimin ton\u00eb, kur t\u00eb heqim Graylog dhe t\u00eb p\u00ebrdorim Kibana. Nuk do kishte asnj\u00eb kuptim t\u00eb p\u00ebrdorim Logstash. Sepse, mund t\u00eb b\u00ebj t\u00eb nj\u00ebjt\u00ebn gj\u00eb me rsyslog. Ai ka nj\u00eb modul p\u00ebr t\u00eb shkruar n\u00eb elasticsearch. Me Graylog po p\u00ebrpiqemi t\u00eb jetojm\u00eb. Madje e kemi b\u00ebr\u00eb edhe disa rregullime. Por ka ende hap\u00ebsir\u00eb p\u00ebr p\u00ebrmir\u00ebsime.<\/p>\n<\/p>\n
N\u00eb lidhje me Kafka. K\u00ebshtu ka ndodhur historikisht. Kur kam ardhur, ajo ishte tashm\u00eb aty dhe n\u00eb t\u00eb tashm\u00eb ishim duke shkruar loge. Ne thjesht ngrit\u00ebm klasterin ton\u00eb dhe u transferuam me loget. Ne e menaxhojm\u00eb at\u00eb, dim\u00eb si ndjehet. N\u00eb lidhje me RabbitMQ\u2026 nuk po na del mir\u00eb me RabbitMQ. Po na del mir\u00eb me RabbitMQ. E kemi n\u00eb prodhim, dhe kemi pasur probleme me t\u00eb. Tani, p\u00ebrpara shitjes, e pat\u00ebm p\u00ebrmir\u00ebsuar at\u00eb, dhe filloi t\u00eb funksiononte normalisht. Por deri at\u00ebher\u00eb nuk isha i gatsh\u00ebm ta nxirrja n\u00eb prodhim. Ka edhe nj\u00eb aspekt tjet\u00ebr. Graylog di t\u00eb lexoj\u00eb versionin AMQP 0.9, nd\u00ebrsa rsyslog di t\u00eb shkruaj\u00eb versionin AMQP 1.0. Dhe nuk ka asnj\u00eb zgjidhje q\u00eb mund t\u00eb b\u00ebj\u00eb t\u00eb dua. Ka ose nj\u00ebra ose tjetra. Prandaj p\u00ebr momentin vet\u00ebm Kafka. Por ka edhe nuanca t\u00eb veta. Sepse omkafka i atij versioni t\u00eb rsyslog, q\u00eb ne po p\u00ebrdorim, mund t\u00eb humbas\u00eb t\u00eb gjith\u00eb bufferin e mesazheve q\u00eb ka marr\u00eb nga rsyslog. Deri tani po pajtohemi me k\u00ebt\u00eb. <\/p>\n
<\/p>\n
Pyetje<\/strong>: A po p\u00ebrdorni Kafka sepse ajo ishte e juaja? Nuk p\u00ebrdoret p\u00ebr asnj\u00eb q\u00ebllim tjet\u00ebr?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Kafka, e cila \u00ebsht\u00eb p\u00ebrdorur nga ekipi i Shkenc\u00ebs s\u00eb t\u00eb Dh\u00ebnave. Ky \u00ebsht\u00eb nj\u00eb projekt krejt\u00ebsisht ndryshe, p\u00ebr t\u00eb cilin, fatkeq\u00ebsisht, nuk mund t\u00eb them asgj\u00eb. Nuk jam n\u00eb dijeni. Ajo ishte n\u00ebn menaxhimin e ekipit t\u00eb Shkenc\u00ebs s\u00eb t\u00eb Dh\u00ebnave. Kur nis\u00ebm log-et, vendos\u00ebm ta p\u00ebrdorim at\u00eb p\u00ebr t\u00eb mos instaluar edhe nj\u00eb tonit ton\u00ebn. Tani e kemi p\u00ebrdit\u00ebsuar Graylog, dhe kemi humbur kompatibilitetin, sepse ka nj\u00eb version t\u00eb vjet\u00ebr t\u00eb Kafka. Na duhej t\u00eb krijonim ton\u00ebn. Nj\u00ebher\u00ebsh, na u hoq\u00ebn k\u00ebto kat\u00ebr tema p\u00ebr \u00e7do API. Krijuam nj\u00eb tem\u00eb t\u00eb gjer\u00eb p\u00ebr t\u00eb gjith\u00eb live-in, nj\u00eb tem\u00eb t\u00eb gjer\u00eb p\u00ebr t\u00eb gjith\u00eb staging-un dhe thjesht e d\u00ebrgojm\u00eb gjith\u00e7ka atje. Graylog e mbledh gjith\u00e7ka paralelisht.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse duhet ky magji me soket\u00ebt? A keni provuar t\u00eb p\u00ebrdorni log-drejtorin syslog p\u00ebr kontejner\u00ebt.<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: N\u00eb momentin kur ne po merreshim me k\u00ebt\u00eb \u00e7\u00ebshtje, marr\u00ebdh\u00ebniet tona me Docker ishin tendosura. Ishte Docker 1.0 ose 0.9. Docker vet\u00eb ishte i \u00e7uditsh\u00ebm. P\u00ebr m\u00eb tep\u00ebr, n\u00ebse duhej t\u00eb d\u00ebrgonim log-et\u2026 Kam nj\u00eb dyshim t\u00eb pa verifikuar se ai kalon t\u00eb gjitha log-et p\u00ebrmes tij, p\u00ebrmes demonit Docker. N\u00ebse nj\u00eb API \u00e7mendet, at\u00ebher\u00eb API-t\u00eb e tjera ngjiten n\u00eb faktin se nuk mund t\u00eb d\u00ebrgojn\u00eb stdout dhe stderr. Nuk e di se \u00e7far\u00eb do t\u00eb sjell\u00eb kjo. Kam nj\u00eb ndjenj\u00eb q\u00eb nuk duhet ta p\u00ebrdorim driver-in syslog t\u00eb Docker k\u00ebtu. Departamenti yn\u00eb p\u00ebr testimin funksional ka grupin e vet t\u00eb vog\u00ebl Graylog me log-e. Ata p\u00ebrdorin driver-at e log-eve t\u00eb Docker dhe duket se gjith\u00e7ka funksionon mir\u00eb atje. Por ata e shkruajn\u00eb GELF direkt n\u00eb Graylog. Kur ne po fillonim gjith\u00e7ka, na duhej t\u00eb funksiononte vet\u00ebm. Ndoshta m\u00eb von\u00eb, kur dikush t\u00eb vij\u00eb dhe t\u00eb thot\u00eb se kjo tashm\u00eb ka funksionuar normalisht p\u00ebr nj\u00ebqind vjet, do ta provojm\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse d\u00ebrgoni mes datacentrove me rsyslog? Pse jo me Kafka?<\/p>\n<\/p>\n
P\u00ebrgjigja<\/strong>: Ne e b\u00ebjm\u00eb k\u00ebt\u00eb dhe at\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb. P\u00ebr dy arsye. N\u00ebse kanali \u00ebsht\u00eb krejt\u00ebsisht i d\u00ebmtuar, at\u00ebher\u00eb as log-et tona, madje edhe n\u00eb form\u00eb t\u00eb kompresuar, nuk kalojn\u00eb p\u00ebrmes tij. Por Kafka lejon q\u00eb ata thjesht t\u00eb humbasin gjat\u00eb procesit. Me k\u00ebt\u00eb metod\u00eb ne lirohemi nga ngjizja e k\u00ebtyre log-eve. Ne thjesht e p\u00ebrdorim Kafka drejtp\u00ebrdrejt n\u00eb k\u00ebt\u00eb rast. N\u00ebse kemi nj\u00eb kanal t\u00eb mir\u00eb dhe duam ta lirojm\u00eb at\u00eb, at\u00ebher\u00eb p\u00ebrdorim rsyslog-in e tyre. Por n\u00eb t\u00eb v\u00ebrtet\u00eb e mund ta konfigurojm\u00eb n\u00eb m\u00ebnyr\u00eb q\u00eb ai t\u00eb heq\u00eb automatikisht at\u00eb q\u00eb nuk kalon. N\u00eb k\u00ebt\u00eb moment, ne thjesht p\u00ebrdorim d\u00ebrgimin rsyslog drejtp\u00ebrdrejt ndonj\u00ebher\u00eb, ndonj\u00ebher\u00eb Kafka.<\/p>\nBurimi: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b. \u041d\u0443\u0436\u043d\u043e \u0440\u0435\u0448\u0438\u0442\u044c \u0441\u0440\u0430\u0437\u0443 \u043a\u0443\u0447\u0443 \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432: \u043a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438 \u0438\u0437 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f; \u043a\u0443\u0434\u0430 \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438; \u043a\u0430\u043a \u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u043b\u043e\u0433\u0438 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438; \u043a\u0430\u043a \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u0438 \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438. \u041f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u043d\u044b\u043d\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24507,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"Luaj](\"https:\/\/img.youtube.com\/vi\/NAeedJv-S3I\/hqdefault.jpg\")
\r\n