artikullin p\u00ebr Badushk\u00ebn<\/a><\/noindex>, e cila ka e shkruajti p\u00ebr k\u00ebt\u00eb. Por ka nj\u00eb moment. Rreth k\u00ebtij thirrje kishte edhe nj\u00eb cik\u00ebl t\u00eb pafund, n\u00eb t\u00eb cilin b\u00ebhej vazhdimisht nj\u00eb p\u00ebrpjekje p\u00ebr ta futur mesazhin n\u00eb soket. K\u00ebt\u00eb nuk e v\u00ebm\u00eb re. Kemi qen\u00eb t\u00eb detyruar ta rreshim librarin\u00eb. Q\u00eb at\u00ebher\u00eb ajo \u00ebsht\u00eb nd\u00ebrruar disa her\u00eb, por tani ne e kemi eliminuar bllokimin n\u00eb t\u00eb gjitha n\u00ebn sistemet. Prandaj, mund t\u00eb ndalojm\u00eb rsyslog-un, dhe asgj\u00eb nuk do t\u00eb bie.<\/p>\n<\/p>\n
Duhet t\u00eb monitorojm\u00eb madh\u00ebsin\u00eb e radh\u00ebve, q\u00eb ndihmon q\u00eb t\u00eb mos bien n\u00eb k\u00ebto gracka. N\u00eb radh\u00eb t\u00eb par\u00eb, mund t\u00eb monitorojm\u00eb kur fillojm\u00eb t\u00eb humbasim mesazhe. N\u00eb radh\u00eb t\u00eb dyt\u00eb, mund t\u00eb monitorojm\u00eb problemet q\u00eb kemi n\u00eb t\u00ebr\u00ebsi me dor\u00ebzimin.<\/p>\n
<\/p>\n
Dhe nj\u00eb moment tjet\u00ebr i pak\u00ebndsh\u00ebm \u2014 amplifikimi 10-fish n\u00eb arkitektur\u00ebn e mikros\u00ebrvitor\u00ebve \u2014 \u00ebsht\u00eb shum\u00eb e leht\u00eb. Nuk kemi shum\u00eb k\u00ebrkesa hyr\u00ebse, por p\u00ebr shkak t\u00eb grafit n\u00eb t\u00eb cilin qarkullojn\u00eb k\u00ebto mesazhe m\u00eb tej, p\u00ebr shkak t\u00eb log-eve t\u00eb aksesit, ne realisht rrisim ngarkes\u00ebn p\u00ebr log-et rreth dhjet\u00eb her\u00eb. Fatkeq\u00ebsisht, nuk kam arritur t\u00eb llogaris numrat e sakt\u00eb, por mikroservisat \u2014 ato jan\u00eb t\u00eb tilla. Duhet ta kemi parasysh k\u00ebt\u00eb. Pra, n\u00eb k\u00ebt\u00eb moment n\u00ebn sistemi q\u00eb mbledh log-et \u00ebsht\u00eb m\u00eb i ngarkuari n\u00eb Lazada. <\/p>\n
<\/p>\n
<\/p>\n
Si ta zgjidhim problemin me elasticsearch? N\u00ebse duhet t\u00eb marrim shpejt log-et n\u00eb nj\u00eb vend, p\u00ebr t\u00eb mos ecur n\u00ebp\u00ebr t\u00eb gjitha makinat, dhe p\u00ebr t'i mbledhur atje, p\u00ebrdorni nj\u00eb ruajtje skedari. Kjo funksionon garantuar. Kjo krijohet nga \u00e7do server. Duhet vet\u00ebm t\u2019i shtoni disa disqe atje dhe t\u00eb vendosni syslog. Pas k\u00ebsaj do t\u00eb keni garantuar t\u00eb gjitha log-et n\u00eb nj\u00eb vend. M\u00eb pas mund t\u00eb filloni t\u00eb konfiguroni elasticsearch, graylog, di\u00e7ka tjet\u00ebr nga ngadal\u00eb. Por do t'i keni t\u00eb gjitha log-et, dhe p\u00ebr m\u00eb tep\u00ebr, mund t'i ruani, sa mjaftojn\u00eb masat e disqeve.<\/p>\n
<\/p>\n
<\/p>\n
N\u00eb momentin e prezantimit tim, skema dukej k\u00ebshtu. Praktikisht kemi ndaluar t\u00eb shkruajm\u00eb n\u00eb skedar\u00eb. Tani, me t\u00eb gjitha gjasat, do t\u00eb \u00e7aktivizojm\u00eb mbetjet. N\u00eb makinat lokale, n\u00eb t\u00eb cilat jan\u00eb aktivizuar API-t\u00eb, do t\u00eb ndalojm\u00eb t\u00eb shkruajm\u00eb n\u00eb skedar\u00eb. S\u00eb pari, ekziston nj\u00eb ruajtje skedari q\u00eb funksionon shum\u00eb mir\u00eb. S\u00eb dyti, n\u00eb k\u00ebto makina vendoset hap\u00ebsira vazhdimisht, \u00ebsht\u00eb e nevojshme ta monitorojm\u00eb at\u00eb vazhdimisht.<\/p>\n
<\/p>\n
Kjo pjes\u00eb me Logstash dhe Graylog, v\u00ebrtet e ngarkon. Prandaj duhet t\u00eb heqim dor\u00eb prej saj. Duhet t\u00eb zgjedhim nj\u00eb nga ato.<\/p>\n
<\/p>\n
<\/p>\n
Ne vendos\u00ebm t\u00eb heqim Logstash dhe Kibana. Sepse kemi nj\u00eb departament t\u00eb siguris\u00eb. Cila \u00ebsht\u00eb lidhja? Lidhja \u00ebsht\u00eb se Kibana pa X-Pack dhe pa Shield nuk lejon ndarjen e t\u00eb drejtave t\u00eb aksesit n\u00eb loge. Prandaj mor\u00ebm Graylog. Pavar\u00ebsisht se nuk m\u00eb p\u00eblqen, ai funksionon. Kemi bler\u00eb pajisje t\u00eb reja, vendos\u00ebm aty nj\u00eb Graylog t\u00eb ri dhe transferuam t\u00eb gjitha loget me formate t\u00eb rrepta n\u00eb nj\u00eb Graylog t\u00eb ve\u00e7ant\u00eb. Ne zgjidh\u00ebm problemin me lloje t\u00eb ndryshme t\u00eb fushave t\u00eb nj\u00ebjta n\u00eb m\u00ebnyr\u00eb organizative.<\/p>\n
<\/p>\n
<\/p>\n
\u00c7far\u00eb p\u00ebrfshihet n\u00eb Graylog t\u00eb ri? Ne thjesht e regjistruam gjith\u00e7ka n\u00eb Docker. Mor\u00ebm nj\u00eb s\u00ebr\u00eb server\u00ebsh, shp\u00ebrndam\u00eb tre instanca Kafka, 7 server\u00eb Graylog versioni 2.3 (sepse d\u00ebshironim Elasticsearch version 5). T\u00eb gjitha k\u00ebto i ngrit\u00ebm mbi RAID nga HDD. V\u00ebm\u00eb re nj\u00eb norm\u00eb indeksimi deri n\u00eb 100 mij\u00eb mesazhe n\u00eb sekond\u00eb. V\u00ebm\u00eb re numrin se 140 terabajt t\u00eb dh\u00ebnash n\u00eb jav\u00eb. <\/p>\n
<\/p>\n
<\/p>\n
Dhe s\u00ebrish, pengesa! Po na presin dy shitje t\u00eb m\u00ebdha. Ne kaluam p\u00ebr 6 milion mesazhe. Graylog nuk po arrin t\u00eb p\u00ebrpunoj\u00eb. Duhet t\u00eb mbijetojm\u00eb s\u00ebrish. <\/p>\n
<\/p>\n
<\/p>\n
K\u00ebshtu mbijetuam. Shtova pak server\u00eb dhe SSD. Aktualisht po jetojm\u00eb me k\u00ebt\u00eb m\u00ebnyr\u00eb. Tani po p\u00ebrpunojm\u00eb tashm\u00eb 160k mesazhe n\u00eb sekond\u00eb. Ende nuk kemi arritur n\u00eb limit, prandaj p\u00ebr momentin \u00ebsht\u00eb e paqart\u00eb se sa realisht mund t\u00eb nxjerrim nga kjo. <\/p>\n
<\/p>\n
<\/p>\n
Ja si jan\u00eb planet tona p\u00ebr t\u00eb ardhmen. Nga k\u00ebto, ndoshta, m\u00eb e r\u00ebnd\u00ebsishmja \u00ebsht\u00eb high availability. P\u00ebr momentin nuk e kemi. Disa makineri jan\u00eb t\u00eb konfiguruara nj\u00ebsoj, por t\u00eb gjitha kalojn\u00eb p\u00ebrmes nj\u00eb makinerie. Duhet t\u00eb shpenzojm\u00eb koh\u00eb p\u00ebr t\u00eb konfiguruar nj\u00eb failover mes tyre.<\/p>\n
<\/p>\n
T\u00eb mbledhim metrika nga Graylog.<\/p>\n
<\/p>\n
T\u00eb b\u00ebjm\u00eb nj\u00eb limit t\u00eb norm\u00ebs q\u00eb nj\u00eb API e r\u00ebn\u00eb nga dora t\u00eb mos na shkat\u00ebrroj\u00eb bandwidth dhe gjith\u00e7ka tjet\u00ebr.<\/p>\n
<\/p>\n
Dhe p\u00ebrfundimisht, t\u00eb n\u00ebnshkruajm\u00eb nj\u00eb SLA me zhvilluesit, se ne mund t\u00eb ofrojm\u00eb kaq shum\u00eb. N\u00ebse shkruani m\u00eb shum\u00eb, nga na vjen keq.<\/p>\n
<\/p>\n
Dhe t\u00eb shkruajm\u00eb dokumentacion.<\/p>\n
<\/p>\n
<\/p>\n
Shkurtimisht, p\u00ebrmbledhja e t\u00eb gjitha atyre q\u00eb kaluam. S\u00eb pari, standartet. S\u00eb dyti, syslog \u2014 tort\u00eb. S\u00eb treti, rsyslog funksionon pik\u00ebrisht si\u00e7 \u00ebsht\u00eb shkruar n\u00eb slajd. Dhe le t\u00eb kalojm\u00eb te pyetjet.<\/p>\n
<\/p>\n
Pyetje<\/strong>.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse vendos\u00ebt t\u00eb mos marr\u00eb...<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Duhet t\u00eb shkruhet n\u00eb skedarin. Ishte shum\u00eb e pak\u00ebndshme. Kur API-ja juaj shkruan mij\u00ebra mesazhe n\u00eb sekond\u00eb, edhe n\u00ebse rrotulloheni nj\u00eb her\u00eb n\u00eb or\u00eb, kjo nuk \u00ebsht\u00eb nj\u00eb mund\u00ebsi. Mund t\u00eb shkruhet n\u00eb pipe. Ndaj zhvilluesit m\u00eb pyet\u00ebn: \"\u00c7far\u00eb do ndodhte n\u00ebse procesi n\u00eb t\u00eb cilin shkruajm\u00eb bie\"? Un\u00eb thjesht nuk e gjeta dot nj\u00eb p\u00ebrgjigje, dhe thash\u00eb: \"E drejt\u00eb, le t\u00eb mos e b\u00ebjm\u00eb ashtu.\"<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse nuk i shkruani thjesht loget n\u00eb HDFS?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Ky \u00ebsht\u00eb hapi tjet\u00ebr. Kemi menduar p\u00ebr t\u00eb n\u00eb fillim, por, p\u00ebr shkak se aktualisht nuk kemi burime p\u00ebr t\u00eb punuar mbi k\u00ebt\u00eb, ai mbetet si nj\u00eb zgjidhje afatgjat\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: Formati kolumnar do t\u00eb ishte m\u00eb i p\u00ebrshtatsh\u00ebm.<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: E kuptoj t\u00eb gjith\u00eb. Ne jemi \"p\u00ebr\" me t\u00eb dyja duar. <\/p>\n<\/p>\n
Pyetje<\/strong>: Ju shkruani n\u00eb rsyslog. Aty mund t\u00eb p\u00ebrdorni si TCP, ashtu edhe UDP. Por n\u00ebse p\u00ebrdorni UDP, si garantoni dor\u00ebzimin?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Ka dy \u00e7\u00ebshtje. S\u00eb pari, un\u00eb i tregoj t\u00eb gjith\u00ebve se nuk garantojm\u00eb dor\u00ebzimin e logeve. Sepse, kur zhvilluesit vijn\u00eb dhe thon\u00eb: 'Le t\u00eb fillojm\u00eb t\u00eb shkruajm\u00eb t\u00eb dh\u00ebna financiare aty, dhe ju do t'i ruani diku p\u00ebr rast se ndodh ndonj\u00eb gj\u00eb', ne u p\u00ebrgjigjemi 'E shk\u00eblqyer! Le t\u00eb fillojm\u00eb t\u00eb bllokojm\u00eb sh\u00ebnimin n\u00eb socket dhe ta b\u00ebjm\u00eb k\u00ebt\u00eb n\u00eb transaksione, p\u00ebr t\u00eb garantuar q\u00eb ne e kemi vendosur n\u00eb socket dhe t\u00eb sigurohemi se ne e kemi marr\u00eb at\u00eb nga ana tjet\u00ebr.' N\u00eb k\u00ebt\u00eb moment, t\u00eb gjith\u00ebve fillon t\u00eb mos u nevojitet. Dhe pasi nuk kan\u00eb nevoj\u00eb, \u00e7far\u00eb pyetje na kan\u00eb ne? N\u00ebse nuk doni t\u00eb garantoni sh\u00ebnimin n\u00eb socket, p\u00ebrse duhet t\u00eb garantojm\u00eb dor\u00ebzimin? Ne b\u00ebjm\u00eb p\u00ebrpjekjet m\u00eb t\u00eb mira. Ne v\u00ebrtet p\u00ebrpiqemi t\u00eb dor\u00ebzojm\u00eb sa m\u00eb shum\u00eb dhe sa m\u00eb mir\u00eb, por nuk japim 100% garanci. Prandaj, nuk duhet t\u00eb shkruani atje t\u00eb dh\u00ebna financiare. P\u00ebr k\u00ebt\u00eb ka databaza me transaksione.<\/p>\n<\/p>\n
Pyetje<\/strong>: Kur API gjeneron nj\u00eb mesazh n\u00eb log dhe transferon kontrollin te mikrosh\u00ebrbimet, a keni hasur ndonj\u00ebher\u00eb n\u00eb problemin se mesazhet nga mikrosh\u00ebrbime t\u00eb ndryshme vijn\u00eb n\u00eb rend t\u00eb gabuar? Kjo shkakton konfuzion.<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: \u00cbsht\u00eb normale q\u00eb ato t\u00eb vijn\u00eb n\u00eb rend t\u00eb ndrysh\u00ebm. Duhet t\u00eb jeni gati p\u00ebr k\u00ebt\u00eb. Sepse \u00e7do d\u00ebrges\u00eb rrjet\u00ebrore nuk ju garanton rendin, ose duhet t\u00eb shpenzoni burime posa\u00e7\u00ebrisht p\u00ebr k\u00ebt\u00eb. N\u00ebse marim depozitat e skedar\u00ebve, \u00e7do API ruan loget n\u00eb skedarin e vet. N\u00eb t\u00eb v\u00ebrtet\u00eb, rsyslog i vendos ato n\u00eb kataloge. \u00c7do API ka loget e veta, ku mund t\u00eb shkoni dhe t\u00eb shikoni, dhe m\u00eb pas sipas timestamp n\u00eb k\u00ebt\u00eb log mund t'i korrespondoni. N\u00ebse ata shkojn\u00eb t\u00eb shikojn\u00eb n\u00eb Graylog, atje do t\u00eb sortohen sipas timestamp. Aty do t\u00eb jet\u00eb gjith\u00e7ka n\u00eb rregull.<\/p>\n<\/p>\n
Pyetje<\/strong>: Timestamp mund t\u00eb ndryshoj\u00eb me milisekonda.<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Timestamp gjenerohet nga vet\u00eb API. Kjo \u00ebsht\u00eb, n\u00eb thelb, e gjith\u00eb ideja. Ne kemi NTP. API gjeneron timestamp edhe n\u00eb mesazhin vet\u00eb. Nuk \u00ebsht\u00eb rsyslog q\u00eb e shton at\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: Nuk \u00ebsht\u00eb shum\u00eb e qart\u00eb si ndodhi nd\u00ebrveprimi midis qendrav\u00eb t\u00eb t\u00eb dh\u00ebnave. Brenda qendr\u00ebs s\u00eb t\u00eb dh\u00ebnave \u00ebsht\u00eb e qart\u00eb se si u mbledh\u00ebn dhe u p\u00ebrpunuan logjet. Si ndodh nd\u00ebrveprimi midis qendrave t\u00eb t\u00eb dh\u00ebnave? Apo secila qend\u00ebr t\u00eb dh\u00ebnash jeton jet\u00ebn e saj?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Pjes\u00ebrisht. \u00c7do vend ndodhet n\u00eb nj\u00eb qend\u00ebr t\u00eb dh\u00ebnash t\u00eb caktuar. Aktualisht nuk kemi shtrirje, q\u00eb nj\u00eb vend t\u00eb jet\u00eb i vendosur n\u00eb qendra t\u00eb dh\u00ebnash t\u00eb ndryshme. Prandaj, nuk \u00ebsht\u00eb e nevojshme t'i bashkojm\u00eb ato. Brenda \u00e7do qendre ka Log Relay. Kjo \u00ebsht\u00eb nj\u00eb server Rsyslog. N\u00eb t\u00eb v\u00ebrtet\u00eb, ka dy makina menaxhimi. Ato jan\u00eb t\u00eb konfiguruara nj\u00ebsoj. Por p\u00ebr momentin, thjesht trafiku kalon p\u00ebrmes nj\u00eb prej tyre. Ajo agregon t\u00eb gjith\u00eb logjet. Ka nj\u00eb radh\u00eb disku p\u00ebr \u00e7do rast. Ajo kompreson logjet dhe i d\u00ebrgon n\u00eb qendr\u00ebn e dh\u00ebnave qendrore (n\u00eb Singapor), nga ku m\u00eb pas ato d\u00ebrgohen n\u00eb Graylog. N\u00eb \u00e7do qend\u00ebr t\u00eb t\u00eb dh\u00ebnave ka edhe nj\u00eb storage t\u00eb tij. N\u00eb rast se na mungon lidhja, ne kemi t\u00eb gjitha logjet atje. Ato do t\u00eb mbesin atje. Ato do t\u00eb ruhen atje.<\/p>\n<\/p>\n
Pyetje<\/strong>: N\u00eb situata jasht\u00ebzakonshme, a e merrni logun nga atje?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Mund t\u00eb shkoni aty (n\u00eb storage) dhe t\u00eb shikoni.<\/p>\n<\/p>\n
Pyetje<\/strong>: Si i monitoroni q\u00eb t\u00eb mos i humbni logjet?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: N\u00eb t\u00eb v\u00ebrtet\u00eb, i humbim logjet dhe ne e monitorojm\u00eb k\u00ebt\u00eb. Monitorimi filloi para nj\u00eb muaji. N\u00eb bibliotek\u00ebn q\u00eb p\u00ebrdor Go API, ka metrika. Ajo di t\u00eb num\u00ebroj\u00eb se sa her\u00eb nuk ka mundur t\u00eb shkruaj\u00eb n\u00eb socket. Atje aktualisht ka nj\u00eb heuristik t\u00eb zgjuar. Ka nj\u00eb buffer. Ai p\u00ebrpiqet t\u00eb shkruaj\u00eb mesazhin nga ai n\u00eb socket. N\u00ebse bufferi mbushet, ai fillon t\u2019i heq\u00eb. Dhe llogarit se sa i ka hequr. N\u00ebse fillojn\u00eb t\u00eb mbushen numrat, ne merremi vesh p\u00ebr k\u00ebt\u00eb. Tani ato d\u00ebrgohen gjithashtu n\u00eb Prometheus, dhe n\u00eb Grafana mund t\u00eb shihni grafik\u00ebt. Mund t\u00eb konfiguroni njoftimet. Por p\u00ebr momentin nuk \u00ebsht\u00eb e qart\u00eb se kujt t'ia d\u00ebrgojm\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: N\u00eb elasticsearch, ju ruani logjet me rezervim. Sa replika keni?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Nj\u00eb replik\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: A \u00ebsht\u00eb vet\u00ebm nj\u00eb replik\u00eb?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Kjo \u00ebsht\u00eb nj\u00eb master dhe nj\u00eb replik\u00eb. T\u00eb dh\u00ebnat ruhen n\u00eb dy kopje.<\/p>\n<\/p>\n
Pyetje<\/strong>: A e keni modifikuar ndonj\u00ebher\u00eb madh\u00ebsin\u00eb e buffer-it t\u00eb rsyslog? <\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Ne shkruajm\u00eb datagramet n\u00eb nj\u00eb socket unix t\u00eb personalizuar. Kjo na v\u00eb menj\u00ebher\u00eb nj\u00eb kufizim prej 128 kilobajt. Nuk mund t\u00eb shkruajm\u00eb m\u00eb shum\u00eb n\u00eb t\u00eb. K\u00ebt\u00eb e kemi specifikuar n\u00eb standard. Ata q\u00eb duan t\u00eb kalojn\u00eb n\u00eb storag\u00eb, shkruajn\u00eb 128 kilobajt. Bibliotekat, p\u00ebr m\u00eb tep\u00ebr, e presin at\u00eb dhe vendosin nj\u00eb flag q\u00eb mesazhi \u00ebsht\u00eb prishur. N\u00eb standardin e vet t\u00eb mesazhit kemi nj\u00eb fush\u00eb t\u00eb ve\u00e7ant\u00eb q\u00eb tregon n\u00ebse \u00ebsht\u00eb prishur gjat\u00eb shkrimit apo jo. Prandaj kemi mund\u00ebsin\u00eb t\u00eb ndjekim edhe k\u00ebt\u00eb moment.<\/p>\n<\/p>\n
Pyetje<\/strong>: A shkruani JSON t\u00eb prishur? <\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: JSON i prishur do t\u00eb hidhet ose gjat\u00eb relay, p\u00ebr shkak t\u00eb paket\u00ebs supe t\u00eb madhe. Ose do t\u00eb hidhet nga Graylog, sepse nuk do t\u00eb jet\u00eb n\u00eb gjendje ta analizoj\u00eb JSON-in. Por ka disa nuanca q\u00eb duhen rregulluar, dhe ato kryesisht lidhen me rsyslog. Kam p\u00ebrfunduar aty disa \u00e7\u00ebshtje, mbi t\u00eb cilat duhet ende t\u00eb punojm\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse Kafka? A keni provuar RabbitMQ? A po has problemi me Graylog n\u00ebn k\u00ebto ngarkesa?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Nuk po funksionojm\u00eb me Graylog. Por me Graylog jemi duke funksionuar. \u00cbsht\u00eb tejet problematik. Ai \u00ebsht\u00eb nj\u00eb gj\u00eb origjinale. Dhe, n\u00eb t\u00eb v\u00ebrtet\u00eb, nuk \u00ebsht\u00eb i nevojsh\u00ebm. Do t\u00eb preferoja t\u00eb shkruaja nga rsyslog direkt n\u00eb elasticsearch dhe pastaj t\u00eb shihja Kibana. Por duhet t\u00eb zgjidhim \u00e7\u00ebshtjen me sigurimin. Ky \u00ebsht\u00eb nj\u00eb variant i mundsh\u00ebm i zhvillimit ton\u00eb, kur do t\u00eb heqim Graylog dhe do t\u00eb p\u00ebrdorim Kibana. Nuk do t\u00eb ket\u00eb kuptim t\u00eb p\u00ebrdorim Logstash. Sepse, mund t\u00eb b\u00ebj t\u00eb nj\u00ebjt\u00ebn gj\u00eb me rsyslog. Dhe ai ka nj\u00eb modul p\u00ebr t\u00eb shkruar n\u00eb elasticsearch. Me Graylog po p\u00ebrpiqemi t\u00eb jetojm\u00eb ndonj\u00ebher\u00eb. Madje e kemi b\u00ebr\u00eb pak tuning. Por ka ende hap\u00ebsir\u00eb p\u00ebr p\u00ebrmir\u00ebsime.<\/p>\n<\/p>\n
Rreth Kafka-s. K\u00ebshtu ka ndodhur historikisht. Kur erdha, ajo tashm\u00eb ekzistonte, dhe ishim duke shkruar skedar\u00ebt e log-ut n\u00eb t\u00eb. Ne thjesht ngrit\u00ebm klasterin ton\u00eb dhe kaluam log-et n\u00eb t\u00eb. Ne e menaxhojm\u00eb, dhe e dim\u00eb si ndihen. Rreth RabbitMQ... nuk shkon mir\u00eb me RabbitMQ. Por me RabbitMQ shkojm\u00eb. E kemi n\u00eb prodhim, dhe kishim disa probleme me t\u00eb. Tani, para shitjeve, e kemi rregulluar dhe tani punon normalisht. Por deri at\u00ebher\u00eb nuk isha i gatsh\u00ebm ta nxirrja n\u00eb prodhim. Ka nj\u00eb moment tjet\u00ebr. Graylog mund t\u00eb lexoj\u00eb versionin AMQP 0.9, nd\u00ebrsa rsyslog mund t\u00eb shkruaj\u00eb versionin AMQP 1.0. Nuk ka asnj\u00eb zgjidhje q\u00eb mund t\u00eb b\u00ebj\u00eb t\u00eb dyja. Ka vet\u00ebm nj\u00ebri ose tjetri. Prandaj, p\u00ebr momentin vet\u00ebm Kafka. Por gjithashtu ka nuanca t\u00eb veta. Sepse omkafka n\u00eb at\u00eb versionin e rsyslog-it q\u00eb po p\u00ebrdorim, mund t\u00eb humbas\u00eb t\u00ebr\u00eb tamponin e mesazheve q\u00eb ka marr\u00eb nga rsyslog. Deri tani po pajtohemi me k\u00ebt\u00eb. <\/p>\n
<\/p>\n
Pyetje<\/strong>: A e p\u00ebrdorni Kafka-n sepse ajo tashm\u00eb ishte atje? Nuk p\u00ebrdoret p\u00ebr ndonj\u00eb q\u00ebllim tjet\u00ebr?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Kafka, e cila ishte atje, p\u00ebrdoret nga ekipi i Data S\u0441ience. Ky \u00ebsht\u00eb nj\u00eb projekt krejt ndryshe, p\u00ebr t\u00eb cilin, fatkeq\u00ebsisht, nuk mund t\u00eb them asgj\u00eb. Nuk jam n\u00eb dijeni. Ajo ka qen\u00eb n\u00ebn menaxhimin e ekipit t\u00eb Data S\u0441ience. Kur hodh\u00ebn log-et, vendos\u00ebn ta p\u00ebrdorin at\u00eb, p\u00ebr t\u00eb mos u detyruar t\u00eb instalonin edhe t\u00eb vet\u00ebn. Tani e kemi p\u00ebrdit\u00ebsuar Graylog-un dhe humb\u00ebm kompatibilitetin, sepse atje \u00ebsht\u00eb nj\u00eb version i vjet\u00ebr i Kafka-s. Na duhej t\u00eb krijonim t\u00eb ton\u00ebn. Nj\u00ebkoh\u00ebsisht u shp\u00ebtuam nga ato kat\u00ebr tema p\u00ebr \u00e7do API. B\u00ebm\u00eb nj\u00eb tem\u00eb t\u00eb gjer\u00eb p\u00ebr t\u00eb gjitha live-t, nj\u00eb tem\u00eb t\u00eb gjer\u00eb p\u00ebr t\u00eb gjitha staging-t dhe thjesht i d\u00ebrgojm\u00eb t\u00eb gjitha aty. Graylog-i i ekzekuton k\u00ebto paralelisht.<\/p>\n<\/p>\n
Pyetje<\/strong>: Pse \u00ebsht\u00eb e nevojshme kjo magji me soket\u00ebt? A keni provuar t\u00eb p\u00ebrdorni log-drajverin syslog p\u00ebr kontejner\u00ebt?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: N\u00eb at\u00eb moment kur u angazhuam me k\u00ebt\u00eb \u00e7\u00ebshtje, marr\u00ebdh\u00ebniet tona me Docker-in ishin tensionuese. Ishte Docker 1.0 ose 0.9. Docker vet\u00eb ishte i \u00e7uditsh\u00ebm. S\u00eb dyti, n\u00ebse duhet t\u00eb fus\u00ebsh log-et n\u00eb t\u00eb... Kam nj\u00eb dyshim t\u00eb pasigurt se ai i kalon t\u00eb gjitha log-et p\u00ebrmes vetes, p\u00ebrmes demonit t\u00eb Docker-it. N\u00ebse nj\u00eb API yn\u00eb \u00ebsht\u00eb duke u \u00e7mendur, at\u00ebher\u00eb API-t\u00eb e tjera hasin v\u00ebshtir\u00ebsi n\u00eb d\u00ebrgimin e stdout dhe stderr. Nuk e di \u00e7far\u00eb do t\u00eb ndodh\u00eb nga kjo. Kam nj\u00eb ndjenj\u00eb intuitive se nuk duhet t\u00eb p\u00ebrdorim drejtuesin syslog t\u00eb Docker-it n\u00eb k\u00ebt\u00eb vend. Departamenti yn\u00eb p\u00ebr testimin funksional ka nj\u00eb kluster t\u00eb vetin Graylog me log-et. Ata p\u00ebrdorin drejtues log-esh t\u00eb Docker-it dhe duket se gjith\u00e7ka shkon mir\u00eb. Por ata e shkruajn\u00eb GELF direkt n\u00eb Graylog. Kur filluam k\u00ebt\u00eb, duhej t\u00eb funksiononte thjesht. Ndoshta m\u00eb von\u00eb, kur dikush t\u00eb thot\u00eb se gjith\u00e7ka funksionon mir\u00eb p\u00ebr nj\u00eb koh\u00eb t\u00eb gjat\u00eb, do ta provojm\u00eb.<\/p>\n<\/p>\n
Pyetje<\/strong>: Po e b\u00ebni d\u00ebrges\u00ebn mes qendrave t\u00eb t\u00eb dh\u00ebnave me rsyslog. Pse jo me Kafka?<\/p>\n<\/p>\n
P\u00ebrgjigje<\/strong>: Ne e b\u00ebjm\u00eb edhe k\u00ebshtu, edhe ashtu n\u00eb t\u00eb v\u00ebrtet\u00eb. P\u00ebr dy arsye. N\u00ebse kanali \u00ebsht\u00eb plot\u00ebsisht i prishur, t\u00eb gjith\u00eb log-et tan\u00eb, p\u00ebrfshir\u00eb ato t\u00eb kompresuar, nuk kalojn\u00eb p\u00ebrmes tij. Nd\u00ebrsa Kafka lejon q\u00eb ato thjesht t\u00eb humbasin gjat\u00eb procesit. Me k\u00ebt\u00eb metod\u00eb ne shmangim ngjeshjen e k\u00ebtyre log-eve. Thjesht p\u00ebrdorim Kafka n\u00eb k\u00ebt\u00eb rast direkt. N\u00ebse kanali yn\u00eb \u00ebsht\u00eb i mir\u00eb dhe duam ta l\u00ebm\u00eb at\u00eb t\u00eb lir\u00eb, at\u00ebher\u00eb p\u00ebrdorim rsyslog-in. Por n\u00eb t\u00eb v\u00ebrtet\u00eb, mund ta konfiguroni n\u00eb m\u00ebnyr\u00eb q\u00eb ai vet\u00eb t\u00eb hedh at\u00eb q\u00eb nuk kalon. N\u00eb k\u00ebt\u00eb moment ne thjesht p\u00ebrdorim d\u00ebrgimin rsyslog direkt n\u00eb disa vende, dhe Kafka n\u00eb t\u00eb tjer\u00ebt.<\/p>\nBurimi: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b. \u041d\u0443\u0436\u043d\u043e \u0440\u0435\u0448\u0438\u0442\u044c \u0441\u0440\u0430\u0437\u0443 \u043a\u0443\u0447\u0443 \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432: \u043a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438 \u0438\u0437 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f; \u043a\u0443\u0434\u0430 \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438; \u043a\u0430\u043a \u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u043b\u043e\u0433\u0438 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438; \u043a\u0430\u043a \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u0438 \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438. \u041f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u043d\u044b\u043d\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24507,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"Luaj](\"https:\/\/img.youtube.com\/vi\/NAeedJv-S3I\/hqdefault.jpg\")
\r\n