artiklit Badu\u0161ka kohta<\/a><\/noindex>, mis sellest kirjutas. Aga seal on \u00fcks n\u00fcanss. Selle kutse \u00fcmber oli veel l\u00f5putu ts\u00fckkel, kus pidevalt \u00fcritati s\u00f5numit socket'isse suruda. Seda me ei m\u00e4rganud. Pidiime raamatukogu \u00fcmber kirjutama. Sellest ajast on see mitmel korral muutunud, kuid praegu oleme k\u00f5ikidest all\u00fcksustest blokeeringutest vabanenud. Seega saab rsyslog'i peatada, ja midagi ei kuku alla.<\/p>\n<\/p>\n
Peame j\u00e4lgima j\u00e4rjekordade suurust, mis aitab v\u00e4ltida nende \u00e4mbrite peale astumist. Esiteks, me saame j\u00e4lgida, millal hakkame s\u00f5numeid kaotama. Teiseks, saame j\u00e4lgida, kas meil on \u00fcldiselt probleeme edastamisega.<\/p>\n
<\/p>\n
Ja veel \u00fcks ebameeldiv hetk \u2014 amplifikatsioon 10 korda mikroteenuste arhitektuuris \u2014 see on v\u00e4ga lihtne. Meil ei ole nii palju sisendeid, kuid nende s\u00f5numitega edasiviiv graaf, access-log'id t\u00f5ttu, suurendab tegelikult logide koormust umbes k\u00fcmme korda. Kahjuks ei j\u00f5udnud ma t\u00e4pseid numbreid arvestada, kuid mikroteenused \u2014 need on sellised. Seda tuleb silmas pidada. Tundub, et praegu on logide kogumise all\u00fcksus \u2014 k\u00f5ige koormatum Lazadas. <\/p>\n
<\/p>\n
<\/p>\n
Kuidas lahendada elasticsearch'i probleem? Kui on vaja kiiresti logisid \u00fchte kohta saada, et mitte joosta ringi k\u00f5igis masinates ja ei koguda neid seal, kasuta failide salvestust. See t\u00f6\u00f6tab garanteeritult. See tehakse igast serverist. Pead lihtsalt sinna kettad panema ja syslog'i seadma. P\u00e4rast seda on sul garanteeritult k\u00f5ik logid \u00fches kohas. Edasi saab rahulikult seadistada elasticsearch'i, graylog'i, v\u00f5i midagi muud. Kuid sul on juba k\u00f5ik logid, ja sa v\u00f5id neid hoida, kui palju kettaribad piisavad.<\/p>\n
<\/p>\n
<\/p>\n
Minu ettekande hetkeks n\u00e4gi skeem v\u00e4lja selline. Faili me praktiliselt ei kirjuta. Praegu t\u00f5en\u00e4oliselt l\u00fclitame viimased v\u00e4lja. Kohalikud masinad, kus API-d k\u00e4ivituvad, me ei hakka enam failidesse kirjutama. Esiteks on olemas failide salvestus, mis t\u00f6\u00f6tab v\u00e4ga h\u00e4sti. Teiseks, neil masinatel on pidevalt ruum otsa saanud, tuleb seda pidevalt j\u00e4lgida.<\/p>\n
<\/p>\n
See osa Logstashist ja Graylogist ajab t\u00f5eliselt n\u00e4rvi. Seet\u00f5ttu tuleb sellest vabaneda. Tuleb valida midagi \u00fchte.<\/p>\n
<\/p>\n
<\/p>\n
Me otsustasime Logstash'i ja Kibana eemaldada. Sest meil on turbosakonn, mis seondub. Side on selles, et Kibana ilma X-Pack'i ja Shieldita ei v\u00f5imalda logide juurdep\u00e4\u00e4su \u00f5iguste piiramist. Seet\u00f5ttu v\u00f5tsime Graylog'i. Seal on k\u00f5ik olemas. Mulle see ei meeldi, aga see t\u00f6\u00f6tab. Ostime uue riistvara, installisime sinna v\u00e4rske Graylog'i ja kandsime k\u00f5ik logid ranged formaadid eraldi Graylog'i. Me lahendasime probleemid erinevate sama t\u00fc\u00fcpi v\u00e4ljadega organisatsiooniliselt.<\/p>\n
<\/p>\n
<\/p>\n
Mis tegelikult uude Graylog'i kuulub. Me lihtsalt panime k\u00f5ik Dockerisse. V\u00f5tsime hulk servereid, jaotat\u00e4it 3 instantsi Kafka'd, 7 serverit Graylog'i versioonis 2.3 (sest soovisin Elasticsearch'i versiooni 5). K\u00f5ik see t\u00f5steti RAID-idega HDD-dest. N\u00e4gime indekseerimism\u00e4\u00e4ra kuni 100 000 s\u00f5numit sekundis. N\u00e4gime numbrit, et 140 terabaidi andmeid n\u00e4dalas. <\/p>\n
<\/p>\n
<\/p>\n
Ja j\u00e4lle \u00e4mbrid! Meil on tulemas kaks allahindlust. Me kasutasime \u00fcle 6 miljoni s\u00f5numi. Meie Graylog ei j\u00f5ua neid t\u00f6\u00f6tlema. Peame j\u00e4lle ellu j\u00e4\u00e4ma. <\/p>\n
<\/p>\n
<\/p>\n
Ellu j\u00e4ime me nii. Lisades veel m\u00f5ned serverid ja SSD-d. Praegu elame sellest viisist. N\u00fc\u00fcd t\u00f6\u00f6tleme juba 160k s\u00f5numit sekundis. Me ei ole veel piirini j\u00f5udnud, seega ei ole selge, kui palju me tegelikult suudame sellest v\u00e4lja pigistada. <\/p>\n
<\/p>\n
<\/p>\n
Siin on meie tulevikuplaanid. Nendest on tegelikult k\u00f5ige t\u00e4htsam, v\u00f5ib-olla, k\u00f5rge saadavus. Meil seda praegu ei ole. Mitmed masinad on seadistatud \u00fchtlaselt, kuid k\u00f5ik liigub praegu l\u00e4bi \u00fche masina. Tuleb aega v\u00f5tta, et seadistada nende vahel failover.<\/p>\n
<\/p>\n
Koguda m\u00f5\u00f5dikud Graylog'ilt.<\/p>\n
<\/p>\n
Seada kiirus piirang, et \u00fcks, hull API, ei h\u00e4vitaks meie ribalaiust ja k\u00f5ike muud.<\/p>\n
<\/p>\n
Ja l\u00f5puks, kirjuta m\u00f5ni SLA arendajatega, et me saame teenindada nii palju. Kui kirjutate rohkem, siis kahjuks ei saa.<\/p>\n
<\/p>\n
Ja kirjutada dokumentatsioon.<\/p>\n
<\/p>\n
<\/p>\n
L\u00fchidalt, kokkuv\u00f5te k\u00f5igest, mida oleme l\u00e4bi elanud. Esiteks, standardid. Teiseks, syslog \u2014 tort. Kolmandaks, rsyslog t\u00f6\u00f6tab just nii, nagu slaidil on kirjutatud. Ja n\u00fc\u00fcd liikume k\u00fcsimuste juurde.<\/p>\n
<\/p>\n
K\u00fcsimused<\/strong>.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Miks ikkagi otsustasite mitte v\u00f5tta\u2026 (filebeat?)<\/p>\n<\/p>\n
Vastus<\/strong>: Tuleb kirjutada faili. V\u00e4ga ei tahtnud. Kui teie API kirjutab tuhandeid teateid sekundis, siis isegi kui sa kord tunnis rotatsiooni teed, pole see ikkagi variant. V\u00f5ib kirjutada pipe'i. Selle peale k\u00fcsisid arendajad: \"Mis juhtub, kui protsess, kuhu me kirjutame, kukub?\" Ma ei leidnud sellele vastust ja \u00fctlesin: \"Noh, ok, siis me ei tee seda.\"<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Miks te ei kirjuta logisid lihtsalt HDFS-i?<\/p>\n<\/p>\n
Vastus<\/strong>: See on j\u00e4rgmine etapp. Me m\u00f5tlesime selle \u00fcle alguses, kuid kuna hetkel ei ole ressursse selleks tegeleda, siis see j\u00e4\u00e4b meil pikemaajalisse lahendusse.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Veergude formaat oleks sobivam.<\/p>\n<\/p>\n
Vastus<\/strong>: Ma saan k\u00f5igest aru. Me oleme \u201epooldavad\u201c m\u00f5lema k\u00e4ega. <\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Te kirjutate rsyslogi. Seal on nii TCP kui ka UDP. Aga kui UDP, siis kuidas te garanteerite kohaletoimetamise?<\/p>\n<\/p>\n
Vastus<\/strong>: On kaks momenti. Esiteks, \u00fctlen kohe, et me ei garanteeri logide kohaletoimetamist. Sest kui arendajad tulevad ja \u00fctlevad: \"Kuidas oleks, kui hakkame kirjutama finantsandmeid ja teie panete need kuhugi, ettevaatusabin\u00f5una, kui midagi juhtub,\" siis me vastame: \"Suurep\u00e4rane! Laske teil luua blokeering kirjutamiseks soketisse ja teha seda tehingute kaupa, et me saaksime kindlustada, et te meile selle soketisse edastate ja olete veendunud, et me saime selle teiselt poolt.\" Ja sel hetkel kaob k\u00f5igil huvi. Kui te ei taha garanteerida kirjutamist soketisse, siis miks peaksime meie garantii andma? Me p\u00fc\u00fcame anda parimat, aga me ei anna 100% garantiid. Seega, \u00e4rge kirjutage sinna finantsandmeid. Selleks on andmebaasid tehingutega.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Kui API genereerib mingi teate logis ja edastab juhtimise mikroteenustele, kas olete kokku puutunud probleemiga, et teated erinevatest mikroteenustest j\u00f5uavad vale j\u00e4rjestuses? Seet\u00f5ttu tekib segadus.<\/p>\n<\/p>\n
Vastus<\/strong>: On normaalne, et need tulevad erinevas j\u00e4rjekorras. Selleks tuleb valmistuda. Sest kettetoimetamine ei garanteeri j\u00e4rjekorda, v\u00f5i tuleb selleks spetsiaalselt ressursse investeerida. Kui vaatame failisalvestusi, siis iga API salvestab logid oma faili. T\u00e4psemalt \u00f6eldes, rsyslog jagab need kaustadesse. Igal API-l on oma logid, kuhu saab minna ja vaadata ning hiljem timestamp'i j\u00e4rgi neid kokku viia. Kui nad vaatavad Graylogis, siis seal need j\u00e4rjestatakse timestamp'i j\u00e4rgi. K\u00f5ik on korras.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Timestamp v\u00f5ib erineda millisekondide v\u00f5rra.<\/p>\n<\/p>\n
Vastus<\/strong>: Timestamp genereerib ise API. Selles ongi kogu asi. Meil on NTP. API genereerib timestamp'i juba s\u00f5numis. Seda ei lisa rsyslog.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Ei ole v\u00e4ga selge andmekeskuste vahelise koost\u00f6\u00f6 osas. Andmekeskuse raames on aru saada, kuidas logid koguti, t\u00f6\u00f6deldi. Kuidas toimub koost\u00f6\u00f6 andmekeskuste vahel? Kas iga andmekeskus elab oma elu?<\/p>\n<\/p>\n
Vastus<\/strong>: Peaaegu. Igas riigis on mingi andmekeskus. Meil ei ole hetkel hajutamist, et \u00fches riigis oleks mitu andmekeskust. Seega ei pea neid \u00fchendama. Iga keskuse sees on Log Relay. See on Rsyslog server. Tegelikult on seal kaks haldusmasinat. Need on \u00fchtmoodi seadistatud. Kuid praegu l\u00e4heb lihtsalt liiklus l\u00e4bi \u00fche neist. See kogub k\u00f5ik logid. Sellel on disk queue igaks juhuks. See tihendab logisid ja saadab need kesksetesse andmekeskustesse (Singapur), kus need saadetakse edasi Graylogi. Ja igas andmekeskuses on oma failide salvestus. Kui meil kaob \u00fchendus, siis on meil k\u00f5ik logid seal. Need j\u00e4\u00e4vad sinna. Need s\u00e4ilivad.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: \u00c4\u00e4rmuslikes olukordades saate sealt logid?<\/p>\n<\/p>\n
Vastus<\/strong>: V\u00f5ib minna sinna (failis\u00fcsteemi) ja vaadata.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Kuidas te j\u00e4lgite, et te ei kaota logisid?<\/p>\n<\/p>\n
Vastus<\/strong>: Tegelikult me kaotame neid ja me j\u00e4lgime seda. J\u00e4lgimine k\u00e4ivitus kuu aega tagasi. Go API-s kasutatavas teegis on meetrika. See suudab arvutada, mitu korda tal ei \u00f5nnestunud soketisse kirjutada. Seal on hetkel nutikas heuristika. Seal on buffer. See p\u00fc\u00fcab kirjutada selle kaudu s\u00f5numi soketisse. Kui buffer t\u00e4itub, hakkab see neid langetama. Ja loendab, kui palju ta on langetanud. Kui seal hakkavad loendurid t\u00e4ituma, saame sellest teada. Need j\u00f5uavad ka prometheus'e kaudu ning Grafanas saab vaadata graafikuid. Saame seada teavitusi. Kuid praegu pole selge, kellele neid saata.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Elasticsearchis hoidsite logisid varukoopiate kaupa. Kui palju teil on koopiaid?<\/p>\n<\/p>\n
Vastus<\/strong>: \u00dcks koopia.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: See on k\u00f5igest \u00fcks koopia?<\/p>\n<\/p>\n
Vastus<\/strong>: See, see, what this master and replica are. Data is stored in two instances.<\/p>\n<\/p>\n
K\u00fcsimus<\/strong>: Have you somehow adjusted the rsyslog buffer size? <\/p>\n<\/p>\n
Vastus<\/strong>: We write datagrams to a custom Unix socket. This immediately imposes a limit of 128 kilobytes. We cannot write more than that. We have specified this in the standard. Those who want to reach the storages write 128 kilobytes. Libraries, by the way, truncate and set a flag indicating that the message has been truncated. In our standard message, there is a special field that shows whether it was truncated when written or not. So we have the ability to track this moment.<\/p>\n<\/p>\n
Question<\/strong>: Do you write broken JSON? <\/p>\n<\/p>\n
Vastus<\/strong>: Broken JSON will be discarded either during relay due to oversized packet or will be discarded by Graylog because it cannot parse the JSON. But there are nuances that need fixing, and they are mostly linked to rsyslog. I have filled out several issues there that still need work.<\/p>\n<\/p>\n
Question<\/strong>: Why Kafka? Have you tried RabbitMQ? Is Graylog not working under such loads?<\/p>\n<\/p>\n
Vastus<\/strong>: We have issues with Graylog. Graylog has issues with us. It's a peculiar thing. And, to be honest, it's not necessary. I would prefer to write directly from rsyslog to Elasticsearch and then check Kibana. But we need to settle issues with the security team. This is a possible direction for our development, where we might discard Graylog and use Kibana. Using Logstash wouldn\u2019t make sense. Because I can accomplish the same functionalities with rsyslog. And it has a module for writing to Elasticsearch. We're trying to coexist with Graylog. We've even slightly tuned it. But there's still room for improvement.<\/p>\n<\/p>\n
Regarding Kafka. It historically happened that way. When I came, it was already there, and logs were already being written to it. We just raised our cluster and migrated logs into it. We manage it; we know how it feels. On RabbitMQ... we don't get along with RabbitMQ. But RabbitMQ works fine for us. We have it in production, and there were problems with it. Just before the sale, it was tuned, and it started working properly. But before that, I was not ready to release it to production. There's another point. Graylog can read version AMQP 0.9, while rsyslog can write version AMQP 1.0. And there is no solution that can handle both. It's either one or the other. So at this moment, only Kafka is available. But that also has its nuances. Because the omkafka of the version of rsyslog that we are using can lose the entire buffer of messages that it pulled from rsyslog. For now, we\u2019re living with it. <\/p>\n
<\/p>\n
Question<\/strong>: You use Kafka because it was available? Not for any other purposes?<\/p>\n<\/p>\n
Vastus<\/strong>: The Kafka that existed is used by the Data Science team. It's a completely separate project that I unfortunately cannot comment on. I'm not aware of its details. It was under the management of the Data Science team. When logs were introduced, they decided to use it so as not to set up a separate one. Now we've upgraded Graylog, and we've lost compatibility because there's an old version of Kafka. We had to set up our own. In the process, we eliminated those four topics for each API. We created one wide topic for all live activities, one wide topic for all staging and just shoot everything into it. Graylog extracts everything in parallel.<\/p>\n<\/p>\n
Question<\/strong>: Why this sorcery with sockets? Have you tried using a syslog log driver for containers?<\/p>\n<\/p>\n
Vastus<\/strong>: When we were asking ourselves this question, our relationship with Docker was strained. It was Docker 1.0 or 0.9. Docker itself was strange. Secondly, if you also push logs into it\u2026 I have an unverified suspicion that it passes all logs through itself, through the Docker daemon. If one API goes haywire, other APIs get stuck because they cannot send stdout and stderr. I don't know what it will lead to. I have a hunch that we should not use the Docker syslog driver at this point. Our functional testing department has its own little Graylog cluster with logs. They use Docker log drivers and supposedly everything is fine there. But they write GELF directly into Graylog. At that time, when we were doing all this, we needed it to just work. Maybe later, if someone comes and says it has been working fine for ages, we will give it a try.<\/p>\n<\/p>\n
Question<\/strong>: You make delivery between data centers on rsyslog. Why not on Kafka?<\/p>\n<\/p>\n
Vastus<\/strong>: Me teeme nii ja naa tegelikult. Kahel p\u00f5hjusel. Kui kanal on t\u00e4iesti katki, siis ei mahtu meie logid isegi kokkusurutud kujul sinna. Kafka v\u00f5imaldab neid lihtsalt kaduma l\u00e4inud protsessi jooksul. Sel viisil vabanevad me nendest logide ummistustest. Me kasutame antud juhul otse Kafka't. Kui meil on hea kanal ja soovime seda vabastada, siis kasutame nende rsyslog'i. Kuid tegelikult saab seda seadistada nii, et see viskab ise v\u00e4lja, mis ei mahtu. Praegu kasutame m\u00f5nes kohas otse rsyslog'i kohaletoimetamist, m\u00f5nes Kafka't.<\/p>\nAllikas: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b. \u041d\u0443\u0436\u043d\u043e \u0440\u0435\u0448\u0438\u0442\u044c \u0441\u0440\u0430\u0437\u0443 \u043a\u0443\u0447\u0443 \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432: \u043a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438 \u0438\u0437 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f; \u043a\u0443\u0434\u0430 \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438; \u043a\u0430\u043a \u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u043b\u043e\u0433\u0438 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438; \u043a\u0430\u043a \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u0438 \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438. \u041f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u043d\u044b\u043d\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24507,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"Vaata](\"https:\/\/img.youtube.com\/vi\/NAeedJv-S3I\/hqdefault.jpg\")
\r\n