the article about Badushka<\/a><\/noindex>, which wrote about this. But there is one point. Around this call, there was also an infinite loop where attempts were constantly made to push a message into the socket. We did not notice that. We had to rewrite the library. Since then, it has changed a few times, but now we have eliminated all locks in all subsystems. Therefore, we can stop rsyslog, and nothing will crash.<\/p>\n<\/p>\n
We need to monitor the queue sizes, which helps avoid these pitfalls. Firstly, we can monitor when we start losing messages. Secondly, we can monitor if we have delivery issues.<\/p>\n
<\/p>\n
And another unpleasant moment \u2014 amplification by 10 times in a microservices architecture \u2014 is very easy. We don\u2019t have that many incoming requests, but due to the graph that these messages traverse further, because of the access logs, we actually increase the load on the logs by about ten times. Unfortunately, I didn\u2019t have time to calculate the exact numbers, but microservices are like that. This needs to be taken into account. Currently, the log collection subsystem is the most loaded one in Lazada. <\/p>\n
<\/p>\n
<\/p>\n
How to solve the elasticsearch problem? If you need to quickly get logs in one place, without running around all the machines and collecting them there, use file storage. This works reliably. It can be set up on any server. You just need to connect some disks and set up syslog. After this, you will definitely have all the logs in one place. Then you can leisurely configure elasticsearch, graylog, or something else. But you will already have all the logs, and you can store them as long as your disk arrays allow.<\/p>\n
<\/p>\n
<\/p>\n
At the time of my report, the scheme looked like this. We have practically stopped writing to files. Now, most likely, we will disconnect the remnants. On the local machines where the APIs are running, we will stop writing to files. Firstly, there is a file storage that works very well. Secondly, these machines constantly run out of space, so it needs to be constantly monitored.<\/p>\n
<\/p>\n
This part with Logstash and Graylog is really a hassle. So we need to get rid of it. We need to choose one.<\/p>\n
<\/p>\n
<\/p>\n
We decided to ditch Logstash and Kibana. Why? The connection lies in the fact that Kibana without X-Pack and Shield does not allow for differentiated access rights to the logs. Therefore, we opted for Graylog. It\u2019s not my favorite, but it gets the job done. We bought new hardware, set up a fresh Graylog, and migrated all the logs with strict formats to a separate Graylog. We resolved the issue of different types of similar fields organizationally.<\/p>\n
<\/p>\n
<\/p>\n
So, what\u2019s included in the new Graylog? We simply recorded everything in Docker. We took a bunch of servers, deployed three Kafka instances, and seven Graylog servers version 2.3 (because we wanted Elasticsearch version 5). All of this was set up on HDD RAID. We observed an indexing rate of up to 100,000 messages per second. We recorded that 140 terabytes of data were processed weekly. <\/p>\n
<\/p>\n
<\/p>\n
And once again, we hit a snag! We have two sales coming up. We migrated over 6 million messages. Graylog can\u2019t keep up with processing them. We need to find a way to survive once more. <\/p>\n
<\/p>\n
<\/p>\n
This is how we managed to survive. We added a few more servers and SSDs. Currently, we are operating in this manner. Now we are processing 160k messages per second. We haven\u2019t hit the limit yet, so it\u2019s still unclear how much we can really pull from this. <\/p>\n
<\/p>\n
<\/p>\n
Here are our plans for the future. Realistically, the most important aspect is high availability. We don\u2019t have it yet. Several machines are configured the same, but everything is still running through one machine. We need to take the time to set up failover between them.<\/p>\n
<\/p>\n
Gather metrics from Graylog.<\/p>\n
<\/p>\n
Implement rate limiting so that one overwhelmed API doesn\u2019t consume all our bandwidth and everything else.<\/p>\n
<\/p>\n
And finally, sign some SLA with the developers, stating that we can handle this amount. If you send more, then sorry.<\/p>\n
<\/p>\n
And write documentation.<\/p>\n
<\/p>\n
<\/p>\n
Briefly, here are the conclusions from everything we\u2019ve experienced. First, standards. Second, syslog is great. Third, rsyslog works exactly as outlined in the slide. Now, let\u2019s move on to questions.<\/p>\n
<\/p>\n
Questions<\/strong>.<\/p>\n<\/p>\n
Question<\/strong>: Why did you decide not to take\u2026 (filebeat?)<\/p>\n<\/p>\n
Answer<\/strong>: We need to write to a file. It was very undesirable. When your API writes thousands of messages per second, rotating once an hour is still not feasible. Writing to a pipe is possible. To which the developers asked me: \"What happens if the process we\u2019re writing to crashes?\" I simply didn\u2019t know how to respond and said, \"Well, okay, let\u2019s not do that.\"<\/p>\n<\/p>\n
Question<\/strong>: Why don't you just write logs in HDFS?<\/p>\n<\/p>\n
Answer<\/strong>: This is the next stage. We considered it at the very beginning, but since there are currently no resources to work on it, it's on our long-term solution list.<\/p>\n<\/p>\n
Question<\/strong>: A columnar format would be more suitable.<\/p>\n<\/p>\n
Answer<\/strong>: I understand everything. We are 'for' it with both hands. <\/p>\n<\/p>\n
Question<\/strong>: You write to rsyslog. Both TCP and UDP are possible. But if it's UDP, how do you guarantee delivery?<\/p>\n<\/p>\n
Answer<\/strong>: There are two points. First, I always tell everyone that we do not guarantee log delivery. Because when developers come and say, 'Let's start writing financial data there, and you store it somewhere in case something happens,' we respond, 'Great! How about you block on writing to the socket and do this in transactions, so you can guarantee that you send it to the socket and ensure we receive it on the other side?' At that moment, it becomes unnecessary for everyone. And if it\u2019s not necessary, what questions do we have? If you don\u2019t want to guarantee writing to the socket, why should we guarantee delivery? We make our best effort. We really try to deliver as much as possible and as well as possible, but we do not provide a 100% guarantee. Therefore, please do not write financial data there. For that purpose, there are databases with transactions.<\/p>\n<\/p>\n
Question<\/strong>: When the API generates a log message and hands over control to microservices, have you encountered the issue where messages from different microservices arrive out of order? This causes confusion.<\/p>\n<\/p>\n
Answer<\/strong>: It's normal for them to arrive in different orders. You need to be prepared for that. Because any network delivery does not guarantee order, or you need to spend resources specifically for that. If we take file storage, each API saves logs in its own file. Actually, rsyslog sorts them into directories. Each API has its own logs that you can go and check, and then you can correlate them by timestamp in that log. If they go to check in Graylog, it will sort them by timestamp. Everything will be fine there.<\/p>\n<\/p>\n
Question<\/strong>: The timestamp can differ by milliseconds.<\/p>\n<\/p>\n
Answer<\/strong>: The timestamp is generated by the API itself. That's the whole point. We have NTP. The API generates the timestamp within the message itself. It's not added by rsyslog.<\/p>\n<\/p>\n
Question<\/strong>: It's not very clear how the interaction between data centers works. Within the data center, it's clear how logs are collected and processed. How does the interaction between data centers occur? Or does each data center operate independently?<\/p>\n<\/p>\n
Answer<\/strong>: Almost. Each country is located in one specific data center. Currently, we don't have distribution where one country is hosted across different data centers. So there's no need to merge them. Inside each center, there is a Log Relay. This is an Rsyslog server. In fact, there are two management machines. They are configured identically. But for now, traffic is just flowing through one of them. It aggregates all the logs. It has a disk queue just in case. It compresses the logs and sends them to the central data center (in Singapore), where they are then forwarded to Graylog. And each data center has its own file storage. In case we lose connection, we have all the logs stored there. They will remain there. They will be saved.<\/p>\n<\/p>\n
Question<\/strong>: Do you get logs from there in case of unusual situations?<\/p>\n<\/p>\n
Answer<\/strong>: You can go there (to the file storage) and check.<\/p>\n<\/p>\n
Question<\/strong>: How do you monitor that you're not losing logs?<\/p>\n<\/p>\n
Answer<\/strong>: Actually, we do lose them, and we monitor that. The monitoring was launched a month ago. In the library used by the Go API, there are metrics. It can count how many times it has failed to write to the socket. There is currently a clever heuristic in place. There is a buffer. It tries to write messages from it to the socket. If the buffer gets full, it starts dropping them. It counts how many it has dropped. If the counters start overflowing, we find out about it. They now also come to Prometheus, and you can see graphs in Grafana. Alerts can be set up. But it's still unclear who to send them to.<\/p>\n<\/p>\n
Question<\/strong>: In Elasticsearch, you store logs with redundancy. How many replicas do you have?<\/p>\n<\/p>\n
Answer<\/strong>: One replica.<\/p>\n<\/p>\n
Question<\/strong>: Is it only one replica?<\/p>\n<\/p>\n
Answer<\/strong>: It's a master and a replica. Data is stored in two instances.<\/p>\n<\/p>\n
Question<\/strong>: Have you adjusted the size of the Rsyslog buffer? <\/p>\n<\/p>\n
Answer<\/strong>: We write datagrams to a custom Unix socket. This immediately imposes a limit of 128 kilobytes. We cannot write more than that. We have defined this in the standard. Anyone who wants to access the storages must write 128 kilobytes. The libraries, by the way, truncate and set a flag indicating that the message has been truncated. In our message standard, there is a special field that shows whether it was truncated during writing or not. So we have the ability to track that aspect as well.<\/p>\n<\/p>\n
Question<\/strong>: Are you writing broken JSON? <\/p>\n<\/p>\n
Answer<\/strong>: Broken JSON will be discarded either during relay because the packet is too large, or it will be rejected by Graylog because it won't be able to parse the JSON. But there are nuances that need fixing, most of which are related to rsyslog. I have already filled in a few issues there that still need work.<\/p>\n<\/p>\n
Question<\/strong>: Why Kafka? Have you tried RabbitMQ? Is Graylog not coping under such loads?<\/p>\n<\/p>\n
Answer<\/strong>: We are not getting along with Graylog. While Graylog is working for us, it is quite problematic. It's a peculiar tool. And, honestly, it\u2019s not necessary. I would prefer to write directly from rsyslog to Elasticsearch and then monitor it with Kibana. But I need to settle the issue with the security team. This is a possible avenue for our development, where we would throw out Graylog and use Kibana instead. There's no point in using Logstash. Because I can do all of that with rsyslog. And it has a module for writing to Elasticsearch. We are trying to make things work with Graylog. We have even tuned it a bit. But there\u2019s still room for improvement.<\/p>\n<\/p>\n
About Kafka. Historically, it has been this way. When I came in, it was already there, and logs were already being written to it. We just set up our cluster and moved the logs over. We manage it, and we know how it's doing. Regarding RabbitMQ... we don't get along with RabbitMQ. But we do have it. It's in production, and we've had issues with it. Before the sale, it was patched up and started to work normally. But before that, I wasn't ready to deploy it to production. There's one more thing. Graylog can read AMQP 0.9, while rsyslog can write AMQP 1.0. And there is no solution that can do both. It's either one or the other. So for now, it's only Kafka. But there are nuances there as well. Because omkafka, the version of rsyslog we use, can lose the entire message buffer that it has pulled from rsyslog. For now, we are dealing with this. <\/p>\n
<\/p>\n
Question<\/strong>: You use Kafka because it was already there? Is it not used for any other purposes?<\/p>\n<\/p>\n
Answer<\/strong>: The Kafka that we had is used by the Data Science team. It's a completely separate project that I unfortunately can't comment on. I am not aware of it. It was under the responsibility of the Data Science team. When they started working with logs, they decided to use it, so as not to set up their own. Now we have upgraded Graylog, and we lost compatibility because it has an old version of Kafka. We had to set up our own. At the same time, we eliminated those four topics for each API. We created one broad topic for all live logs, one broad topic for all staging logs, and simply funnel everything there. Graylog pulls all of this in parallel.<\/p>\n<\/p>\n
Question<\/strong>: Why is this tweaking with sockets necessary? Have you tried using the syslog log driver for containers?<\/p>\n<\/p>\n
Answer<\/strong>: At the time we were addressing this issue, our relationship with Docker was tense. It was either Docker 1.0 or 0.9. Docker itself was quite peculiar. Secondly, putting logs into it\u2026 I have an unverified suspicion that it passes all logs through itself via the Docker daemon. If one API crashes, the other APIs get stuck because they can't send stdout and stderr. I don't know what this will lead to. I have a gut feeling that we shouldn't use the Docker syslog driver in this instance. Our functional testing department has its own little Graylog cluster for logs. They use Docker log drivers and seem to have everything working fine. But they write GELF directly to Graylog. At the time we were planning all of this, we needed it to simply work. Perhaps later, when someone comes in and says that it has been working fine for ages, we will give it a try.<\/p>\n<\/p>\n
Question<\/strong>: You handle delivery between data centers using rsyslog. Why not Kafka?<\/p>\n<\/p>\n
Answer<\/strong>: We actually do both for two reasons. If the channel is completely broken, even our logs in compressed form can't get through. Kafka allows us to simply lose them in the process. This way, we avoid the bottleneck of those logs. In this case, we directly use Kafka. If we have a good channel and want to free it up, we use rsyslog. But in reality, it can be configured to drop what doesn't make it through. Currently, we use rsyslog delivery directly in some places and Kafka in others.<\/p>\nSource: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u041b\u043e\u0433\u0438 \u2014 \u0432\u0430\u0436\u043d\u0430\u044f \u0447\u0430\u0441\u0442\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u044b, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0430\u044f \u043f\u043e\u043d\u044f\u0442\u044c, \u0447\u0442\u043e \u043e\u043d\u0430 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 (\u043b\u0438\u0431\u043e \u043d\u0435 \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442), \u043a\u0430\u043a \u043e\u0436\u0438\u0434\u0430\u0435\u0442\u0441\u044f. \u0412 \u0443\u0441\u043b\u043e\u0432\u0438\u044f\u0445 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b \u0440\u0430\u0431\u043e\u0442\u0430 \u0441 \u043b\u043e\u0433\u0430\u043c\u0438 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u0438\u0441\u0446\u0438\u043f\u043b\u0438\u043d\u043e\u0439 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 \u043e\u043b\u0438\u043c\u043f\u0438\u0430\u0434\u044b. \u041d\u0443\u0436\u043d\u043e \u0440\u0435\u0448\u0438\u0442\u044c \u0441\u0440\u0430\u0437\u0443 \u043a\u0443\u0447\u0443 \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432: \u043a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438 \u0438\u0437 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f; \u043a\u0443\u0434\u0430 \u043f\u0438\u0441\u0430\u0442\u044c \u043b\u043e\u0433\u0438; \u043a\u0430\u043a \u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u043b\u043e\u0433\u0438 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438; \u043a\u0430\u043a \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u0438 \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438. \u041f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u043d\u044b\u043d\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24507,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"Play](\"https:\/\/img.youtube.com\/vi\/NAeedJv-S3I\/hqdefault.jpg\")
\r\n