{"id":53590,"date":"2019-12-05T00:00:00","date_gmt":"2019-12-04T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-my-v-tsian-ukroshhali-terabajty-logov"},"modified":"2020-02-18T14:01:30","modified_gmt":"2020-02-18T11:01:30","slug":"kak-my-v-tsian-ukroshhali-terabajty-logov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-v-tsian-ukroshhali-terabajty-logov","title":{"rendered":"How we at CIAN tamed terabytes of logs","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"How
\n
\nHello everyone, my name is Alexander, I work at CIAN as an engineer, focusing on system administration and automating infrastructure processes. In the comments to one of our previous articles, we were asked to explain how we generate 4 TB of logs per day and what we do with them. Yes, we have a lot of logs and for their processing, a separate infrastructure cluster has been created that allows us to promptly resolve issues. In this article, I will describe how we adapted it over the past year to handle the constantly growing data stream.
\n<\/a><\/noindex><\/p>\n

Where we started<\/h3>\n

\n\"How
\n
\nIn recent years, the load on cian.ru has been growing rapidly, and by the third quarter of 2018, the site's traffic reached 11.2 million unique users per month. At that time, during critical moments, we were losing up to 40% of logs, which hindered our ability to address incidents quickly and we spent a lot of time and effort resolving them. We also often struggled to identify the root cause of problems, which would then recur after some time. It was a nightmare that needed to be addressed.<\/p>\n

At that time, we used a cluster of 10 data nodes with ElasticSearch version 5.5.2 with standard index settings for log storage. It was implemented over a year ago as a popular and accessible solution; the log stream was not very large then, so there was no point in devising non-standard configurations.\u00a0<\/p>\n

Log processing was handled by Logstash on different ports across five ElasticSearch coordinators. One index, regardless of its size, consisted of five shards. Hourly and daily rotations were organized, resulting in about 100 new shards appearing in the cluster every hour. As long as the volume of logs was manageable, the cluster performed well, and no one paid much attention to its settings.\u00a0<\/p>\n

Challenges of Rapid Growth<\/h3>\n

\nThe volume of generated logs grew very rapidly due to two converging processes. On one hand, the number of users of the service was increasing. On the other hand, we began actively transitioning to a microservices architecture, breaking down our old monoliths into C# and Python. Several dozen new microservices replacing parts of the monolith were generating significantly more logs for the infrastructure cluster.\u00a0<\/p>\n

It was the scaling that led us to a point where the cluster became virtually unmanageable. When logs started coming in at a rate of 20,000 messages per second, frequent ineffective rotation increased the number of shards to 6,000, with over 600 shards per node.\u00a0<\/p>\n

This caused issues with memory allocation, and when a node failed, all shards had to move simultaneously, multiplying the traffic and burdening the other nodes, which made writing data to the cluster nearly impossible. During this time, we were without logs. And when there was a problem with proxy server<\/a> we would lose 1\/10 of the cluster overall. The large number of small-sized indexes added complexity.<\/p>\n

Without logs, we couldn't understand the reasons behind incidents and could end up repeating the same mistakes sooner or later, which was unacceptable to our team\u2019s ideology, as all our work mechanisms are specifically designed to avoid repeating issues. We needed complete logging and delivery of logs almost in real-time, as our on-call engineers monitored alerts not only from metrics but also from logs. To illustrate the scale of the problem \u2014 at that time, the total log volume was about 2 TB per day.\u00a0<\/p>\n

We set a goal \u2014 to completely eliminate log loss and reduce the delivery time to the ELK cluster to a maximum of 15 minutes during emergencies (this figure later became our internal KPI).<\/p>\n

The new rotation mechanism and hot-warm nodes<\/h3>\n

\n\"How
\n
\nWe began transforming the cluster by upgrading ElasticSearch from version 5.5.2 to 6.4.3. Our version 5 cluster crashed again, so we decided to shut it down and completely upgrade it \u2014 after all, there were no logs anyway. So this transition took us just a couple of hours.<\/p>\n

The most significant transformation at this stage was the implementation of Apache Kafka on three nodes, with the coordinator acting as an intermediate buffer. The message broker saved us from losing logs during issues with ElasticSearch. At the same time, we added 2 nodes to the cluster and switched to a hot-warm architecture with three 'hot' nodes placed in different racks in the data center. We redirected logs that absolutely couldn't be lost \u2014 nginx logs and application error logs \u2014 to these nodes. Minor logs such as debug, warning, etc., were sent to the remaining nodes, and after 24 hours, 'important' logs moved from the 'hot' nodes.<\/p>\n

To avoid increasing the number of small indexes, we switched from time-based rotation to a rollover mechanism. There was much information on the forums stating that size-based rotation is very unreliable, so we decided to use rotation based on the number of documents in the index. We analyzed each index and recorded the number of documents after which the rotation should occur. This way, we achieved an optimal shard size of no more than 50 GB.\u00a0<\/p>\n

Cluster Optimization<\/h3>\n

\n\"How
\n
\nHowever, we didn't completely eliminate problems. Unfortunately, small indexes still appeared: they didn't reach the specified volume, weren't rotated, and were removed by the global cleanup of indexes older than three days, since we removed time-based rotation. This led to data loss because the index disappeared completely from the cluster, and attempts to write to a nonexistent index broke the logic of the curator we used for management. The alias for writing transformed into an index and disrupted the rollover logic, causing uncontrolled growth of some indexes up to 600 GB.\u00a0<\/p>\n

For example, for the rollover config:<\/p>\n

curator-elk-rollover.yaml\n\n---\nactions:\n  1:\n    action: rollover\n    options:\n      name: \"nginx_write\"\n      conditions:\n        max_docs: 100000000\n  2:\n    action: rollover\n    options:\n      name: \"python_error_write\"\n      conditions:\n        max_docs: 10000000\n<\/code><\/pre>\n
In the absence of a rollover alias, the following error occurred:<\/p>\n
ERROR     alias \"nginx_write\" not found.\nERROR     Failed to complete action: rollover.  : Unable to perform index rollover with alias \"nginx_write\".\n<\/code><\/pre>\n
We left the solution to this problem for the next iteration and tackled another issue: we switched to a pull logic for Logstash, which processes incoming logs (removing unnecessary information and enriching them). We placed it in Docker, which we run through docker-compose; we also set up logstash-exporter there, which sends metrics to Prometheus for real-time monitoring of the log flow. This allowed us to smoothly adjust the number of Logstash instances responsible for processing each type of log.<\/p>\n
While improving the cluster, traffic to cian.ru grew to 12.8 million unique users per month. As a result, our transformations somewhat lagged behind the changes in production, and we found that the 'warm' nodes could not handle the load and were slowing down the entire log delivery. We received 'hot' data without failures, but we had to intervene in the delivery of others and perform a manual rollover to evenly distribute the indices.\u00a0<\/p>\n
At the same time, scaling and adjusting the settings of Logstash instances in the cluster was complicated by the fact that this was a local docker-compose, and all actions were performed manually (to add new endpoints, it was necessary to go through all servers manually and execute docker-compose up -d everywhere).<\/p>\n
Log Redistribution<\/h3>\n
\nIn September of this year, we continued to decompose the monolith, the load on the cluster was increasing, and the log flow approached 30,000 messages per second.\u00a0<\/p>\n
\"How
\n
\nWe began the next iteration with hardware upgrades. We reduced from five coordinators to three, replaced the data nodes, and gained both cost savings and storage capacity. We use two configurations for the nodes:\u00a0<\/p>\n