{"id":83188,"date":"2020-05-29T07:43:02","date_gmt":"2020-05-29T05:43:02","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-perezhili-rezkij-rost-nagruzki-x10-na-udalenke-i-kakie-vyvody-sdelali"},"modified":"2020-05-29T07:43:02","modified_gmt":"2020-05-29T05:43:02","slug":"kak-my-perezhili-rezkij-rost-nagruzki-x10-na-udalenke-i-kakie-vyvody-sdelali","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-perezhili-rezkij-rost-nagruzki-x10-na-udalenke-i-kakie-vyvody-sdelali","title":{"rendered":"How we coped with a tenfold increase in workload while working remotely and what conclusions we drew","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Hello, Habr! We have been living in a very interesting situation for the last couple of months, and I would like to share our story of scaling the infrastructure. During this time, SberMarket quadrupled its orders and launched the service in 17 new cities. The explosive growth in demand for grocery delivery required us to scale our infrastructure. Read about the most interesting and useful insights below.<\/p>\n

\"How
\n<\/a><\/noindex>
\nMy name is Dima Bobylev, and I am the technical director of SberMarket. Since this is the first post in our blog, I would like to say a few words about myself and the company. Last autumn, I participated in the Young Leaders of the Runet competition. For the contest, I wrote a short story<\/a><\/noindex> about how we at SberMarket see our internal culture and approach to service development. Although I didn\u2019t win the contest, I was able to clarify the main principles of developing an IT ecosystem for myself. <\/p>\n

When managing a team, it is important to understand and find a balance between what the business needs and the needs of each individual developer. Currently, SberMarket is growing 13 times year-over-year, which impacts the product, necessitating constant increases in the volume and pace of development. Despite this, we allocate sufficient time for developers to conduct preliminary analysis and to write high-quality code. This structured approach not only aids in creating a working product but also in its further scaling and development. As a result of this growth, SberMarket has already become a leader among grocery delivery services: we deliver around 18,000 orders daily, whereas there were about 3,500 at the beginning of February.<\/p>\n

\"How
\nOnce a client asked a SberMarket courier to deliver groceries contactlessly\u2014directly onto their balcony.<\/i><\/p>\n

But let's get to the specifics. Over the past few months, we have been actively scaling our company's infrastructure. This need was driven by both external and internal factors. Alongside our growing client base, the number of connected stores increased from 90 at the beginning of the year to over 200 by mid-May. Of course, we prepared, reserving our main infrastructure and anticipating the possibility of both vertical and horizontal scaling of all virtual machines hosted in Yandex Cloud. However, practice showed: 'Everything that can go wrong will go wrong.' Today, I want to share some of the most interesting situations that arose during these weeks. I hope our experience proves useful to you.<\/p>\n

Slave is fully battle-ready<\/h3>\n

\nEven before the pandemic began, we faced an increase in requests for our backend servers. The trend of ordering products with home delivery began to pick up steam, and with the introduction of the first self-isolation measures due to COVID-19, the load dramatically increased before our eyes throughout the day. There was a need to quickly offload the master servers of the main database and shift some read requests to the replica servers (slave).<\/p>\n

We had been preparing for this step in advance, and two slave servers had already been launched for such maneuvers. They mainly handled batch tasks for generating informational feeds to exchange data with partners. These processes created extra load and had justifiably been moved 'out of scope' a couple of months earlier.\u00a0<\/p>\n

Since replication was occurring on the Slave, we adhered to the concept that applications could only interact with them in read-only mode. The Disaster Recovery Plan stipulated that in the event of a disaster, we could simply mount the Slave in place of the Master and redirect all read and write requests to the Slave. However, we also wanted to use the replicas for the analytics department's needs, so the servers were not completely switched to read-only status, and each host had its set of users, with some having write permissions to save intermediate calculation results.<\/p>\n

Up to a certain load level, we had enough capacity in both writing and reading while processing HTTP requests. In mid-March, just when Sbermarket decided to fully switch to remote work, we experienced a surge in RPS. More of our clients were isolating or working from home, which reflected on our load metrics.<\/p>\n

The performance of the 'master' was no longer sufficient, so we began offloading some of the heaviest read requests to the replica. To transparently route write requests to the master and read requests to the slave, we used the Ruby gem \"Octopus<\/a><\/noindex>\". We created a special user with the _readonly postfix without write permissions. However, due to a configuration error on one of the hosts, some write requests were sent to the slave server on behalf of a user who had the appropriate permissions.<\/p>\n

The problem did not manifest immediately, as the increased load caused the slaves to lag behind. Data inconsistencies were discovered in the morning when, after overnight imports, the slaves had not 'caught up' with the master. We attributed this to high loads on the service itself and the import related to the launch of new stores. However, serving data with several hours of delay was unacceptable, and we switched processes to the second analytical slave because it had more resources and was not burdened with read requests (which we used to explain to ourselves the absence of replication lag).higher level of isolation, as if one controller is broken, the problem is confined to that specific context).<\/strong>When we figured out the reasons for the 'spreading' of the main slave, the analytical one had already gone down for the same reason. Despite having two additional servers that we planned to shift load to in case the master failed, a regrettable error turned out that at the critical moment there were none available.<\/p>\n

However, since we not only performed a database dump (the restoration at that moment took about 5 hours) but also a snapshot of the master server, we managed to start the replica within 2 hours. However, after that, we faced a lengthy replication log replay (because the process runs in a single-threaded mode, but that's a completely different story).<\/p>\n

However, since we were not only performing a database dump (the restore at that time took about 5 hours) but also taking a snapshot of the master server, we were able to launch the replica within 2 hours. However, after that, we faced the prolonged task of applying the replication log (because the process runs in single-threaded mode, but that's a different story altogether).<\/p>\n

Output:<\/strong> After such an incident, it became clear that we needed to abandon the practice of limiting user access and declare the entire server as readonly. With this approach, we can be confident that replicas will be available at critical moments.<\/p><\/blockquote>\n

<\/p>\n

Optimizing even a single heavy query can \"bring the database back to life.\"<\/h3>\n

\nAlthough we constantly update the catalog on the site, the queries we directed to Slave servers had a slight delay compared to the Master. The time it took to identify and resolve the issue of slaves \"suddenly dropping off\" exceeded the \"psychological barrier\" (during which price updates could occur, and customers would see outdated data), and we had to redirect all queries to the main database server. As a result, the site was slow... but at least it was operational. And while the Slave was recovering, we had no choice but to optimize.\u00a0<\/p>\n

While the Slave servers were recovering, the minutes dragged on slowly, the Master remained overloaded, and we focused all our efforts on optimizing active tasks according to the \"Pareto Principle\": we selected the top queries that caused most of the load and began tuning. This was done on the fly.<\/p>\n

An interesting effect was that a MySQL overloaded to the brim responded to even slight process improvements. Optimizing a couple of queries that accounted for just 5% of the total load already showed a noticeable reduction in CPU usage. As a result, we managed to provide an acceptable resource reserve for the Master to work with the database and gain the necessary time to restore replicas.\u00a0<\/p>\n

Output:<\/strong> Even a small optimization allows us to \"survive\" during overload for several hours. This was just enough time for us to restore the servers with replicas. By the way, we will discuss the technical side of query optimization in one of the upcoming posts. So, subscribe to our blog if this could be useful to you.<\/p><\/blockquote>\n

<\/p>\n

Organize monitoring of partner service performance.<\/h3>\n

\nWe handle client order processing, which is why our services constantly interact with external APIs\u2014these are gateways for sending SMS, payment platforms, routing systems, geocoders, the FNS service, and many other systems. And when the load began to grow rapidly, we started to hit the limitations of our partner services' APIs that we hadn't even considered before.<\/p>\n

An unexpected exceedance of partner service quotas can lead to downtime for your own systems. Many APIs block clients that exceed limits, and in some cases, an excess of requests can overload a partner's production environment.\u00a0<\/p>\n

For example, during the increase in delivery volumes, the accompanying services struggled with tasks such as distribution and route determination. As a result, orders were made, but the service responsible for creating routes was non-functional. It must be said that our logistics team did the near impossible under these conditions, and the clear interaction of the team helped to compensate for temporary service failures. However, such a volume of requests cannot be processed manually on a constant basis, and after a while, we would face an unacceptable gap between orders and their fulfillment.\u00a0<\/p>\n

A series of organizational measures were taken, and the coordinated work of the team helped buy time while we negotiated new terms and awaited service upgrades from some partners. There are other APIs that boast high endurance and outrageous rates in case of high traffic. For example, initially, we used a well-known mapping API for determining delivery point addresses. But after a month, we received a hefty bill of almost 2 million rubles. After that, we decided to quickly replace it. I won\u2019t do any advertising, but I will say that our expenses have significantly decreased.
\n\"How<\/p>\n

Output: <\/strong>It is essential to continuously monitor the operating conditions of all partner services and keep them in mind. Even if today it seems that they provide you with a 'large reserve,' that doesn't mean they won't become an obstacle to growth tomorrow. And of course, it's better to negotiate financial terms for increased service requests in advance.\u00a0<\/p><\/blockquote>\n

<\/p>\n

Sometimes it turns out that \"more gold is needed<\/a><\/noindex>\" (c) does not help<\/h3>\n

\nWe are accustomed to encountering bottlenecks in the main database or on application servers, but during scaling, issues can arise in unexpected places. For full-text search on the site, we use the Apache Solr engine. As the load increased, we noticed a decrease in response time, and the server's CPU load reached 100%. What could be simpler \u2014 we can allocate more resources to the Solr container.<\/p>\n

Instead of the expected performance boost, the server simply \u2018died\u2019. It immediately loaded to 100% and responded even more slowly. Initially, we had 2 cores and 2 GB of RAM. We decided to do what usually helps \u2014 we gave the server 8 cores and 32 GB. Everything became much worse (how and why exactly \u2014 we will explain in a separate post).\u00a0<\/p>\n

In just a few days, we delved into the intricacies of this issue and achieved optimal performance with 8 cores and 32 GB. This configuration still allows us to continue increasing the load, which is crucial because growth is not only in clients but also in the number of connected stores \u2014 their numbers doubled in just 2 months.\u00a0<\/p>\n

Output: <\/strong>Standard methods like \u2018adding more hardware\u2019 do not always work. Therefore, when scaling any service, it is essential to understand how it utilizes resources and to test its performance under new conditions in advance.\u00a0\n<\/p><\/blockquote>\n

<\/p>\n

Stateless \u2014 the key to simple horizontal scaling<\/h3>\n

\nOverall, our team adheres to the well-known approach: services should not have internal state (stateless) and must be independent of the execution environment. This allowed us to handle increased loads through simple horizontal scaling. However, we had one exception service \u2013 the handler for long-running background tasks. It was responsible for sending emails and SMS, processing events, generating feeds, importing prices and stock levels, and handling images. As it turned out, it depended on a local file storage and existed in a single instance.\u00a0<\/p>\n

As the number of tasks in the handler's queue grew (which naturally occurred with the increase in orders), the performance of the host where the handler and file storage were located became a limiting factor. Consequently, updates to the assortment and prices, notifications to users, and many other critical functions became stuck in the queue. The Ops team promptly migrated the file storage to an S3-like network storage, allowing us to deploy several powerful machines to scale the background tasks handler.<\/p>\n

Output: <\/strong>The Stateless rule must be adhered to for all components without exception, even if it seems that \"we definitely won't hit a limit here.\" It's better to spend some time organizing the work of all systems correctly than to rush to rewrite code and fix a service that is experiencing overload later.<\/p><\/blockquote>\n

<\/p>\n

7 Principles for Intensive Growth<\/h2>\n

\nDespite the availability of additional capacity, we stumbled upon several pitfalls during our growth. During this time, the number of orders increased more than fourfold. We are currently delivering over 17,000 orders daily across 62 cities and plan to expand our reach even further \u2014 in the first half of 2020, we expect to launch the service throughout Russia. To cope with the growing load, considering the lessons learned, we established 7 core principles for operating under constant growth:<\/p>\n

    \n
  1. Incident Management<\/strong>. We created a board in Jira, where each incident is recorded as a ticket. This will help effectively prioritize and execute tasks related to the incident. After all, it\u2019s not scary to make mistakes \u2014 what\u2019s frightening is making the same mistake twice. In cases where incidents recur before we can fix the root cause, an action plan should be ready, because during high load, it's crucial to respond swiftly.<\/li>\n
  2. Monitoring <\/strong>is required for all infrastructure elements without exception. It is thanks to this that we were able to forecast load growth and correctly identify 'bottlenecks' for prioritization of elimination. Most likely, under high load, everything you didn't even think about will break down or start to lag. Therefore, it is best to create new alerts immediately after the first incidents occur, to monitor and anticipate them.<\/li>\n
  3. Proper alerts<\/strong> are essential during a rapid increase in load. Firstly, they must indicate exactly what has failed. Secondly, there shouldn't be too many alerts, because an abundance of non-critical alerts leads to all notifications being ignored completely.<\/li>\n
  4. Applications must be stateless. <\/strong>We have confirmed that there should be no exceptions to this rule. Complete independence from the execution environment is required. For this, you can store shared data in a database or, for example, directly in S3. Better yet, follow the rules. https:\/\/12factor.net<\/a><\/noindex>During a sharp increase in demand, there simply isn't time to optimize code, and you'll have to handle the load by directly increasing computational resources and horizontal scaling.<\/li>\n
  5. Quotas and performance of external services. <\/strong>During rapid growth, problems can arise not only in your infrastructure but also in external services. The most frustrating thing is when this happens not due to a failure, but because of reaching quotas or limits. So external services must scale as well as you do.\u00a0<\/li>\n
  6. Separate processes and queues. <\/strong>This is very helpful when there is a bottleneck at one of the gateways. We would not have encountered delays in data transmission if the filled SMS sending queues had not interfered with the exchange of notifications between information systems. Moreover, it would have been easier to increase the number of workers if they operated separately.<\/li>\n
  7. Financial realities.<\/strong> When there is an explosive growth in data streams, there is no time to think about tariffs and subscriptions. But they must be kept in mind, especially if you are a small company. A large bill can be issued by the owner of any API, as well as your hosting provider. So it's important to read contracts carefully.<\/li>\n<\/ol>\n

    <\/p>\n

    Conclusion<\/h2>\n

    \nDespite some losses, we survived that phase, and today we strive to adhere to all the principles we found. Each machine has the potential for an easy 4x performance increase to handle unexpected challenges.\u00a0<\/p>\n

    In upcoming posts, we will share our experiences investigating performance dips in Apache Solr, discuss query optimization, and explain how interaction with the tax authority helps the company save money. Subscribe to our blog so you don't miss anything, and let us know in the comments if you've faced similar issues during traffic growth.<\/p>\n

    \"How<\/p>\n

    Only registered users can participate in the survey. Please log in<\/a><\/noindex>, please.<\/p>\n

    Have you ever experienced slowdowns or service crashes during sudden load increases due to:<\/h2>\n