This article compiles some general templates to help engineers work with large-scale services that receive millions of user requests.

Based on the author's experience, this is not an exhaustive list, but definitely effective advice. So, let's get started.

Translated with support from Mail.ru Cloud Solutions.

Basic Level

The measures listed below are relatively simple to implement but yield high returns. If you haven't taken them before, you will be surprised by the significant improvements.

Infrastructure as Code

The first part of the advice is to implement infrastructure as code. This means you should have a software method for deploying all infrastructure. It sounds complicated, but we are actually talking about the following code:

Deploying 100 virtual machines

with Ubuntu

2 GB RAM on each

they will have the following code

with such parameters

You can track changes in the infrastructure and quickly revert to them using a version control system.

The modernist in me says that you can use Kubernetes/Docker to do all of the above, and he's right.

Additionally, automation can be ensured with Chef, Puppet, or Terraform.

Continuous Integration and Delivery

To create a scalable service, it is important to have a build and test pipeline for each pull request. Even if the test is the simplest, it at least ensures that the code you are deploying compiles.

At this stage, you are answering the question: Will my build compile and pass tests, is it valid? This may seem like a low bar, but it solves many problems.

There is nothing more beautiful than seeing those checkmarks

For this technology, you might consider Github, CircleCI, or Jenkins.

Load Balancers

So, we want to deploy a load balancer to redirect traffic and ensure even load across all nodes or the service's operation in case of failure:

A load balancer typically helps distribute traffic well. Best practice is redundant balancing, so you have no single points of failure.

Load balancers are usually configured in the cloud you are using.

RayID, correlation ID, or UUID for requests

Have you ever encountered an error in an application with a message like this: “Something went wrong. Please save this ID and send it to our support team.”?

A unique identifier, correlation ID, RayID, or any of its variants is a unique identifier that allows tracking a request throughout its lifecycle. This allows you to trace the entire request path in the logs.

The user makes a request to system A, then A connects to B, which connects to C, saves in X, and then the request returns to A.

If you were to remotely connect to virtual machines and attempt to trace the request path (and manually correlate which calls are made), you would go insane. Having a unique identifier greatly simplifies this process. It's one of the simplest things you can do to save time as the service grows.

Intermediate level

Here, the advice becomes more complex than previous ones, but the right tools make the job easier, ensuring a return on investment even for small and medium-sized businesses.

Centralized logging

Congratulations! You’ve deployed 100 virtual machines. The next day, the CEO comes in and complains about an error encountered during service testing. He provides the relevant identifier we discussed earlier, but you have to sift through the logs of 100 machines to find the one that caused the failure. And it has to be located before tomorrow's presentation.

While this sounds like a fun adventure, it's better to ensure that you have the ability to search across all logs from one place. I solved the centralized logging problem using the built-in functionality of the ELK stack: it supports log collection with search capabilities. This will really help tackle the issue of finding a specific log. As a bonus, you can create charts and other fun things.

ELK stack functionality

Monitoring agents

Now that your service is up and running, you need to ensure it operates smoothly. The best way to do this is to run several agents, which work in parallel and check that it is operating and performing basic operations.

At this stage, you verify that The launched build performs well and operates normally..

For small and medium projects, I recommend Postman for monitoring and documenting APIs. Overall, you just need to ensure you have a way to know when a failure occurs and receive timely notifications.

Auto-scaling based on load.

It's very simple. If you have a virtual machine handling requests and it approaches 80% memory usage, you can either increase its resources or add more virtual machines to the cluster. Automating these operations is great for elastic power scaling under load. But you should always be cautious about how much money you spend and set reasonable limits.

In most cloud services, you can configure auto-scaling using more servers or more powerful servers.

Experimentation system.

A good way to safely deploy updates is to test something with 1% of users for an hour. You've probably seen such mechanisms in action. For example, Facebook shows parts of its audience a different color or changes the font size to see how users perceive the changes. This is called A/B testing.

Even the release of a new feature can be launched as an experiment, and then you can determine how to deploy it. You also have the ability to 'roll back' or change the configuration on the fly considering a feature that causes degradation of your service.

Advanced level.

Here are tips that are quite difficult to implement. You will probably need a bit more resources, so it will be challenging for a small or medium-sized company to manage this.

Blue-green deployments.

This is what I call the 'Erlang' way of deployment. Erlang became widely used when telephone companies emerged. Software switches were used for routing phone calls. The main task of the software for these switches was to avoid dropping calls during system updates. Erlang has a great way of loading a new module without crashing the previous one.

This step depends on the presence of a load balancer. Let's say you have version N of your software, and then you want to deploy version N+1.

You could simply stop the service and deploy the next version at a time that is convenient for your users, resulting in some downtime. But let’s assume that you have really strict SLA conditions. Thus, a 99.99% SLA means you can go offline only for 52 minutes a year.

If you genuinely want to achieve such metrics, you need two deployments simultaneously:

the current one (N);
the next version (N+1).

You instruct the load balancer to redirect a percentage of traffic to the new version (N+1), while you actively monitor for regressions.

Here we have a green deployment N that is functioning normally. We are trying to transition to the next version of this deployment.

First, we send a truly small test to see if our deployment N+1 works with a limited amount of traffic:

Finally, we have a set of automated checks that we ultimately run until our deployment is complete. If you are very, very careful, you can also keep your deployment N indefinitely for quick rollback in case of a bad regression:

If you want to take it to an even more advanced level, let everything in the blue-green deployment run automatically.

Anomaly detection and automatic mitigation

Given that you have centralized logging and good log collection, you can already set higher goals. For example, proactively forecast failures. Functions are tracked on monitors and in the logs, and various graphs are built — you can predict in advance what might go wrong:

With anomaly detection, you begin to study some hints provided by the service. For example, a spike in CPU load may indicate that a hard drive is failing, while a spike in the number of requests means scaling is necessary. Such statistics can make the service proactive.

By receiving such analytical data, you can scale in any dimension, proactively and reactively modifying the characteristics of machines, databases, connections, and other resources.

That's it!

This priority list will save you many issues when you scale your cloud service.

The author of the original article invites readers to leave their comments and make edits. The article is distributed as open source; the author accepts pull requests on Github. on Github..