Hello, Habr! I'm Artem Karamyshev, the head of the system administration team . Over the past year, we have launched many new products. We aimed to ensure that our API services are easily scalable, fault-tolerant, and ready for rapid user load growth. Our platform is built on OpenStack, and I want to discuss the fault-tolerance issues we had to address to achieve a reliable system. I believe this will be interesting for those developing products on OpenStack as well.
The overall fault tolerance of the platform is composed of the resilience of its components. So we will gradually go through all the levels where we identified risks and addressed them.
The video version of this story, based on a report at the Uptime Day 4 conference organized by , can be viewed .
The fault tolerance of the physical architecture
The public part of the MCS cloud is currently based in two Tier III data centers, with dark fiber reserved at the physical level through different routes, providing 200 Gbps bandwidth. The Tier III level ensures the necessary fault tolerance for the physical infrastructure.
Dark fiber is reserved at both the physical and logical levels. The process of channel reservation has been iterative, with challenges arising, and we continuously improve the connectivity between data centers.
For example, not long ago, during work in a chamber near one of the data centers, an excavator accidentally pierced a pipe, which contained both the primary and backup optical cables. Our fault-tolerant communication channel with the data center became vulnerable at one point, in the chamber. Consequently, we lost part of our infrastructure. We learned from this, took several actions, including laying additional fiber optics through the neighboring chamber.
Data centers have points of presence for communication providers to whom we transmit our prefixes via BGP. For each network direction, the best metric is selected, allowing us to provide different clients with the highest quality of connection. If the connection through one provider is interrupted, we reconfigure our routing through available providers.
In case of a provider malfunction, we automatically switch to the next one. In the event of a failure at one of the data centers, we have a mirrored copy of our services at a second data center, which takes on the entire load.
Fault tolerance of the physical infrastructure
What we use for application-level fault tolerance
Our service is built on a number of open-source components.
ExaBGP — a service that implements a range of functions using a dynamic routing protocol based on BGP. We actively use it to announce our white IP addresses through which users access the API.
HAProxy — a high-load balancer that allows for the configuration of very flexible traffic balancing rules at different layers of the OSI model. We use it for balancing in front of all services: databases, message brokers, API services, web services, our internal projects — everything stands behind HAProxy.
API application — a web application written in Python, through which users manage their infrastructure and services.
Worker application (hereafter simply worker) — in OpenStack services, this is an infrastructure daemon that allows the transmission of API commands to the infrastructure. For example, disk creation occurs specifically in the worker, while the creation request happens in the API application.
Standard architecture of OpenStack Application
Most services developed for OpenStack try to follow a unified paradigm. A service usually consists of two parts: the API and workers (backend executors). Generally, the API is a WSGI application written in Python that runs either as a standalone process (daemon) or via an existing web server like Nginx or Apache. The API processes user requests and sends further instructions to the worker application for execution. This transfer happens via a message broker, typically RabbitMQ, with others being poorly supported. Once messages reach the broker, they are processed by the workers, which return a response if necessary.
This paradigm implies isolated common points of failure: RabbitMQ and the database. However, RabbitMQ is isolated within a single service and can conceptually be unique to each service. Therefore, at MCS, we maximize the separation of these services, creating a separate database and a separate RabbitMQ for each individual project. This approach is beneficial because in the event of a failure at some vulnerable points, not the entire service breaks down but only a part of it.
The number of worker applications is not limited, so the API can easily scale horizontally behind load balancers to increase performance and fault tolerance.
In some services, coordination within the service is necessary—when complex sequential operations occur between the API and the workers. In this case, a central coordination system, a clustered system like Redis, Memcache, etcd, is used, allowing one worker to inform another that a task is assigned to it ("please don’t take it"). We use etcd. Generally, workers actively communicate with the database, writing to and reading from it. We utilize MariaDB, which is hosted in a multi-master cluster.
Such a classic single service is organized in a manner commonly accepted for OpenStack. It can be viewed as a closed system, for which scaling and fault tolerance methods are quite obvious. For instance, to ensure fault tolerance for the API, it is sufficient to place a load balancer in front of it. Scaling the workers is achieved by increasing their quantity.
A weak point in the entire setup is RabbitMQ and MariaDB. Their architecture deserves a separate article. In this article, I want to focus on the fault tolerance of the API.
OpenStack Application Architecture. Load balancing and fault tolerance of the cloud platform.
Making the HAProxy load balancer fault-tolerant using ExaBGP.
To ensure our APIs are scalable, fast, and fault-tolerant, we put a load balancer in front of them. We chose HAProxy. In my opinion, it has all the necessary characteristics for our task: multi-layer OSI load balancing, a management interface, flexibility and scalability, a wide range of balancing methods, and session table support.
The first problem that needed to be solved was the fault tolerance of the load balancer itself. Simply deploying a load balancer also creates a point of failure: if the load balancer fails, the service goes down. To avoid this, we used HAProxy together with ExaBGP.
ExaBGP enables service status checking. We used this mechanism to monitor HAProxy's health and, in case of issues, to take the HAProxy service offline from BGP.
ExaBGP+HAProxy Diagram
- We install the necessary software, ExaBGP and HAProxy, on three servers.
- On each server, we create a loopback interface.
- On all three servers, we assign the same public IP address to this interface.
- The public IP address is announced over the internet via ExaBGP.
Fault tolerance is achieved by announcing the same IP address from all three servers. From a networking perspective, the same address is available from three different next hops. The router sees three identical routes and selects the most prioritized one based on its own metrics (which is usually the same option), directing traffic only to one of the servers.
In the event of problems with HAProxy or if a server goes down, ExaBGP stops announcing the route, and the traffic smoothly switches to another server.
Thus, we achieved fault tolerance for the load balancer.
Fault tolerance of HAProxy load balancers.
The scheme turned out to be imperfect: we learned how to back up HAProxy, but we didn't learn how to distribute the load within the services. Therefore, we slightly expanded this scheme by moving to load balancing among multiple public IP addresses.
DNS-Based Load Balancing with BGP
The issue of load balancing before our HAProxy remains unresolved. However, it can be solved quite simply, as we have done on our end.
To balance three servers, you will need 3 public IP addresses and the good old DNS. Each of these addresses is defined on the loopback interface of each HAProxy and announced on the internet.
In OpenStack, a service catalog is used to manage resources, which specifies the API endpoint of various services. In this catalog, we write the domain name — public.infra.mail.ru, which resolves through DNS to three different IP addresses. As a result, we achieve load distribution among the three addresses via DNS.
However, since we do not manage the server selection priorities when announcing public IP addresses, it is not load balancing yet. Typically, only one server will be selected based on the seniority of the IP address, while the other two will remain idle, as no metrics are specified in BGP.
We began announcing routes through ExaBGP with different metrics. Each load balancer announces all three public IP addresses, but one of them—primary for this load balancer—is announced with the lowest metric. So while all three load balancers are operational, requests to the first IP address go to the first load balancer, requests to the second go to the second, and requests to the third go to the third.
What happens when one of the load balancers fails? Upon the failure of any load balancer, its primary address is still announced by the other two, and traffic is redistributed between them. Thus, we provide the user with several IP addresses through DNS. By balancing via DNS and using different metrics, we achieve even load distribution across all three load balancers while maintaining fault tolerance.
HAProxy Load Balancing with DNS + BGP
Interaction between ExaBGP and HAProxy
Thus, we have implemented fault tolerance in case of server failure based on route announcement cessation. However, HAProxy can also go down for reasons other than server failure: administrative errors, service malfunctions. We aim to remove a broken load balancer from under the load in these cases, which requires another mechanism.
Therefore, expanding on the previous scheme, we implemented a heartbeat between ExaBGP and HAProxy. This is a software implementation of the interaction between ExaBGP and HAProxy, where ExaBGP uses custom scripts to check the status of applications.
To achieve this, it is necessary to configure a health checker in the ExaBGP config that can verify HAProxy's status. In our case, we set up a health backend in HAProxy, and on the ExaBGP side, we perform a simple GET request. If the announcement stops occurring, it is likely that HAProxy is not functioning, and it should not be announced.
HAProxy Health Check
HAProxy Peers: session synchronization
The next step was to synchronize sessions. It is challenging to maintain client session information when working with distributed load balancers. However, HAProxy is one of the few load balancers that can do this thanks to its Peers functionality — the ability to share session tables between different HAProxy processes.
There are different load balancing methods: simple ones, such as , and advanced ones, where the client session is remembered, ensuring they hit the same server as before each time. We wanted to implement the latter.
In HAProxy, stick-tables are used to preserve client sessions. They save the original client IP address, the selected target address (backend), and some auxiliary information. Typically, stick-tables are used to retain the source-IP + destination-IP pair, which is especially useful for applications that cannot pass user session context when switching to another load balancer, for example — in RoundRobin balancing mode.
If we can teach the stick-table to move between different HAProxy processes (between which the load balancing occurs), our load balancers will be able to work with a single pool of stick-tables. This will enable seamless client network switching in the event of one of the load balancers going down, allowing client sessions to continue on the same backends that were previously selected.
For proper operation, the issue of the source IP address of the load balancer from which the session was established must be resolved. In our case, this is a dynamic address on the loopback interface.
The proper functioning of peers is achieved only under certain conditions. That is, the TCP timeouts must be sufficiently long, or the switching must be fast enough so that the TCP session doesn't disconnect. Nevertheless, this allows for seamless switching.
In our IaaS, we have a service built on the same technology. It's , called Octavia. It is based on two HAProxy processes and originally includes support for peers. In this service, they have proven to be very effective.
The image schematically depicts the movement of peers tables between three HAProxy instances, and a configuration is suggested on how to set this up:
HAProxy Peers (session synchronization)
If you implement a similar scheme, its operation needs to be tested thoroughly. It is not guaranteed that it will work in the same way in 100% of cases. However, at least you won't lose stick tables when you need to remember the client's source IP.
Limiting the number of simultaneous requests from the same client
Any services that are publicly accessible, including our APIs, can be susceptible to request floods. Their causes can vary greatly, from user errors to targeted attacks. We periodically experience DDoS attacks based on IP addresses. Clients often make mistakes in their scripts, effectively causing mini-DDoS attacks against us.
In one way or another, it is necessary to provide additional protection. An obvious solution becomes to limit the number of requests to the API and not waste CPU time processing malicious requests.
To implement such limitations, we use rate limits organized on the basis of HAProxy, with the help of the same stick tables. The limits are easy to configure and allow for restrictions on the number of requests to the API from users. The algorithm remembers the source IP from which requests are made and restricts the number of simultaneous requests from one user. Naturally, we calculated the average load profile on the API for each service and set the limit to approximately 10 times this value. We continue to closely monitor the situation and keep our finger on the pulse.
What does this look like in practice? We have clients who constantly use our APIs for auto-scaling. They create about two to three hundred virtual machines in the morning and remove them by the evening. To create a virtual machine in OpenStack, along with PaaS services, it requires at least 1000 API requests, as interactions between services also occur via APIs.
Such task transfers place a significant load. We assessed this load, collected daily peaks, increased them tenfold, and that became our rate limit. We keep our finger on the pulse. We often see bots and scanners trying to check if we have any CGA scripts they can run, and we actively cut them off.
How to update the codebase without users noticing
We implement fault tolerance at the level of code deployment processes as well. Failures can occur during rollouts, but their impact on service availability can be minimized.
We constantly update our services and must ensure the process of updating the codebase has no effect on users. This challenge has been solved by utilizing the management capabilities of HAProxy and implementing Graceful Shutdown in our services.
To solve this problem, it was necessary to ensure controlling the load balancer and the 'proper' shutdown of services:
- In the case of HAProxy, control is carried out through the stats file, which is essentially a socket and defined in the HAProxy config. Commands can be sent to it via stdio. However, our main configuration control tool is Ansible, so it has a built-in module for managing HAProxy, which we actively use.
- Most of our API and Engine services support graceful shutdown technologies: when shutting down, they wait for the current task to fully complete, whether it's an HTTP request or some background task. The same occurs with the worker. It knows all the tasks it is performing and finishes when everything is successfully completed.
Thanks to these two aspects, the safe algorithm for our deployments looks as follows.
- The developer compiles a new code package (for us, this is RPM), tests it in the dev environment, tests it in stage, and leaves it in the stage repository.
- The developer sets the task for deployment with a detailed description of the "artifacts": the version of the new package, a description of the new functionality, and any other details about the deployment if needed.
- The system administrator begins the update. He starts the Ansible playbook, which in turn does the following:
- It takes the package from the stage repository and updates the package version in the production repository.
- It compiles a list of backends of the service being updated.
- It shuts down the first backend service in HAProxy and waits for its processes to complete. Thanks to graceful shutdown, we are confident that all current client requests will finish successfully.
- After the API, workers, and HAProxy are completely stopped, the code is updated.
- Ansible starts the services.
- For each service, it calls specific "handles" that perform unit testing based on a set of predefined key tests. A basic check of the new code occurs.
- If no errors were found in the previous step, the backend is activated.
- We move on to the next backend.
- After all backends are updated, functional tests are run. If they're insufficient, the developer reviews any new functionality he added.
At this point, the deployment is complete.
Service update cycle
This scheme would not be feasible if we did not have one rule. We support both the old and new versions in production simultaneously. During the software development phase, it is ensured that even if there are changes in the service database, they will not break the previous code. As a result, a gradual update of the codebase occurs.
Conclusion
Sharing my thoughts on fault-tolerant WEB architecture, I want to emphasize its key points once again:
- physical fault tolerance;
- network fault tolerance (load balancers, BGP);
- fault tolerance of the software in use and being developed.
Wishing everyone stable uptime!
Source: habr.com
