In the process of moving from a monolithic application to a microservice architecture, we face new challenges.
In a monolithic application, it's usually fairly easy to determine which part of the system has an error. Most likely, the problem is in the code of the monolith itself, or in the database. But when we start looking for a problem in the microservice architecture, everything is not so obvious. You need to find the entire path that the request went from start to finish, select it from hundreds of microservices. Moreover, many of them also have their own storages, in which both logical errors and problems with performance and fault tolerance can also occur.
I have been looking for a tool that would help deal with such problems for a long time (I wrote about it on HabrΓ©: , ), but ended up making my own open source solution. In the article, I talk about the benefits of the service mesh approach and share a new tool for its implementation.
Distributed tracing is a common solution to the problem of finding bugs in distributed systems. But what if the system has not yet implemented such an approach to collecting information about network interactions, or, worse, in part of the system it already works properly, but in part it does not exist, since it was not added to the old services? To determine the exact root cause of a problem, it is necessary to have a complete picture of what is happening in the system. It is especially important to understand which microservices are involved in the main critical business paths.
Here, the service mesh approach can come to our aid, which will take care of all the machinery for collecting network information at a level lower than the services themselves work. This approach allows us to capture all traffic and analyze it on the fly. Moreover, applications should not even know anything about it.
Service mesh approach
The main idea of ββthe service mesh approach is to add another infrastructure layer above the network, which will allow us to do any thing with inter-service interaction. Most implementations work like this: an additional sidecar container with a transparent proxy is added to each microservice, through which all incoming and outgoing traffic of the service passes. And this is the place where we can do client balancing, apply security policies, impose limits on the number of requests, and collect important information on the interaction of services in production.
Solutions
There are already several implementations of this approach: ΠΈ . They provide a lot of options out of the box. But at the same time comes a big overhead on resources. Moreover, the larger the cluster in which such a system operates, the more resources will be required to maintain the new infrastructure. At Avito, we operate kubernetes clusters that host thousands of service instances (and their number continues to grow rapidly). In the current implementation, Istio consumes ~300Mb of RAM per service instance. Due to the large number of possibilities, transparent balancing also affects the overall response time of services (up to 10ms).
As a result, we looked at exactly what features we need right now, and decided that the main reason why we started implementing such solutions was the ability to collect tracing information from the entire system transparently. We also wanted to have control over the interaction of services and do various manipulations with the headers that are passed between services.
As a result, we came to our decision:β .
Netramesh
is a lightweight service mesh solution with infinite scalability regardless of the number of services in the system.
The main goals of the new solution were a small resource overhead and high performance. From the main features, we wanted to immediately be able to transparently send tracing spans to our Jaeger system.
Today, most cloud solutions are implemented in Golang. And, of course, there are reasons for this. It is convenient and quite simple to write network applications in Golang that work asynchronously with I / O and scale as necessary to the cores. And, which is also very important, the performance is sufficient to solve this problem. Therefore, we also chose Golang.
Performance
We have focused our efforts on achieving maximum performance. For a solution that is deployed next to each instance of the service, a small consumption of RAM and CPU time is required. And, of course, the response delay should also be small.
Let's see what the results are.
RAM
Netramesh consumes ~10Mb without traffic and 50Mb maximum with a load of up to 10000 RPS per instance.
Istio envoy proxy always consumes ~300Mb in our clusters with thousands of instances. This does not allow scaling it to the entire cluster.
With Netramesh, we have reduced memory consumption by ~10 times.
CPU
CPU usage is relatively equal under load. It depends on the number of requests per unit of time to the sidecar. Values ββat 3000 requests per second at the peak:
There is another important point: Netramesh - a solution without a control plane and without load does not consume processor time. With Istio sidecars always update service endpoints. As a result, we can see the following picture without load:
We use HTTP/1 to communicate between services. The increase in Istio response time when proxying through envoy was up to 5-10ms, which is quite a lot for services that are ready to respond in a millisecond. With Netramesh this time has been reduced to 0.5-2ms.
Scalability
The small amount of resources spent by each proxy makes it possible to place it next to each service. Netramesh was deliberately created without a component's control plane to simply keep each sidecar lightweight. Often in service mesh solutions, the control plane distributes service discovery information to each sidecar. Along with it comes information about timeouts, balancing settings. All this allows you to do a lot of useful things, but, unfortunately, it bloats sidecars in size.
Service discovery
Netramesh does not add any additional mechanisms for service discovery. All traffic is transparently proxied through netra sidecar.
Netramesh supports the HTTP/1 application protocol. A configurable list of ports is used to define it. Typically, there are several ports on the system through which HTTP communication takes place. For example, we use 80, 8890, 8080 for the interaction of services and external requests. In this case, they can be set using the environment variable NETRA_HTTP_PORTS.
If you use Kubernetes as an orchestrator and its Service entity mechanism for intra-cluster communication between services, then the mechanism remains exactly the same. First, the microservice obtains a service IP address using kube-dns and opens a new connection to it. This connection is first established with the local netra-sidecar and all TCP packets initially arrive in netra. Next, netra-sidecar establishes a connection to the original destination. NAT on the pod IP on the node remains exactly the same as without netra.
Distributed tracing and context skipping
Netramesh provides the functionality needed to send tracing spans about HTTP interactions. Netra-sidecar parse the HTTP protocol, measure request delays, get the necessary information from HTTP headers. Ultimately, we get all the traces in a single Jaeger system. For fine tuning, you can also use the environment variables provided by the official library .
But there is a problem. Until the services generate and pass a special uber header, we will not see connected tracing spans in the system. And this is what we need to quickly find the cause of problems. Here Netramesh has a solution again. Proxies read HTTP headers and, if they don't have an uber trace id, generate one. Netramesh also stores information about incoming and outgoing requests in the sidecar and matches them by enriching them with the necessary outgoing request headers. All you need to do in services is to forward just one header X-Request-Id, which can be configured with an environment variable NETRA_HTTP_REQUEST_ID_HEADER_NAME. To control the size of the context in Netramesh, you can set the following environment variables: NETRA_TRACING_CONTEXT_EXPIRATION_MILLISECONDS (the time for which the context will be stored) and NETRA_TRACING_CONTEXT_CLEANUP_INTERVAL (frequency of context cleaning).
It is also possible to combine multiple paths on your system by marking them with a special session marker. Netra allows you to install HTTP_HEADER_TAG_MAP to turn HTTP headers into appropriate tracing span tags. This can be especially useful for testing. After passing the functional test, you can see which part of the system was affected by filtering by the corresponding session key.
Determining the Source of a Request
To determine where the request came from, you can use the functionality of automatically adding a header with the source. With an environment variable NETRA_HTTP_X_SOURCE_HEADER_NAME you can specify a header name that will be automatically set. By using NETRA_HTTP_X_SOURCE_VALUE you can set the value to which the X-Source header will be set for all outgoing requests.
This allows uniform distribution of this useful header throughout the network. Then you can already use it in services and add it to logs, metrics.
Traffic routing and Netramesh internals
Netramesh consists of two main components. The first, netra-init, sets up network rules for intercepting traffic. He uses to intercept all or part of the traffic on the sidecar, which is the second main component of Netramesh. You can configure which ports to intercept for incoming and outgoing TCP sessions: INBOUND_INTERCEPT_PORTS, OUTBOUND_INTERCEPT_PORTS.
The tool also has an interesting feature - probabilistic routing. If you use Netramesh exclusively to collect tracing spans, then in a production environment you can save resources and enable probabilistic routing using variables NETRA_INBOUND_PROBABILITY ΠΈ NETRA_OUTBOUND_PROBABILITY (from 0 to 1). The default value is 1 (all traffic is intercepted).
After a successful interception, the netra sidecar accepts the new connection and uses SO_ORIGINAL_DST socket option to get the original destination. Netra then opens a new connection to the original IP address and establishes a two-way TCP communication between the parties, listening to all passing traffic. If a port is defined as HTTP, Netra tries to parse and trace it. If the HTTP parsing fails, Netra falls back to TCP and transparently proxy the bytes.
Building a dependency graph
After receiving a lot of tracing information in Jaeger, I want to get a complete graph of interactions in the system. But if your system is loaded enough and billions of tracing spans accumulate per day, it becomes not so easy to make their aggregation. There is an official way for this: . However, it will take hours to build a complete graph and force Jaeger to download the entire dataset for the past day.
If you are using Elasticsearch to store tracing spans, you can use , which will build the same graph in minutes, using the features and capabilities of Elasticsearch.
How to use netramesh
Netra can be easily added to any service running any orchestrator. You can see an example .
At the moment, Netra does not have the ability to automatically inject sidecars into services, but there are plans to implement it.
Future of Netramesh
The main goal is to achieve minimum resource costs and high performance, providing the main features for observability and control of inter-service interaction.
In the future, Netramesh will support other application layer protocols besides HTTP. L7 routing will be available in the near future.
Use Netramesh if you encounter similar issues and send us questions and suggestions.
Source: habr.com