🥇Use cases for service mesh

Note: translation.: The author of this article (Luc Perkins) is a developer advocate at CNCF, home to Open Source projects like Linkerd, SMI (Service Mesh Interface), and Kuma (by the way, have you ever wondered why Istio is not on this list?). Once again, trying to bring a better understanding of the trendy hype called 'service mesh' to the DevOps community, he presents 16 characteristic features offered by such solutions.

Today, service mesh — one of the hottest topics in software engineering (and rightly so!). I find this technology incredibly promising and dream of witnessing its widespread adoption (of course, when it makes sense). However, it is still shrouded in a veil of mystery for most people. Even those who are well acquainted with it often struggle to articulate its benefits and what exactly it entails (including your humble servant). In this article, I will attempt to rectify this by outlining various use cases of 'service meshes'.

* Note: This translation ('service mesh') will be used consistently throughout the article for the still new term.

But first, I want to make a few remarks:

I have never worked with service meshes and have only used them outside of projects initiated for my own education. On the other hand, I did write a lot of documentation for Twitter's internal service mesh in 2015 (at that time, it wasn't even called a 'service mesh') and participated in developing the website and documentation for Linkerd, so that counts for something.
My list is indicative and incomplete. There may be use cases I am unaware of, and undoubtedly new options will arise as the technology develops and grows in popularity.
At the same time, not every existing implementation of service mesh supports all the mentioned use cases. Therefore, expressions like 'service mesh can...' should be read as 'specific, and possibly all popular implementations of service mesh can...'.
The order of examples does not hold any significance.

Summary list:

service discovery;
encryption;
authentication and authorization;
load balancing;
circuit breaking;
auto-scaling;
canary deployments;
blue-green deployments;
health checks;
load shedding;
Traffic mirroring;
Isolation;
Rate limiting, retries, and timeouts;
Telemetry;
Audit;
Visualization.

1. Service Discovery

TL;DR: Connect to other services in the network using simple names.

Services should be able to automatically 'discover' each other using adequate names — for example, service.api.production, pets/staging or cassandra. Cloud environments are characterized by their elasticity, and one name can represent multiple instances of a service. Clearly, in such a situation, it's physically impossible to hardcode all IP addresses.

Moreover, when one service discovers another, it should be able to send requests to that service without fearing they will be directed to a non-functional instance. In other words, the service mesh must monitor the health of all service instances and maintain an up-to-date list of hosts.

Each service mesh implements service discovery mechanisms in its own way. Currently, the most common method is delegating to external processes like Kubernetes DNS. In the past, at Twitter, we used the naming system Finagle. Furthermore, the service mesh technology enables the creation of custom naming mechanisms (although I haven't encountered any implementation of SM with such functionality).

2. Encryption

TL;DR: Eliminate unencrypted traffic between services and let this process be automated and scalable.

It's reassuring to know that attackers cannot penetrate your internal network. Firewalls handle this well. But what happens if a hacker does get inside? Will they be able to do anything they want with inter-service traffic? Let's hope that doesn't occur. To prevent such a scenario, a zero-trust network should be implemented, where all traffic between services is encrypted. Most modern service meshes achieve this through mutual TLS (mutual TLS, mTLS). In some cases, mTLS operates in entire clouds and clusters (I believe interplanetary communications will be arranged similarly someday).

Of course, mTLS is not mandatory for service meshEach service can take care of its own TLS, but this means finding a way to generate certificates, distribute them across the service hosts, and include code in the application that will load these certificates from files. And don't forget to update these certificates after certain intervals. Service meshes automate mTLS with systems like SPIFFE, which, in turn, automate the process of issuing and rotating certificates.

3. Authentication and Authorization

TL;DR: Establish who is initiating the request and determine what they are allowed to do before the request reaches the service.

Services often want to know who is making the request (authentication), and using this information, decide what the subject is allowed to do (authorization). In this case, the pronoun "who" may refer to:

Other services. This is called "peer authentication". For example, a servicewants to access the service web db. Service meshes typically solve such problems using mTLS: certificates in this case serve as necessary identifiers.Some users (individuals). This is called "request authentication".
For example, a user haxor69wants to buy a new lamp. Service meshes provide various mechanisms, such as JSON Web Tokens. Many of us have done this in application code. A request comes in, we look up the table, find the user and compare the password, then check the column.
permissions usersetc. In the case of a service mesh, this happens even before the request reaches the service. After establishing who made the request, we need to determine what this subject is allowed to do. Some service meshes allow you to set basic policies (on who can do what) in the form of YAML files or via the command line, while others offer integration with frameworks like Open Policy Agent.

The ultimate goal is to enable your services to accept any requests, confidently assuming they originate from a trusted source and that this action is permitted.4. Load Balancing and TL;DR: Distribute load across service instances according to a specific pattern.

4. Load Balancing

TL;DR: Distribute the load across service instances following a specific pattern.

"Service" within a service sect often consists of many identical instances. For example, today the service cache consists of 5 copies, and tomorrow their number may increase to 11. Requests directed to cache, must be distributed according to a specific goal. For instance, to minimize latency or maximize the likelihood of hitting a working instance. The round-robin algorithm is most commonly used, but there are many others — for example, the weighted (weighted) request method (where preferred targets can be selected), ring (ring) hashing (using consistent hashing for upstream hosts) or the least connections method (preference is given to the instance with the fewest requests).

Classic load balancers also have other features, such as HTTP caching and DDoS protection, but they are not very relevant for east-west traffic (i.e., traffic occurring within the data center — note from translator) (a typical application area for service mesh). Of course, it is not necessary to use a service mesh for load balancing, but it allows for defining and controlling balancing policies for each service from a centralized management layer, thus eliminating the need to launch and configure separate balancers in the network stack.

5. Circuit Breaking

TL;DR: Stop traffic to a problematic service and manage damage in worst-case scenarios.

If, for any reason, a service cannot handle the traffic, the service mesh provides several options to address this issue (others will be discussed in the relevant sections). Circuit breaking is the most drastic way to disconnect a service from traffic. However, it doesn't make sense by itself — a backup plan is necessary. Backpressure (backpressure) on services performing requests can be considered (just remember to configure your service mesh for this!), or, for example, coloring the status page red and redirecting users to another version of the page with a 'falling whale' ('Twitter is down').

Service meshes allow not only to determine when to disconnect and what this will be followed. In this case, 'when' can include any combination of specified parameters: total number of requests over a certain period, number of parallel connections, pending requests, active retries, etc.

You are unlikely to want to abuse circuit breaking, but it's good to know that there's a backup plan just in case.

6. Auto-scaling

TL;DR: Increase or decrease the number of service instances based on the specified criteria.

Service meshes are not orchestrators, so they do not perform scaling by themselves. However, they can provide the information on which orchestrators will base their decisions. Since service meshes have access to all traffic between services, they possess extensive information about what is happening: which services are encountering problems, which are severely underloaded (the resources allocated to them are wasted), etc.

For example, Kubernetes scales services based on the usage of CPU and memory by pods (see our report “Auto-scaling and Resource Management in Kubernetes” — ed. note), but if you decide to scale based on any other metric (in our case, traffic-related), a special metric will be needed. Guidelines like this one show how to do it with Envoy, Istio and Prometheus, but the process itself is quite complex. We wish that the service mesh could simplify it, allowing for simple conditions like 'increase the number of service instances auth, if the number of pending requests exceeds the threshold for one minute.'

7. Canary Deployments

TL;DR: Test new features or service versions on a subset of users.

Let's say you're developing a SaaS product and about to roll out its new cool version. You've tested it in staging, and it performed excellently. However, you still have certain concerns about its behavior in real-world conditions. In other words, it's necessary to check the new version on actual tasks without risking user trust. Canary deployments are perfect for this. They allow you to showcase a new feature to a subset of users. This subset can consist of the most loyal users, those using the free version of the product, or users who have expressed a desire to be "guinea pigs."

Service meshes make this possible, allowing you to specify criteria that determine who sees which version of the application, and routing traffic accordingly. Meanwhile, nothing changes for the services themselves. Version 1.0 of the service assumes that all requests come from users who should see it, while version 1.1 believes the same about its users. And you can adjust the percentage of traffic between the old and new versions, redirecting an increasing number of users to the new one if it performs stably and your "guinea pigs" give their approval.

8. Blue-green deployments

TL;DR: Roll out the new cool feature, but be ready to revert everything immediately.

The essence of blue-green deployments is to launch a new "blue" service alongside the old "green" one. If everything goes smoothly and the new service performs well, the old one can be gradually shut down. (Unfortunately, someday this new "blue" service will share the fate of the "green" and disappear…) Blue-green deployments differ from canary releases in that the new feature covers all users (rather than a part); the idea here is to have a "fallback option" ready in case something goes wrong.

Service meshes offer a very convenient way to test a 'blue' service and instantly switch to the operational 'green' service in case of issues. Not to mention the wealth of information they provide (see the 'Telemetry' section below) about the 'blue' service's performance, which helps determine its readiness for full-scale usage.

Note: translation.: You can learn more about different deployment strategies in Kubernetes (including the mentioned canary, blue/green, and others) in from one of the authors..

9. Health Checks

TL;DR: Monitor which service instances are healthy and respond to those that are not.

Health Checks (health check) help determine whether service instances are ready to accept and process traffic. For example, in the case of HTTP services, a health check may look like a GET request to an endpoint /health. A response 200 OK indicates that the instance is healthy, while any other response means it is not ready to handle traffic. Service meshes allow you to specify both the method by which health is checked and the frequency of these checks. This information can then be used for other purposes—for instance, load balancing and circuit breaking.

Thus, health checks are not an independent use case; they are typically used to achieve other objectives. Depending on the results of the health checks, external actions regarding other service mesh goals may be required: for example, updating a status page, creating an issue on GitHub, or filling in a JIRA ticket. And service meshes offer a convenient mechanism for automating all of this.

10. Load Shedding

TL;DR: Redirect traffic in response to temporary spikes in usage.

If a certain service becomes overloaded with traffic, you can temporarily redirect part of that traffic elsewhere (that is, 'shed' it) (shed) there. For example, to a backup service or data center, or to the permanent Pulsar Topic. As a result, the service continues to handle part of the requests instead of crashing and stopping all processing entirely. Throttling is preferable to breaking the chain, but it is still important not to abuse it. It helps prevent cascading failures that can take down downstream services.

11. Traffic Parallelization/Mirroring

TL;DR: Send one request to multiple locations at once.

Sometimes there is a need to send a request (or a set of requests) to multiple services simultaneously. A typical example is routing part of the production traffic to a staging service. The main web server of production sends a request to the downstream service products.production and only to it. The service mesh intelligently copies this request and sends it to products.staging, which the web server is not even aware of.

Another related use case for the service mesh that can be implemented on top of traffic parallelization is regression testing. It involves sending the same requests to different versions of the service and checking whether all versions behave in the same way. I have not yet come across an implementation of service mesh with an integrated regression testing system like Diffy, but the idea itself seems promising.

12. Isolation

TL;DR: Break your service mesh into mini-networks.

Also known as segmentation, isolation is the art of dividing the service mesh into logically separate segments that know nothing about each other. Isolation is somewhat akin to creating virtual private networks. The fundamental difference is that you can still take advantage of all the benefits of the service mesh (such as service discovery) but with added security. For example, if an attacker manages to penetrate a service in one of the subnets, they will not be able to see which services are running in other subnets or intercept their traffic.

Moreover, there can also be organizational benefits. You might want to break services into subnets depending on the company structure and relieve developers from the cognitive load of keeping the entire service mesh in mind.

13. Rate Limiting, Retries, and Timeouts

TL;DR: No need to include urgent request management tasks in the codebase anymore.

All these items could be considered as separate use cases, but I decided to combine them due to one common feature: they take on request lifecycle management tasks typically handled by application libraries. If you are developing a web server in Ruby on Rails (not integrated with a service mesh) that makes requests to backend services via gRPC, the application will have to decide what to do if N requests fail. It will also need to determine how much traffic these services can handle and 'hardcode' those parameters using a special library. Additionally, the application must decide when it's time to give up and let the request timeout. To change any of the above parameters, the web server will need to be stopped, reconfigured, and restarted.

Delegating these tasks to the service mesh means not only that service developers won't need to think about them, but also that they can be considered on a more global scale. If a complex service chain is used, say, A → B → C → D → E, the entire request lifecycle must be taken into account. If there's a need to extend timeouts in service C, it makes sense to do it all at once rather than in parts: updating the service code and waiting for the pull request to be accepted and the CI system to deploy the updated service.

14. Telemetry

TL;DR: Collect all the necessary (and not so necessary) information from the services.

Telemetry is a general term that encompasses metrics, distributed tracing, and logs. Service meshes offer mechanisms for collecting and processing all three types of data. Here it gets a bit murky as the number of possibilities is vast. For collecting metrics, there are Prometheus and other tools, for collecting logs one can use fluentd, Loki, Vector and others. (for example, ClickHouse with our loghouse for K8s — ed. note), for distributed tracing, there is Jaeger and so on. Each service mesh may support certain tools and not others. It will be interesting to see if the project Open Telemetry can provide some convergence.

In this case, the advantage of the service mesh technology is that sidecar containers can, in principle, collect all the aforementioned data from their services. In other words, you have a unified telemetry gathering system at your disposal, and the service mesh can process all this information in various ways. For example:

tail logs from a certain service in the CLI;
monitor the volume of requests from the service mesh's dashboard;
collect distributed traces and redirect them to a system like Jaeger.

Attention, subjective opinion: Generally speaking, telemetry is an area where strong interference from the service mesh is undesirable. Collecting basic information and monitoring some 'golden metrics' in real-time, such as the percentage of successful requests and latency, is fine, but let's hope we do not witness the emergence of Frankenstein stacks that try to replace specialized systems, some of which have already proven themselves well and are well-studied.

15. Audit

TL;DR: Those who forget the lessons of history are doomed to repeat them.

Audit is the art of observing important events in the system. In the case of the service mesh, this may mean tracking who made requests to specific endpoints of certain services or how many times a certain security-related event occurred in the last month.

It is clear that auditing is very closely related to telemetry. The difference is that telemetry is usually associated with things like performance and technical consistency, whereas auditing may relate to legal and other issues that extend beyond the strictly technical sphere (for example, compliance with GDPR — the EU General Data Protection Regulation).

16. Visualization

TL;DR: Long live React.js — an inexhaustible source of quirky interfaces.

Perhaps there is a more appropriate term, but I don’t know it. I simply mean the graphical representation of the service mesh or some of its components. These visualizations may include indicators such as average latencies, information on the configuration of sidecar containers, health check results, and alerts.

Working in a service-oriented environment involves a much greater cognitive load compared to His Majesty the Monolith. Therefore, cognitive pressure must be reduced at all costs. A simple graphical interface for a service mesh that allows users to click a button and get the desired result can be crucial for the popularity of this technology.

Not included in the list

Initially, I intended to include a few more use cases in the list, but then I decided against it. Here they are along with the reasons for my decision:

Multi-datacenter. In my view, this is not so much a use case as it is a narrow and specific area of application for service meshes or some set of features like service discovery.
Ingress and egress. This is a related area, but I limited myself (perhaps artificially) to the use case of "east-west traffic." Ingress and egress deserve a separate article.

Conclusion

That's all for now! Again, this list is quite conditional and likely incomplete. If you think I missed something or made a mistake, feel free to reach out to me on Twitter (@lucperkins). Please adhere to the rules of politeness.

P.S. from the translator

The illustration for the article is based on an image from the article “What is a Service Mesh (and when to use one)?” (author — Gregory MacKinnon). It shows how part of the functionality from applications (in green) has transitioned to the service mesh, establishing relationships between them (in blue).