Note: translation.: Service mesh is a phenomenon that still lacks a stable translation into English (over 2 years ago, we suggested the term âservice gridâ, while later some colleagues began to actively promote the combination âservice sieveâ). Ongoing discussions about this technology have led to a situation where marketing and technical aspects are too closely intertwined. This excellent material from one of the authors of the original term aims to provide clarity for engineers and beyond.
Comic by
Introduction
If youâre a software engineer working somewhere in the realm of backend systems, the term âservice meshâ has probably settled firmly in your mind over the last couple of years. Due to a strange twist of fate, this phrase is gaining traction in the industry, while the hype and associated marketing offers are snowballing down the slope, showing no signs of slowing down.
Service mesh originated in the murky, biased waters of the cloud native ecosystem. Unfortunately, this means that a significant portion of the related debate ranges from âlow-calorie chatterâ toâusing a technical termâsheer nonsense. But if you cut through the noise, you can find that service mesh serves a very real, specific, and important purpose.
In this publication, I will attempt to do just that: present an honest, in-depth, engineer-focused guide to service mesh. I intend to answer not only the question: âWhat is it?â, â but also âWhy?â, as well as âWhy now?â. Finally, I will try to outline why (in my opinion) this particular technology has generated such insane buzz, which in itself is an interesting story.
Who am I?
Hello everyone! My name is . I am one of the creators of â the very first service mesh project and the project that is responsible for the emergence of the term service mesh as such (sorry, guys!). (Revision note.: By the way, at the dawn of this term's emergence, over 2.5 years ago, we had already translated early material by the same author titled â».) I also lead â a startup focused on creating cool service mesh stuff like Linkerd and .
You might guess that I have quite a biased and subjective opinion on this matter. However, I will try to minimize bias (with the exception of one section: âWhy is there so much talk about service mesh?â, in which I will still share my prejudiced thoughts). I will also make every effort to ensure that this guide is as objective as possible. In specific examples, I will mainly rely on the experience of Linkerd, pointing out any known differences (if they exist) in the implementations of other types of service mesh.
Okay, it's time to get to the good stuff.
What is a service mesh?
Despite all the hype, a service mesh is structurally quite simple. It is just a bunch of userspace proxies located "close" to the services (we'll talk a bit about what "close" means), plus a set of management processes. Collectively, the proxies are called data plane, while the management processes are referred to as control plane. The data plane intercepts calls between services and does "various things" with them; the control plane, accordingly, coordinates the behavior of the proxies and provides you, the operator, access to the API, allowing you to manipulate the network and measure it as a whole.
What kind of proxies are these? They are TCP proxies of the "Layer 7-aware" category (i.e., "understanding" the 7th layer of the OSI model) , like HAProxy and NGINX. You can choose a proxy to your liking; Linkerd uses a proxy written in Rust, plainly named . We built it specifically for the service mesh. Other meshes prefer different proxies (Envoy is a common choice). However, the choice of proxy is merely a matter of implementation.
What do these proxy servers do? Obviously, they proxy calls to and from the services (strictly speaking, they act as both forward and reverse proxies, handling both incoming and outgoing calls). And they implement a set of functions focused on calls between services. This focus on traffic between services differentiates service mesh proxies from, say, API gateways or ingress proxies (the latter focus on calls coming into the cluster from the outside world). (Note: translation.: for a comparison of existing Ingress controllers for Kubernetes, many of which already use the mentioned Envoy, see .)
So, we've covered the data plane. The control plane is simpler: it is a set of components that provide all the mechanics necessary for the data plane to work in a coordinated manner, including service discovery, issuing TLS certificates, aggregating metrics, etc. The data plane informs the control plane of its behavior; in turn, the control plane provides an API that allows modifying and monitoring the behavior of the data plane as a whole.
Below is a diagram of the control plane and data plane in Linkerd. As can be seen, the control plane consists of several different components, including a Prometheus instance that collects metrics from the proxy servers, as well as other components such as destination (service discovery), identity (certificate authority, CA) and public-api (endpoints for web and CLI). In contrast, the data plane consists of a simple linkerd-proxy next to the application instance. This is just a logical diagram; in real-world deployments, you might have three replicas of each control plane component and hundreds or thousands of proxies in the data plane.
(The blue rectangles in this diagram represent the boundaries of Kubernetes pods. It is evident that the containers with linkerd-proxy are in the same pod as the application containers. This type of diagram is known as a sidecar container..)
The service mesh architecture has several important implications. First, since the role of the proxy is to intercept calls between services, the service mesh makes sense only if your application is built on a set of services. A mesh in could be used with monoliths, but it is clearly redundant for a single proxy, and its functionality is unlikely to be needed.
Another important implication is that the service mesh requires a huge number of proxies. In fact, Linkerd attaches a linkerd-proxy to every instance of every service (other implementations add proxies to each node/host/virtual machine. In any case, that's a lot). Such intensive use of proxies brings several additional complications:
- Proxies in the data plane must be fast,since each call involves a pair of requests to the proxy: one on the client side, one on the server side.
- Proxies also need to be small and lightweight.Each will consume memory and CPU resources, and this consumption will grow linearly with the application.
- You will need a mechanism for deploying and updating a large number of proxies. Doing this manually is not an option.
In general, a service mesh looks like this (at least from a bird's-eye view): you deploy a bunch of userspace proxies that 'do something' with internal, inter-service traffic, and you use a control plane to monitor and manage them.
It's time for the question 'Why?'
What is the purpose of a service mesh?
For those encountering the concept of a service mesh for the first time, it's understandable to feel a bit apprehensive. The structure of a service mesh means that it will not only increase latency in the application, but it will also consume resources and add a bunch of new mechanisms to the infrastructure. First, you install the service mesh, and then you suddenly find that you need to manage hundreds (if not thousands) of proxies. The question arises, who would willingly take this on?
The answer to this question is twofold. First, the operational costs associated with deploying these proxies can be significantly reduced due to some changes happening in the ecosystem (more on this later).
Second, such a setup is actually a great way to introduce additional logic into the system. And not only because a service mesh can add many new features, but also because this can be done without interfering in the ecosystem. In fact, the entire model of a service mesh is based on this premise: in a multi-service system, regardless of what the individual services do, the traffic between them is an ideal point for adding functionality. For example, in Linkerd (as in most meshes), the functionality is primarily focused on HTTP calls, including HTTP/2 and gRPC*. The functionality is quite richâit can be categorized into three classes:
Features related to
- reliability. Retrying requests, timeouts, canary approaches (traffic splitting/redirection), etc.monitoring.
- reliability. Aggregating metrics on success rates, latencies, and request volumes for each service or specific routes; building service topology maps, etc.security.
- reliability. Mutual TLS, access control, etc..
* From Linkerd's perspective, gRPC is practically indistinguishable from HTTP/2: it's just that protobuf is used in the payload. From a developer's standpoint, these two aspects do differ.
Many of these mechanisms operate at the request level (hence the term 'L7 proxy'). For example, if the Foo service sends an HTTP call to the Bar service, the linkerd-proxy on the Foo side can perform intelligent load balancing, directing calls from Foo to Bar instances based on observed latency; it can retry the request if necessary (and if it is idempotent); it can log the response code and waiting time, etc. Similarly, the linkerd-proxy on the Bar side can reject a request if it is unauthorized or exceeds the request limit; it can record latency on its side, etc.
Proxies can also do 'something' at the connection level. For example, the linkerd-proxy on the Foo side can initiate a TLS connection while the linkerd-proxy on the Bar side can terminate it, and both sides can verify each other's TLS certificates*. This not only provides encryption between services but also a cryptographically secure method of authenticating services: Foo and Bar can 'prove' that they are who they claim to be.
* 'Each other' means that the client certificate is also verified (mutual TLS). In 'classic' TLS, for instance, between a browser and a server, typically only one side's certificate (the server's) is verified.
Regardless of whether they operate at the request or connection level, it is important to emphasize that all service mesh functions are operational in nature. Linkerd cannot transform the semantics of the payload â for instance, adding fields to a JSON fragment or altering protobuf. We will discuss this important feature later when addressing ESB and middleware.
Such is the set of functions that a service mesh offers. This raises the question: why not implement them directly in the application? And why even deal with proxies?
Why A Service Mesh Is A Good Idea
Although the capabilities of a service mesh are intriguing, its primary value actually lies not in its features. After all, we can implement them directly in the application (later we will see that this was the origin of the service mesh). If we try to express this thought in one sentence, the value of the service mesh is as follows: it provides functions that are critically important for the operation of modern server software, uniformly for the entire stack and independently of the application code..
Let's analyze this sentence.
«Functions that are critically important for the operation of modern server software. If you are creating a transactional server application connected to the public internet, handling requests from the outside world and responding to them within a short timeâsuch as a web application, an API server, and indeed the vast majority of other modern applicationsâand if you implement it as a set of services that synchronously interact with each other, and if you are constantly upgrading this software, adding new capabilities, and if you are forced to maintain this system in a working state during modificationsâcongratulations, you are creating modern server software. And all those wonderful functions listed above are indeed critically important for you. The application must be reliable, secure, and you should have the ability to monitor what it is doing. These are precisely the issues that the service mesh helps to address.
(Okay, I still let my conviction slip into the previous paragraph that this approach is a modern way to create server software. Others prefer to develop monoliths, 'reactive microservices,' and other things that don't fall under the definition provided above. These people certainly have their own opinions that differ from mine. For my part, I believe they are 'wrong'âthough in any case, the service mesh is not particularly useful for them).
«Uniformly for the entire stack. The functions provided by the service mesh are not just critically important. They apply to all services in the application regardless of what language they are written in, what framework they use, who wrote them, how they were deployed, and all the other subtleties of their development and application.
«Independently of the application codeFinally, a service mesh not only provides unified functionality for the entire stackâit does so in a way that does not require application modifications. The fundamental basis of service mesh functionality, including tasks related to configuration, updates, operations, maintenance, etc., lies solely at the platform level and is independent of the application. The application can change without affecting the service mesh. In turn, the service mesh can change without any involvement from the application.
In short, a service mesh not only provides critical features but does so in a global, uniform, and application-independent manner. Therefore, while the functionality of a service mesh can be implemented in the service code (for example, as a library included in each service), this approach will not ensure the consistency and independence that are so valuable in a service mesh.
And all it takes is to add a bunch of proxies! I promise we will soon examine the operational costs associated with adding these proxies. But first, letâs pause and look at this idea of independence from the perspective of various people.
Who benefits from a service mesh?
As uncomfortable as it may be, for a technology to become an integral part of the ecosystem, it must be adopted by people. So who is interested in a service mesh? Who gains from its use?
If you are developing modern server software, you can roughly envision your team as a group of service owners, who collectively develop and implement business logic, and platform owners, who focus on building the internal platform on which these services operate. In smaller organizations, these may be the same individuals, but as the company grows, these roles typically become more pronounced and even split into sub-roles⊠(There is much to say about the changing nature of DevOps, the organizational impact of microservices, etc. But for now, letâs accept these descriptions as a given).
From this perspective, the clear beneficiaries of a service mesh are platform owners. Ultimately, the goal of the platform team is to create an internal platform where service owners can implement business logic in a way that guarantees their maximum independence from the grim details of its operation. A service mesh not only offers capabilities that are critical to achieving this goal, but it does so in a way that does not impose dependencies on service owners.
Service owners also benefit, albeit in a more indirect way. The goal of a service owner is to be as productive as possible in implementing business process logic, and the less they have to worry about operational issues, the better. Instead of dealing with the implementation of, say, retry policies or TLS, they can focus exclusively on business tasks and hope that the platform will take care of everything else. This is a significant advantage for them.
The organizational value of such a separation between platform and service owners is hard to overstate. I believe it contributes the main to the value of the service mesh.
We learned this lesson when one of the early advocates of Linkerd shared with us why they chose a service mesh: because it allowed them to 'minimize the chatter.' Here are some details: a team from a large company migrated their platform to Kubernetes. Since the application handled sensitive information, they wanted to encrypt all communications within the clusters. However, the situation was complicated by the presence of hundreds of services and hundreds of development teams. The prospect of reaching out to everyone and convincing them to incorporate TLS support into their plans was not appealing at all. By installing Linkerd, they shifted responsibility from developers (who viewed it as an unnecessary hassle) to platform developers, for whom it was a top priority. In other words, Linkerd solved for them not just a technical but an organizational problem.
In short, a service mesh is more of a solution to a not just technical but a socio-technical problem. (Thanks for introducing this term.)
Will a service mesh solve all my problems?
Yes. I mean, no!
Looking at the three function classes mentioned aboveâreliability, security, and observabilityâit becomes clear that a service mesh is not a complete solution for any of these problems. While Linkerd can make retries (if it knows they are idempotent), it cannot make decisions about what to return to the user if a service has completely failedâsuch decisions must be made by the application. Linkerd can keep track of successful requests, but it cannot peek into a service and provide its internal metricsâthis kind of tooling must be within the application. And although Linkerd is capable of organizing mTLS, complete security solutions require much more.
A subset of functions in these areas offered by the service mesh relates to platform features. By this, I mean features that are:
- Independent of business logic. The way histogram calls between Foo and Bar are constructed is completely independent of what why Foo calls Bar.
- It's difficult to implement properly. In Linkerd, retries are parameterized by various fancy things like retry budgets (retry budgets), as a straightforward, blunt approach to implementing such things is sure to lead to the so-called 'retry storm' (retry storm) and other issues characteristic of distributed systems.
- Are most effective when applied uniformly. The TLS mechanism only makes sense when applied everywhere.
Since these functions are implemented at the proxy level (not at the application level), the service mesh provides them at the platform, not the application level. Thus, it doesn't matter what language the services are written in, what framework they use, who wrote them, or why. Proxies operate beyond all these details, and the fundamental basis of this functionality, including tasks related to configuration, updates, operation, maintenance, etc., lies solely at the platform level.
Examples of service mesh capabilities
In conclusion, I would like to say that a service mesh is not a complete solution for ensuring reliability, observability, or security. The scope of these areas necessitates the mandatory involvement of service owners, Ops/SRE teams, and other stakeholders within the company. A service mesh provides only a "slice" at the platform level for each of these areas.
Why has service mesh become popular right now?
You are probably wondering: okay, if service mesh is so great, why didnât we start deploying millions of proxies in our stacks ten years ago?
There is a simple answer to this question: ten years ago, everyone was building monoliths, and no one needed a service mesh. This is true, but in my opinion, such an answer misses the essence. Even ten years ago, the concept of microservices as a promising way to create large-scale systems was widely discussed and applied in companies like Twitter, Facebook, Google, and Netflix. The general sentimentâat least in those parts of the industry with which I interactedâwas that microservices were the "right way" to build large systems, even if it was incredibly challenging.
Of course, while there were companies exploiting microservices ten years ago, they were not simply plugging proxies everywhere to form a service mesh. However, upon closer examination, they were doing something similar: many of those companies mandated the use of a specific internal library for network interactions (sometimes referred to as a fat client library. fat client library).
Netflix had Hysterix, Google had Stubby, and Twitter had the Finagle library. Finagle, for instance, was mandatory for every new service at Twitter. It handled both the client and server sides of connections, allowed for request retries, supported request routing, load balancing, and metrics. It provided a consistent layer of reliability and observability for the entire Twitter stack, regardless of what exactly the service was doing. Of course, it only worked for JVM languages and was based on the programming model that had to be used for the entire application. However, its capabilities were almost identical to those of a service mesh. (The first version of Linkerd was essentially just Finagle wrapped in proxy form.)
Thus, ten years ago there were not only microservices but also specialized proto-service-mesh libraries addressing the same issues that service meshes solve today. However, there was no service mesh at that time. Another shift had to happen before it emerged.
And this is where a deeper answer lies, hidden in another change that occurred over the past 10 years: there was a dramatic decrease in the cost of deploying microservices. The companies mentioned earlierâTwitter, Netflix, Facebook, Googleâwere large-scale enterprises with substantial resources. They not only had the need but also the capacity to create, deploy, and operate large-scale applications based on microservices. The energy and effort put forth by Twitter engineers in transitioning from a monolithic to a microservices approach is simply astonishing. (Honestly, the fact that they succeeded is equally impressive.) Such infrastructural maneuvers were then impossible for smaller companies.
Let's move to the present. Today, there are startups where the ratio of microservices to developers is 5:1 (or even ), and more importantly, they are successfully managing them! If a startup of 5 people can effortlessly operate 50 microservices, then something has clearly reduced the cost of their implementation.
1500 microservices at Monzo; each line is a prescribed network rule allowing traffic
The sharp decrease in the cost of operating microservices is the result of one process: the rise in popularity of containers and orchestrators. This is, in fact, the profound answer to what has contributed to the emergence of service mesh. The same technology has made both service mesh and microservices attractive: Kubernetes and Docker.
Why? Well, Docker resolves one major issueâthe packaging problem. By packaging an application and its (non-network) runtime dependencies into a container, Docker transforms the application into a portable unit that can be deployed and run anywhere. At the same time, it significantly simplifies the operation of multi-language Stack: since a container is an atomic unit of execution, it does not matter what is inside for deployment and operation purposes, whether it is an application on JVM, Node, Go, Python, or Ruby. You simply run it, and that's all.
Kubernetes takes everything to a new level. Now that there are a bunch of "executables" and numerous machines on which to run them, there is a need for a tool that can match them together. Broadly speaking, you give Kubernetes many containers and many machines, and it matches them up (of course, this is a dynamic and constantly changing process: new containers move around the system, machines are started and stopped, etc. However, Kubernetes takes all of this into account).
After setting up Kubernetes, the time spent on deploying and operating one service is not significantly different from that spent on deploying and operating ten services (in fact, they are practically identical even for 100 services). Add to this containers as a packaging mechanism that encourages multilingual implementation, and you get a plethora of new applications implemented as microservices, written in different languages â just the environment that service mesh is well suited for.
So, we have come to the answer to the question of why the idea of service mesh has become popular right now: the uniformity that Kubernetes provides for services is directly applicable to the operational tasks posed by the service mesh. You package proxies into containers, give Kubernetes the task of attaching them wherever possible, and voila! You end up with a service mesh, with all the deployment mechanics managed by Kubernetes. (At least, from a bird's-eye perspective. Of course, there are many nuances in this process.)
In summary: the reason why service mesh has become popular right now, rather than ten years ago, is that Kubernetes and Docker have not only significantly increased the need for it, simplifying the implementation of applications as sets of multilingual microservices, but also greatly reduced the costs of its operation by providing mechanisms for deploying and supporting sidecar-proxy fleets.
Why is there so much talk about service mesh?
WarningIn this section, I resort to all sorts of assumptions, guesses, speculations, and insider information.
When you search for the phrase "service mesh," you'll stumble upon a plethora of recycled, low-calorie content, strange projects, and a kaleidoscope of distortions worthy of an echo chamber. This is characteristic of any trendy new technology, but with service mesh, the issue is particularly acute. Why?
Well, part of it is my fault. I've done everything I could to promote Linkerd and service mesh at every opportunity, through countless blog posts and articles like this one. But I'm not that powerful. To truly answer this question, we should discuss the overall situation a bit. However, itâs impossible to do so without mentioning one project: â an open-source service mesh co-developed by Google, IBM, and Lyft.
(These three companies have very different roles: Lyft's involvement seems to be limited to just the name; they created Envoy but neither use Istio nor participate in its development. IBM contributes to Istio and uses it. Google is actively involved in Istio development but, as far as I can tell, doesnât actually use it.)
The Istio project is notable for two reasons. First, the enormous marketing efforts that Google, in particular, is applying to its promotion. In my estimation, most people aware of the concept of service mesh today learned about it through Istio for the first time. The second notable aspect is how poorly Istio has been received. In this regard, I am clearly a vested interest, but in trying to remain as objective as possible, I still can't help but , which is rather unusual (though not unique: systemd comes to mind, âŠ) for an open-source project.
(In practice, Istio seems to have issues not only with complexity and UX but also with performance. For instance, during the , conducted by a third party, experts found instances where Istio's tail latency was 100 times worse than that of Linkerd, as well as situations of resource shortages, where Linkerd continued to function successfully while Istio completely failed.)
Putting aside my theories about why this happened, I believe that the overwhelming hype surrounding service mesh is largely due to Google's involvement. Specifically, it's a combination of the following three factors:
- Google's aggressive promotion of Istio;
- the corresponding disapproving, critical attitude towards the project;
- and the recent rapid rise in popularity of Kubernetes, the memories of which are still fresh.
Together, these factors create a sort of intoxicating, oxygen-deprived environment where the ability to think rationally diminishes, leaving only a peculiar version of .
From Linkerd's perspective, I would describe this as a mixed blessing. I mean, it's great that service mesh has entered the mainstreamâsomething that didn't happen in 2016, when Linkerd first emerged and it was genuinely hard to attract attention to the project. That problem is no longer present! But the downside is that the situation with service mesh today is so tangled that it's nearly impossible to understand which projects truly fall under the service mesh category (let alone figure out which one is best suited for a specific use case). This certainly complicates things for everyone (and, indeed, in some cases, Istio or another project might be a better fit than Linkerd, as the latter is not a one-size-fits-all solution).
From Linkerd's side, our strategy has been to ignore the noise, continue focusing on addressing real issues in the community, and essentially wait for the hype to settle down. Ultimately, the hype will die down, and we can continue to work peacefully.
For now, we all have to be a bit patient.
Will service mesh be useful to me, a humble software engineer?
The following questionnaire will help clarify the answer to this question:
Are you solely focused on implementing business logic? In that case, service mesh will not be useful to you. Of course, you may find it interesting, but ideally, service mesh should not directly affect anything in your environment. Keep working on what you are being paid for.
Are you maintaining a platform at a company that uses Kubernetes? Yes, in this case, you definitely need a service mesh (unless you are using K8s just to run a monolith or batch processing â in which case, I would like to know why you need K8s at all). You will likely find yourself in a situation with many microservices written by different people. They all interact with each other and are entangled in a web of runtime dependencies, and you need to find a way to handle all of this. Using Kubernetes allows you to choose a service mesh that fits your needs. To do this, familiarize yourself with their capabilities and features and answer the question of whether any existing project suits you at all (I recommend starting your research with Linkerd).
Are you managing the platform in a company that does NOT use Kubernetes but does use microservices? In this case, a service mesh would be beneficial, but its implementation will be non-trivial. Of course, you could simulate the operation of a service mesh by placing a bunch of proxies, but a significant advantage of Kubernetes is precisely its deployment model: managing these proxies manually would require much more time, effort, and cost.
Are you responsible for the platform in a company that works with monoliths? In this case, you probably donât need a service mesh. If you are working with monoliths (or even sets of monoliths) that have clearly defined and rarely changing interaction patterns, a service mesh wonât offer you much. So you can simply ignore it and hope it disappears like a bad dreamâŠ
Conclusion
Perhaps calling a service mesh "the most hyped technology in the world" is an overstatement â that dubious honor likely belongs to Bitcoin or AI. It might rank in the top five. But if you cut through the layers of noise and hype, it becomes clear that a service mesh provides real benefits to those developing applications in Kubernetes.
I would encourage you to try Linkerd â its installation in a Kubernetes cluster (or even in Minikube on a laptop) , and you will see for yourself what I mean.
FAQ
â If I ignore the service mesh, will it go away?
â I'm afraid I'll have to disappoint you: the service mesh is here to stay.
â But I DON'T WANT to use a service mesh!
â Well, that's fine! Just read my questionnaire above to understand if you should at least familiarize yourself with its basics.
â Isn't this just good old ESB/middleware dressed up in a new guise?
â A service mesh handles operational logic rather than semantic. of enterprise service buses (). Maintaining this separation helps the service mesh avoid the same fate.
â How does a service mesh differ from API gateways?
â There are a million articles on this topic. Just Google it.
â Is Envoy a service mesh?
â No, Envoy is not a service mesh; it is a proxy server. It can be used to organize a service mesh (and much more â itâs a general-purpose proxy). But on its own, it is not a service mesh.
â Is Network Service Mesh a service mesh?
â No. Despite the name, it is not a service mesh (isn't that the wonders of marketing?).
â Will a service mesh help my reactive asynchronous system based on message queues?
â No, a service mesh wonât help you.
â Which service mesh should I use?
â , it's obvious.
â This article is terrible! / The author should be condemned!
â Please share the link to it with all your friends so they can see for themselves!
Acknowledgments
As you may have guessed from the title, this article was inspired by the fantastic treatise by Jay Kreps, "" I met Jay ten years ago while interviewing at LinkedIn, and since then he has been an inspiration for me.
Although I love to call myself a âLinkerd developer,â the reality is that I am more of a maintainer of the README.md file for the project. Today, there are , , people working on Linkerd, and this project wouldnât have come to fruition without the involvement of the wonderful community of contributors and users.
And in conclusion, a special thanks to the creator of Linkerd, (primus inter pares), who dove headfirst into all this service mesh chaos with me many years ago.
P.S. from the translator
Also read in our blog:
- «»;
- «»;
- «»;
- «»;
- «».
Source: habr.com
