September 19 in Moscow the first HUG (Highload++ User Group) thematic meetup, which was dedicated to microservices. There was a report “Microservices Operation: Size Matters Even If You Have Kubernetes”, in which we shared Flant’s extensive experience in the field of operating projects with microservice architecture. First of all, it will be useful to all developers who are thinking about applying this approach in their current or future project.
Introducing (50 minutes, much more informative than the article), as well as the main extract from it in text form.
NB: Video and presentation are also available at the end of this post.
Introduction
Usually a good story has an opening, a main plot, and a denouement. This report is more like a plot, and tragic. It is also important to note that it presents a view of microservices from the side operation.
I'll start with this chart, the author of which (in 2015) Martin Fowler:
It shows how in the case of a monolithic application that has reached a certain value, productivity begins to fall. Microservices differ in that the initial productivity with them is lower, but as the complexity grows, the degradation of efficiency is not so noticeable for them.
I will supplement this graph for the case of using Kubernetes:
Why did the application with microservices get better? Because such an architecture puts forward serious requirements for the architecture, which in turn are perfectly covered by the capabilities of Kubernetes. On the other hand, some of this functionality will be useful for a monolith, especially for the reason that a typical monolith today is not quite a monolith (details will be later in the report).
As you can see, the final graph (when both a monolithic and a microservice application in an infrastructure with Kubernetes) is not very different from the original one. Further, we will talk about applications operated using Kubernetes.
Useful and harmful microservices
And here is the main idea:
What is a normal microservice architecture? It should bring you real benefits, increasing the efficiency of your work. Going back to the chart, here it is:
If you call her useful, then on the other side of the graph will be harmful microservice (interferes with work):
Returning to the “main thought”: is it worth trusting my experience at all? Since the beginning of this year I have looked 85 projects. Not all of them were microservices (about a third to a half of them had such an architecture), but this is still a large number. We (Flant) as outsourcers manage to see a wide variety of applications developed both in small companies (with 5 developers) and in large ones (~500 developers). An added bonus is that we've seen these apps live and evolve over the years.
Why microservices?
When asked about the benefits of microservices, there is from the already mentioned Martin Fowler:
- clear boundaries of modularity;
- independent deployment;
- freedom of choice of technologies.
I talked a lot with architects and software developers and asked why they need microservices. And he made his list of their expectations. Here's what happened:
If you describe “in sensations” some of the points, then:
- clear module boundaries: here we have a terrible monolith, and now everything will be neatly laid out in Git repositories, in which everything is “on the shelves”, warm and soft are not mixed;
- deployment independence: we will be able to roll out services independently so that development goes faster (publish new features in parallel);
- development independence: we can give this microservice to one team / developer, and that one to another, thanks to which we can develop faster;
- боGreater reliability: if partial degradation occurs (one microservice out of 20 falls), then only one button will stop working, and the system as a whole will continue to function.
Typical (harmful) microservice architecture
To explain why the reality is not what we expect, I will introduce collective an image of a microservice architecture based on experience from many different projects.
An example would be an abstract online store that is going to compete with Amazon or at least OZON. Its microservice architecture looks like this:
For a combination of reasons, these microservices are written on different platforms:
Since each microservice must have autonomy, many of them need their own database and cache. The final architecture is as follows:
What are its consequences?
Fowler's and on this score - about the "payback" for the use of microservices:
And we'll see if our expectations were met.
Clear boundaries of modules ...
But how many microservices do we actually need to fixto roll out the change? Can we even figure out how everything works without a distributed tracer (after all, any request is processed by half of the microservices)?
There is a pattern "”, and here it turned out to be a distributed lump of dirt. In confirmation of this, here is an approximate illustration of how requests go:
Deploy independence…
Technically, it has been achieved: we can roll each microservice separately. But in practice, you need to consider that it always rolls out many microservices, and we need to take into account order of their release. In a good way, we generally need to test in a separate circuit whether we are rolling out the release in the correct order.
Freedom of choice of technology…
She is. It is only worth remembering that often freedom borders on lawlessness. It is very important here not to choose technologies just to “play” with them.
Design Independence…
How to make a test circuit for the entire application (from so many components)? And yet it still needs to be kept up to date. All this leads to the fact that actual number of test circuits, which we can in principle contain, turns out to be minimal.
How about deploying all this locally?.. It turns out that often the developer does his work independently, but “at random”, because he has to wait until the circuit is free for testing.
Separate scaling…
Yes, but it is limited in the scope of the used DBMS. In the given architecture example, Cassandra will not have problems, but MySQL and PostgreSQL will.
Боgreater reliability...
Not only that, in reality, the failure of one microservice often breaks the correct functioning of the entire system, but there is also a new problem: making every microservice fault-tolerant is very difficult. Because microservices use different technologies (memcache, Redis, etc.), each needs to be thought out and implemented, which, of course, is possible, but requires huge resources.
load measurement...
This is really all right.
The “lightness” of microservices…
We not only have huge network overhead (requests for DNS, etc. are multiplying), but due to the many subrequests, we started replicate data (store caches), which resulted in a significant amount of storage.
And here is the result of meeting our expectations:
But that's not all!
Because:
- Most likely we will need a message bus.
- How to make a consistent backup at the right time? The only one real option is to disable traffic for this. But how to do it in production?
- If we are talking about supporting several regions, then organizing sustainability in each of them is a very time-consuming task.
- There is a problem of making centralized changes. For example, if we need to update the PHP version, then we need to make a commit to each repository (and there are dozens of them).
- The growth of operational complexity offhand turns out to be exponential.
What to do with all this?
Start with a monolithic application. Fowler's experience that almost all successful microservice applications started with a monolith that got too big and then broke. At the same time, almost all systems built as microservices from the very beginning experienced serious problems sooner or later.
Another valuable thought is that for a project with a microservice architecture to be successful, you must know very well and subject area, and how to make microservices. And the best way to learn the subject area is to make a monolith.
But what if we are already in this situation?
The first step to solving any problem is to accept it and understand that it is a problem, that we do not want to suffer anymore.
If, in the case of an overgrown monolith (when we have run out of the opportunity to buy additional resources for it), we cut it, then in this case the opposite story turns out: when excessive microservice no longer helps, but hinders - cut off the excess and enlarge!
For example, for the collective image discussed above ...
Get rid of the most dubious microservices:
Combine all the microservices responsible for generating the frontend:
... into one microservice written in one (modern and normal, as you yourself think) language / framework:
It will have one ORM (one DBMS) and first a couple of applications:
... but in general, you can transfer much more there, getting the following result:
And in Kubernetes, we run all this in separate instances, which means that we can still measure the load and scale them separately.
Summarizing
Look at the bigger picture. Very often, all these problems with microservices arise from the fact that someone took their task, but wanted to "play microservices".
In the word "microservices", the part "micro" is superfluous. They are "micro" only for the reason that they are smaller than a huge monolith. But don't think of them as something small.
And for the final thought, let's go back to the original schedule:
Note written on it (top right) comes down to the fact that the skills of the team that makes your project are always primary - they will play a key role in your choice between microservices and a monolith. If the team lacks skills, but it starts doing microservices, the story will definitely be fatal.
Videos and slides
Video from the speech (~50 minutes; unfortunately, it does not convey the numerous emotions of the visitors, which largely determined the mood of the report, but as it is):
Report presentation:
PS
Other reports on our blog:
- «» (Dmitry Stolyarov; May 28, 2018 at RootConf);
- «» (Dmitry Stolyarov; November 7, 2017 on HighLoad++);
- «» (Dmitry Stolyarov; June 6, 2017 at RootConf);
- «» (Dmitry Stolyarov; November 8, 2016 on HighLoad++);
- «» (Dmitry Stolyarov; May 31, 2016 at RootConf).
You may also be interested in the following publications:
- «";
- «";
- «».
Source: habr.com