🥇Microservices: size matters, even if you have Kubernetes

September 19 in Moscow the the first thematic meetup of HUG (Highload++ User Group), which was dedicated to microservices. The report "Operating Microservices: Size Matters, Even if You Have Kubernetes" shared extensive experience from the company "Flant" in operating projects with microservices architecture. This will primarily benefit all developers contemplating the application of this approach in their current or future projects.

Introducing the video of the presentation (50 minutes, significantly more informative than the article), as well as the main highlights in text form.

NB: The video and presentation are also available at the end of this publication.

Introduction

Usually, a good story has a beginning, main plot, and conclusion. This presentation resembles more of a tragic beginning. It is also essential to note that it presents a perspective on microservices from exploitation.

I will start with this chart, created by became Martin Fowler in 2015:

It shows how, in the case of a monolithic application reaching a certain size, productivity starts to decline. Microservices differ in that their initial productivity is lower, but as complexity grows, the degradation in efficiency is less noticeable.

I will supplement this chart for the case of using Kubernetes:

Why has the application with microservices improved? Because this architecture imposes serious requirements that are excellently met by Kubernetes capabilities. On the other hand, some of this functionality will also be beneficial for monoliths, especially since today’s typical monolith is not entirely monolithic (details will follow later in the presentation).

As can be seen, the final chart (when both monolithic and microservice applications are in a Kubernetes infrastructure) does not differ much from the initial one. The focus will be on applications operated using Kubernetes.

Useful and harmful microservices

And here is the main thought:

What is a normal microservices architecture? It should deliver real benefits by increasing work efficiency. Referring back to the chart, there it is:

If we call it beneficial, then on the other side of the chart will be harmful microservices (hinders work):

Returning to the 'main point': is it worth trusting my experience? Since the beginning of this year, I have reviewed 85 projects. Not all of them were microservices (about a third to half had such architecture), but that's still a large number. We (the company 'Flant') as outsourcers manage to see a wide variety of applications developed both in small companies (with 5 developers) and in large ones (~500 developers). An additional plus is that we see how these applications live and evolve over many years.

Why microservices?

There is a quite specific answer from the aforementioned Martin Fowler:

clear boundaries of modularity;
independent deployment;
freedom of technology choice.

I have talked a lot with architects and software developers, asking them why they need microservices. I compiled my list of their expectations. Here’s what I came up with:

If I were to describe 'in sensations' some of the points, they would be:

clear boundaries of modules: we have this terrible monolith, and now everything will be neatly organized in Git repositories, where everything is 'sorted out' without mixing the warm with the soft;
independent deployment: we will be able to roll out services independently, enabling faster development (simultaneously launching new features);
independent development: we can give this microservice to one team/developer while another team handles a different one, allowing us to develop faster;
bhigher level of isolation, as if one controller is broken, the problem is confined to that specific context).greater reliability: if there is partial degradation (one microservice out of 20 goes down), only one button stops working, while the system as a whole continues to function.

Typical (harmful) microservice architecture

To explain why in reality everything is not as we expect, I will present a composite image of microservice architecture, based on experiences from many different projects.

An example will be an abstract online store aiming to compete with Amazon or at least OZON. Its microservice architecture looks like this:

For a variety of reasons, these microservices are written on different platforms:

Since each microservice needs to be autonomous, many of them require their own database and cache. The final architecture turns out to be as follows:

What are its consequences?

Fowler has an article on this matter about the 'payment' for using microservices: We'll see if our expectations were met.

Clear boundaries of modules…

how many microservices do we actually need to fix

However, , in order to roll out a change? Can we even figure out how everything works without a distributed tracer (after all, any request is handled by half of the microservices)?There exists a pattern of a '

big ball of mud', and here we have an even more distributed ball of mud. In confirmation of this — here's a rough illustration of how requests flow:Independence of deployment…

Technically it is achieved: we can roll out each microservice separately. But in practice, we must consider that many microservices are always rolled out

, and we need to take into account the order of their rollout.Ideally, we need to test in a separate environment whether we are rolling out the release in the correct order. Freedom of technology choice…It exists. Just keep in mind that often freedom borders on chaos. It's very important not to choose technologies just to 'play' with them.

Independence of development…

How to create a test environment for the entire application (from such a multitude of components)? And we also need to keep it up to date. All this leads to

the actual number of test environments

, which we can support, turns out to be minimal.And deploying all of this locally?.. It turns out that often the developer does their work independently, but 'randomly', because they have to wait for a testing environment to become available. Separate scaling….

Yes, but it is limited in the types of databases used. In the architecture example given, there will be no problems with Cassandra, but there will be with MySQL and PostgreSQL.

G

reater reliability…

Not only does the failure of one microservice often break the correct functioning of the entire system, but there’s also a new problem:higher level of isolation, as if one controller is broken, the problem is confined to that specific context).making each microservice fault-tolerant is very difficult.

Because different technologies (memcache, Redis, etc.) are used in microservices, for each one you have to think through and implement everything, which, of course, is possible but requires enormous resources. Making each microservice fault-tolerant is very challenging. This is because different technologies (memcache, Redis, etc.) are used in microservices, and each one requires careful planning and implementation, which is certainly possible but demands significant resources.

Measurability of load…

Everything is indeed going well with this.

The 'lightness' of microservices…

Not only have we faced huge network overheads (with increased DNS requests, etc.), but due to numerous sub-requests, we have also started replicating data (storing caches), which has led to a significant amount of storage.

And here's how it aligns with our expectations:

But that's not all!

Because:

We will likely need a message bus.
How to create a consistent backup at a specific point in time? The only real option is to stop traffic for this. But how to do it in production?
If we are talking about supporting multiple regions, organizing resilience in each of them is a very labor-intensive task.
The problem of making centralized changes arises. For example, if we need to update the PHP version, a commit will be required in each repository (and there are dozens of them).
The increase in operational complexity seems to be exponential.

What to do with all this?

Start with a monolithic application. Fowler's experience says shows that almost all successful microservice applications started as monoliths that became too large and were then broken up. At the same time, almost all systems built as microservices from the outset faced serious issues sooner or later.

Another valuable insight is that for a microservices architecture project to be successful, you must know very well both the domain area and how to create microservices. The best way to learn the domain area is to create a monolith.

But what if we have already found ourselves in this situation?

The first step to solving any problem is to acknowledge it and understand that it is a problem we no longer want to suffer from.

If in the case of an overgrown monolith (when we have run out of resources to purchase for it), we cut it, here we have the opposite situation: when excessive microservices are no longer helpful, but hinder— cut out the excess and consolidate!

For example, for the aforementioned composite image…

Eliminate the most questionable microservices:

Consolidate all microservices responsible for generating the frontend:

… into a single microservice, written in one (modern and proper, as you see it) language/framework:

It will have one ORM (one DBMS) and initially a couple of applications:

… but in fact, much more can be transferred, achieving such a result:

Moreover, in Kubernetes, we launch all of this as separate instances, which means we can still measure the load and scale them independently.

In summary

Look at the bigger picture. Very often, all these problems with microservices arise because someone took their task but wanted to 'play with microservices.'

In the word 'microservices', the part 'micro' is unnecessary.They are 'micro' only because they are smaller than a huge monolith. But you shouldn’t think of them as something small.

And for the final thought, let’s return to the initial graph:

The note written to it (top right) boils down to the fact that the skills of the team working on your project are always primary — it is they who will play a key role in your choice between microservices and a monolith. If the team lacks skills but starts creating microservices, the story will certainly end tragically.

Videos and slides

Video of the presentation (~50 minutes; unfortunately, it does not convey the numerous emotions of the attendees, which largely defined the mood of the report, but here it is):