Andrey Nikolsky, Director of Operations at Banki.ru, spoke at last year's conference about orphan services: how to identify an orphan in infrastructure, the issues with orphan services, what to do with them, and how to handle the situation if nothing helps.
Below is the text version of the report.
Hello, colleagues! My name is Andrey, and I manage operations at Banki.ru.
We have large services, which are monolithic services; we have services in a more classical sense, and we also have very small ones. In my workmanlike terminology, I say that if a service is simple and small, it’s micro, and if it’s not very simple and not small, it’s just a service.
Benefits of services
I'll quickly run through the benefits of services.
First — scalability. You can quickly do something on a service and launch it into production. When traffic comes in, you clone the service. When more traffic arrives, you clone again and live with that. This is a nice bonus, and, in principle, when we started, this was considered the most important reason for why we do all this.
Second, isolated development, where you have several development teams with different developers in each team, and each team is working on its own service.
There is a nuance with teams. Developers can be different. For example, there are . I first saw this concept from Maxim Dorofeev. Sometimes snowflake people exist in some teams, while in others, they do not. This makes the various services used in the company somewhat uneven.
Look at the picture: this is a good developer, he has big hands, he can do a lot. The main problem is where those hands come from.
Services allow the use of different programming languages that are better suited for different tasks. Some services are in Go, some in Erlang, some in Ruby, some in PHP, and some in Python. Overall, it’s possible to expand very broadly. There are nuances here as well.
Service-oriented architecture is primarily about DevOps. This means if you lack automation and deployment processes, if you're configuring manually, your configurations may vary from service instance to instance, and you have to intervene and make adjustments, then you're in trouble.
For instance, if you have 20 services and need to deploy manually, you'd end up with 20 consoles, pressing 'enter' simultaneously like a ninja. That's not ideal.
If your service is after testing (if there's any testing, of course) and still needs further tweaks to work in production, I have bad news for you.
If you rely on specific Amazon services and are working in Russia, two months ago you too might have thought, 'Everything is burning around me, I'm fine, all is well.'
We use Ansible for deployment automation, Puppet for convergence, Bamboo for deployment automation, and Confluence to document everything somehow.
I won't dwell on this in detail because this talk is more about interaction practices, rather than technical implementation.
There have been instances where Puppet on the server runs with Ruby 2, while an application is written for Ruby 1.8, and they won't work together. There’s some issue that arises. When you need to maintain multiple Ruby versions on one machine, you usually start facing problems.
For example, we provide each developer with a platform that contains approximately everything we have, all services that can be developed, so they have an isolated environment where they can break and build things as they wish.
Sometimes, you need a specially compiled package with support for something. That's quite strict. I listened to a talk where a Docker image weighs 45 GB. Of course, it's easier on Linux, as everything is smaller, but still, there won’t be enough space.
Additionally, there are conflicting dependencies when one part of your project depends on a library of one version, while another part relies on a different version, and the libraries cannot coexist at all.
We have websites and services running on PHP 5.6, and we're ashamed of them, but what can we do? This is one of our platforms. There are more websites and services on PHP 7, and we aren't ashamed of them. Each developer has their own database where they happily tinker.
If you are working in a company with a single programming language, then three virtual machines per developer sounds reasonable. However, if you have different programming languages, the situation gets worse.
You end up having websites and services on this, on that, then another platform for Go, one for Ruby, and some Redis on the side. All this turns into a vast support landscape, and something can break at any time.
That's why we replaced programming language perks with the use of different frameworks, as PHP frameworks vary significantly, with different capabilities, communities, and support. You can build a service so that you already have something ready-made for it.
Each service has its own team
Our main advantage, which crystallized over several years, is that each service has its own team. This is convenient for a large project, allowing time to be saved on documentation, and managers know their projects well.
Support tasks can be efficiently funneled. For example, if the insurance service breaks down, the team handling insurance immediately steps in to fix it.
New features are quickly developed because when you have a single atomic service, you can swiftly integrate something into it.
And when you break your service, which inevitably happens, you don't affect other services, and developers from other teams don't come running at you with bats saying, 'Oh no, don't do that.'
As always, there are nuances. We have stable teams, and the managers are firmly anchored to their teams. There are clear documents, and managers closely monitor everything. Each team with a manager has several services, and there is a specific area of expertise.
If teams are fluid (which we sometimes use), there's a good method called 'star map.'
You have a list of services and people. A star means that a person is an expert in that service, while a book means that the person is studying that service. The goal of that person is to replace the book with a star. If there's nothing written next to a service, then problems start, which I will discuss further.
How do orphan services arise?
The first problem, the first way to have an orphan service in your infrastructure is employee layoffs. Has anyone ever experienced deadlines coming down from business before tasks were assessed? Sometimes the timelines are tight, and there's simply not enough time for documentation. "We need to launch the service in production, and we'll finish the documentation later."
If the team is small, it sometimes happens that there’s one developer who writes everything, while others are on standby. "I wrote the main architecture; you go ahead and sketch the interfaces." Then at some point, the manager may leave. During that period when the manager has left and a new one hasn't been appointed yet, developers start to decide where the service is headed and what is happening with it. And as we know (let's go back a few slides), in some teams, there are snowflake individuals, sometimes a snowflake team lead. Then they resign, and we end up with an orphan service.
At the same time, the tasks from support and from the business don't go anywhere; they accumulate in the backlog. If there were any architectural mistakes during the service development, they also settle in the backlog. The service slowly degrades.
How to identify an orphan service?
This list describes the situation fairly well. Has anyone recognized anything in their infrastructure?
Regarding documented workarounds: there is a service, and generally, it works, it has a two-page manual on how to use it, but no one knows how it works internally.
Or, for example, there’s a URL shortener. We currently use three different URL shorteners for various purposes across different services. These are the consequences.
Now I'll be Captain Obvious. What needs to be done? First, the service needs to be handed over to another manager or a different team. If your team lead hasn’t quit yet, you should include someone in that other team who understands at least something about the service when you realize the service is like an orphan.
The main thing: you must have well-documented handover procedures. In our case, I usually keep an eye on this because I need it all to work. Managers want it to be handed over quickly, and what happens to it afterward is less important to them.
The next way to create an orphan is, "Let's outsource it; it will be faster, and then we'll hand it over to the team." It's clear that everyone has certain plans in the team, there's a queue. Often, the business client thinks that the outsourcer will do things just like the tech department within the company, even though their motivations are different. Outsourcing can lead to strange technological and algorithmic solutions.
For instance, we had a service where Sphinx was used in various unexpected places. I'll tell you later what had to be done.
Outsourcers often have custom-made frameworks. It's just bare PHP with copy-pasting from a previous project, where you can find all sorts of things. There are large hacks in the deployment scripts, where you need to use complex Bash scripts to change a few lines in some file, while those deployment scripts are invoked by some third script. As a result, when you change the deployment system and choose something else, your service stops working. Because you needed to place another 8 links between different folders. Or sometimes it works with a thousand records but fails with a hundred thousand.
I'll continue to steer the ship. Accepting a service from outsourcing is a procedure that is mandatory. Has anyone ever experienced a situation where a service comes from outsourcing, but it’s not accepted anywhere? It’s certainly not as popular as the orphan service, but still.
Services need to be checked, they need reviewing, and passwords need to be changed. We once received a service where the admin interface had "if login == 'admin' && password == 'admin'..." written directly in the code. We sat there thinking, and this was written by people in 2018?
Testing storage capacity is also necessary. You need to see what will happen with a hundred thousand records before you launch this service into production.
It should not be embarrassing to send a service for rework. When you say, "We will not accept this service; we have 20 tasks, get them done, then we'll accept it," that is normal. Your conscience should not ache from the thought that you are putting a manager in a tough spot or that the business will spend money. The business will end up spending more later.
We had a case where we decided to conduct a pilot project through outsourcing.
It was delivered on time, and that was the only quality criterion. Therefore, another pilot project was initiated, not quite a pilot anymore. These services were accepted, and administratively it was said, here's your code, here’s your team, here’s your manager. The services have really started to generate profit. Yet, in reality, they remain orphans, no one understands how they work, and managers are completely distancing themselves from the tasks associated with them.
There's another great concept — guerrilla development. When a certain department, usually marketing, wants to test a hypothesis, they order a service entirely outsourced. It starts receiving traffic, they finalize the documents, sign agreements with the contractor, it goes into operation, and they say: “Guys, we have this service, it’s already getting traffic, it’s making us money, let's accept it.” We respond: “Wait, how did that happen?”.
And there's another way to end up with an orphan service: when a team suddenly becomes overloaded, the management says: “Let’s transfer this service from this team to another team that has a lighter load.” Then we hand it over to a third team and change the manager. And in the end, we again have an orphan.
What is the problem with orphans?
For those who don't know, it’s the raised warship Vasa from Sweden, famous for sinking just five minutes after its launch. And the King of Sweden, by the way, did not execute anyone for it. It was built by two generations of engineers who didn’t know how to build such ships. A predictable effect.
The ship could have sunk much worse, for instance, if the king were on board during a storm. But it sank immediately, which, from an Agile perspective, is good — failing early.
If we fail early, there are usually no problems. For example, during the acceptance phase, it gets sent for revisions. But if we fail after going into production, when money has been invested, then there can be issues. Consequences, as they are called in business.
What are the dangers of orphan services:
- The service may suddenly break down.
- The service takes a long time to fix or doesn’t get fixed at all.
- Security issues.
- Problems with improvements and updates.
- If an important service breaks down, the company's reputation suffers.
What to do with orphan services?
I will reiterate what needs to be done. First, there must be documentation. Seven years at Banki.ru taught me that testers should not take developers' words at face value, and operations shouldn't trust everyone blindly. Everything needs to be verified.
Second, interaction schemes need to be documented, as sometimes services that are not well-understood have dependencies that no one mentioned. For example, developers may connect a service to their own API keys for Yandex.Maps or Dadata. Once your free limit runs out, everything breaks, and you have no idea what happened. All such pitfalls should be documented: the service uses Dadata, SMS, and so on.
Third, handling technical debt. When you create workarounds or adopt a service and say that something needs to be done, you must ensure that it gets done. Otherwise, a small issue can turn into a big problem, and you might end up in a deeper hole.
We had a case with architectural tasks involving Sphinx. In one of the services, Sphinx was used to input lists. Just a simple list with pagination, but it was re-indexed every night. It was composed of two indices: one large index that was built every night and a smaller one that was combined with it. Each day, there was a 50% chance it would fail during deployment, and news updates would stop appearing on the homepage. Initially, this took 5 minutes while the index was rebuilt, but as the index grew, it began taking up to 40 minutes. When we eliminated it, we breathed a sigh of relief because it became clear that soon our index would be re-indexing for an entire working day. That would be a failure for our portal: eight hours without news would halt the business.
Plan for working with orphan services
In reality, this is very difficult to achieve because DevOps is all about communication. You want to maintain good relationships with your colleagues, but when you hit your coworkers and managers over the head with regulations, they may develop mixed feelings towards those who do this.
In addition to all these points, there's one more important thing: specific people must be responsible for each particular service and each specific part of the deployment process. When those people are unavailable and you have to bring in someone else to study the matter, it becomes challenging.
If all this hasn't helped and you still have a service without a caretaker that no one wants to adopt, with no documentation being written, and the team assigned to this service refusing to do anything, there's a straightforward solution — start over.
In other words, you take the service requirements again and write a new service, a better one, on a superior platform, without any strange technological solutions. Then you migrate to it in real-time.
We had a situation where we took a service built on Yii 1 and realized that we couldn't further develop it because our developers experienced with Yii 1 had run out. All developers were proficient in Symfony 3. So, what did we do? We allocated time, assembled a team, appointed a manager, rewrote the project, and gradually redirected traffic to it.
After that, the old service can be deleted. This is my favorite procedure: taking a service from configuration management and cleaning it up, then making sure all instances in production are shut down, leaving no traces for the developers. The repository in Git remains.
That's all I wanted to share, I'm open for discussion; it's a controversial topic and many have dabbled in it.
The slides mentioned that you standardized the programming languages. An example given was image resizing. But is it really necessary to stick to just one language? Because resizing an image in PHP could indeed also be done in Golang.
In fact, it's not mandatory, just like all practices. In some cases, it may even be undesirable. However, you must understand that if your company has 50 people in the tech department, of which 45 are PHP developers, 3 are devops skilled in Python, Ansible, Puppet, and something similar, and only one of them is writing a Go service for resizing images, then when he leaves, expertise goes with him. At the same time, you will need to look for a specific developer on the market who knows this language, especially if it's rare. So from an organizational standpoint, this is problematic. From a devops perspective, you won't just clone a ready-made set of playbooks that you use to deploy services; you will have to write them from scratch.
We are currently developing a service on Node.js, and this will be a platform alongside each developer with a separate language. But we thought about whether it's worth the effort. So, it's a matter of sitting down and thinking it over.
How do you monitor your services? How do you collect and track logs?
We collect logs in Elasticsearch and store them in Kibana, and depending on whether it’s production or testing environments, different log shippers are used. Some use Lumberjack, others something else; I can't remember. There are also certain places in particular services where we install Telegraf and send logs to other locations separately.
How do you live with Puppet and Ansible in the same environment?
Actually, we currently have two environments, one for Puppet and another for Ansible. We are working on hybridizing them. Ansible is a good environment for initial setup; Puppet is not great for initial setup because it requires hands-on work with the platform, and Puppet ensures configuration convergence. This means that the platform maintains itself in an up-to-date state, while an Ansible-managed machine needs to have playbooks run on it regularly to stay updated. That's the difference.
How do you maintain compatibility? Do you have configurations in both Ansible and Puppet?
This is our major pain point; we manually maintain compatibility and think about how to transition away from all this. Currently, Puppet deploys packages and maintains some links, while Ansible, for example, deploys code and adjusts the fresh application configs.
The presentation covered different Ruby versions. What is the solution?
We've encountered this in one place, and we constantly have to keep it in mind. We simply disabled the part that was running on the Ruby version incompatible with the applications and kept it separate.
This year's conference will take place on December 7 at 'Technopolis'. We are accepting submissions for presentations until November 11. us if you want to speak.
Registration for participants is open, join us!
Source: habr.com
