🥇Legacy Services in Your Infrastructure

Hello! My name is Pasha Chernyak, I’m a lead developer at QIWI, and today I want to talk about the inevitable: Legacy.

Let's start with the question: what is a Legacy service? Is it a service that the developer has not touched for a week/month/year? Or is it a service that was written by a less experienced programmer, for example, specifically you, but a year ago? And now you're cooler and more experienced. Or is a Legacy service something you decided to never commit to again and are gradually preparing a replacement for? In any case, leaving such a service unattended and not updating it is a ticking time bomb that could explode later.

Before we move on to how we at QIWI work with our Legacy services, I'll tell you how we organized the services in Wallet. I have been responsible for its functionality for two years now. If there is any problem, I always get the first call. I usually don't have the audacity to call someone else at 11 PM, so I end up sitting down to figure out all the services in our domain.

But like any person, I love to sleep at night, so I tried to figure out the operation: 'Guys, why are you calling me?' To which I received a succinct response like 'Who else?'. Because I fix services, and the guys simply don’t know who else to call.

So, at one of the backend team retrospectives for Wallet, we decided that we needed to create a table listing our wallet services, microservices, and monoliths, along with the people responsible for them. Tables are generally useful within reasonable limits.

In addition to the information about who is responsible for what, there were answers to questions: who owns the service, who is responsible for its development, architecture, and lifecycle. The people responsible for this service are those who can fix it if needed. The service owner has the right to leave +2 in commits, and the responsible parties must also be present at the review before this service accepts a new commit.

As time went on, new practices began to be adopted, such as migration to Kubernetes, various checkstyle, spotbugs, ktlint, having logs in Kibana, and autodiscovery of services instead of specifying addresses directly, among other useful features. Our table allowed us to maintain the relevance of our services everywhere. For us, it serves as a checklist indicating that this service can perform certain tasks, while there are others it cannot yet do. But we continued to evolve, realizing that we were missing information about our services, such as where the service source code is located, where build tasks are launched in TeamCity, how they are deployed, where the end-to-end test sources are stored, architectural grooming photos, and decisions made. Ideally, we wanted all this information to be readily available when needed. Therefore, our table became the starting point for information retrieval.

However, QIWI, while still preserving the spirit of a startup, is a large company. We’ve been around for 12 years, and teams change: people leave, new people join, and new teams are formed. We found several services on our domain that were inherited from previous developers. Some came from developers from other teams, and some were indirectly related to the Wallet, which is why we now have that service on our balance. Why investigate how everything works? The service works, and we have product features that need to be implemented.

As it happens

But at some point, we discover that the service stops performing its function, something is broken — what to do in such a situation? The service simply ceased to work. Completely. We found out about it, first by chance, and secondly, after six months. It happens. The only thing we knew was which virtual machines the service was running on and where its source code was located. We do a git clone and immerse ourselves in the thoughts of the person who wrote this several years ago, but what do we see? No familiar Spring Boot, even though we are used to it; we are full stack and all that. Maybe there is Spring Framework? But no.

The guy who wrote all this was harsh and wrote everything in pure Java. There are no familiar tools for the developer, and the idea arises — we should rewrite it all. After all, we have microservices, and from every toaster comes the familiar message, “Guys, microservices are what you need!” If something goes wrong, you can easily pick any language and everything will be fine.

The thing is, right now we don’t have a client who is responsible for this service. What were their business requirements, what is this service supposed to do? And the service is tightly integrated into your business processes.

Now tell me, how easy is it to rewrite a service without knowing its business requirements? It’s unclear how the service logs, whether there are metrics — that’s unknown. What they might be, if they exist — that’s even more uncertain. And at the same time, the service has a huge number of classes with unclear business logic. Some of it goes into some database, which we also don’t know anything about yet.

Where do we start?

With the most logical thing — the presence of tests. There’s usually some kind of logic written there, and we can draw conclusions about what's happening. Right now, TDD is fashionable, but we see that the same situation existed 5 years ago: there are almost no unit tests, and they won’t tell us anything significant. Except perhaps some verification of how some xml is signed with some custom certificate.

We couldn't figure out anything from the code, so we decided to check what was wrong in the virtual machine. We opened the service logs and found an error with the HTTP client: a self-signed certificate embedded in the application's resources had expired. We contacted our analysts; they requested a new certificate, which was issued, and the service is now operational again. One would think that's the end of it. But is it really? The service is operational, performing some function necessary for our business. We have certain application development standards, which you likely have as well. For instance, logging should not be stored on the node in a folder but should go to some storage, like Elasticsearch, and be viewed in Kibana. We can also recall the golden metrics. This includes service load, the number of requests to the service, whether it's live or not, and how the HealthCheck is performing. At the very least, these metrics will help determine when it can be safely decommissioned and forgotten like a bad dream.

What to do

Therefore, we add such an old service to a table and then look for volunteer developers to take charge of it and bring it up to standard: they will write at least some information about the service, add links to dashboards in Grafana, and tasks for deployment, because we can't just upload files using FTP manually.

The main question is, how much time will all this volunteer activity take? One sprint for a reasonably experienced developer, for example, during a 20% technical debt period. But how much time did it take to understand all the embedded logic for communicating with some government system and upgrade it to newer technologies? I can't guarantee this — it could be a month, or maybe two months of team work. I'm speaking from experience with integrations with new services at this time.

In the end, the business value output from this is none whatsoever. Absolutely none. Taking service support and spending a little time on it is normal. But after our typical struggles with the service, we added it to the table, included information about it, and perhaps, someday we will rewrite it. But for now, it meets our service operation standards.

As a result, I would like to propose a plan for what to do with legacy services.

Rewriting legacy from scratch is a bad idea.
Seriously, you don't even need to think about it. It's clear that it would be nice, and some advantages are seen, but usually, it's not needed by anyone, including yourselves.

Guide
Uncover the source codes of your applications, create a guide that indicates what and where it is located and how it works, and include a project description (a sort of readme.md) so you can quickly understand where the logs and metrics are. The developer who will be dealing with this after you will thank you.

Understand the Domain
If you own a domain, try to stay on top of things. It sounds cliché, yes, but not everyone ensures that services are unified. Yet working in one standard is actually significantly easier.

Only registered users can participate in the survey. Please log in, please.