On April 25, we at Mail.ru Group held a conference about clouds and around - . A few highlights:
- On the same stage gathered the main — Mail.ru Cloud Solutions, #CloudMTS, SberCloud, Selectel, Rostelecom Data Center and Yandex.Cloud talked about the specifics of our cloud market and their services;
- Colleagues from Bitrix24 told how they ;
- Leroy Merlin, Otkritie, Burger King and Schneider Electric provided an interesting - what tasks their business sets for IT and what technologies, including cloud ones, they see as the most promising.
All videos from the mailto:CLOUD conference can be viewed , and here you can read how the discussion about microservices went. Alexander Deulin, head of the research and development center for business systems at MegaFon, and Sergey Sergeev, director of information technology at the M.Video-Eldorado group, shared their successful cases of getting rid of monoliths. They also discussed related issues of IT strategy, processes and even HR.
Participants of the discussion
- Sergey Sergeev, Group Chief Information Officer "M.Video-Eldorado";
- Alexander Deulin, Head of the Center for Research and Development of Business Systems MegaFon;
- Moderator - Dmitry Lazarenko, Head of PaaS Mail.ru Cloud Solutions.
After Alexander Deulin's speech Sergey Sergeev from M.Video-Eldorado and discussion moderator Dmitry Lazarenko, Mail.ru Cloud Solutions join him for discussion.
Below we have prepared a transcript of the discussion for you, but you can also watch the video:
Transition to microservices - response to market needs
Dmitriy:
Have you had a successful experience of migrating to microservices? And in general: where do you see the greatest business benefit from using microservices or moving from monoliths to microservices?
Sergei:
We have already gone some way in migrating to microservices and have been using this approach for more than three years. The first need that justified the need for microservices was the endless integration of various front-end products with the back-office. And each time we were forced to do additional integration, development, implementing our own rules for the operation of a particular service.
At some point, we realized that we needed to speed up the operation of our systems and the output of functionality. At that moment, such concepts as microservices, a microservice approach, already existed on the market, and we decided to try it. It started in 2016. Then the platform was laid and the first 10 services were implemented by a separate team.
One of the first services, the most heavily loaded, was the price calculation service. Every time you visit any channel, the M.Video-Eldorado group of companies, be it a website or a retail store, pick up a product there, see the price on the website or in the "Cart", the cost is automatically calculated by one service. Why this is needed: before that, each system had its own principles for working with promos - with discounts and so on. Our back office deals with pricing, the discount functionality is implemented in another system. It had to be centralized and create such a unique, separable service in the form of a business process that would allow us to implement it. That's pretty much how we got started.
The value of the first results was very great. Firstly, we were able to make separable entities that allow us to work separately, aggregated. Secondly, we have reduced the cost of ownership in terms of integration with more systems.
Over the past three years, we have added three front-line systems. They were difficult to maintain with the same amount of resources that the company could afford. Therefore, the task arose to look for new exits, responding to the market in terms of speed, in terms of internal costs and efficiency.
How to evaluate the success of the transition to microservices
Dmitriy:
How is success in migrating to microservices determined? What was "before" in each company? What metric did you use to determine the success of the transition, who actually determined it?
Sergei:
First of all, it was born inside IT as an enabler - “unlocking” new features. We had a need to do everything faster for the same amount of money, responding to the challenges of the market. Now success is expressed in the number of services reused by different systems, the unification of processes among themselves. Now it is, but at that moment it was an opportunity to create a platform and confirm the hypothesis that we can do it, this will give an effect and calculate the business case.
Alexander:
Success is more of an inner feeling. Businesses always want more, and the depth of our backlog is proof of our success. It seems so to me.
Sergei:
Yes, I agree. For three years, we have already had more than two hundred services and backlogs. The need for resources within the team is only growing - by 30% annually. This happens because people felt: it's faster, different, other technologies, all this is developing.
Microservices will come, but the core will remain
Dmitriy:
It's like a never ending process where you invest in development. Is the transition to microservices for business already over or not?
Sergei:
It's very easy to answer. Do you think that replacing phones is an endless process? We ourselves buy phones every year. And here it is: as long as there is a need for speed, in adapting to the market, some changes will be required. This does not mean that we refuse standard things.
But we cannot immediately embrace and redo everything. We have legacy, standard integration services that we had before: enterprise buses and so on. But there is a backlog, and there is a need too. The number of mobile applications and their functionality is growing. At the same time, no one says that you will be given 30% more money. That is, on the one hand, there are always needs, and on the other, the search for efficiency.
Dmitriy:
Life in good shape. (Laughs)
Alexander:
In general, yes. We do not have revolutionary approaches to removing the core part from the landscape. Systematic work is underway to decompose systems so that they are more in line with the microservice architecture, to reduce the influence of systems on each other.
But we plan to keep the core part, as there will always be some platforms that we buy in the operator landscape. Again, a healthy balance is needed: we should not rush into cutting out the core. We put the systems side by side, and now it turns out that they are already above many core parts. Further, developing the functionality, we make the necessary representations for all channels that work with our communication services.
How to sell microservices to a business
Dmitriy:
I'm still wondering - for those who have not switched, but are going to: how easy was it to sell this idea to a business and was it a gamble, an investment project? Or it was a conscious strategy: now we are going into microservices and that's it, nothing will stop us. How was it for you?
Sergei:
We were not selling an approach, but a business benefit. There was a problem in business, and we tried to remove it. At that moment, it was expressed in the fact that different channels used different principles for calculating the price - separately for promotions, for promotions, and so on. It was difficult to maintain, there were errors, we listened to complaints from customers. That is, we sold the elimination of the problem, but came with the fact that we needed money to create a platform. And they showed a business case using the example of the first stage of investment: how we will continue to pay for it and what it will allow us to do.
Dmitriy:
Did you somehow fix the time of the first stage?
Sergei:
Yes, sure. We took 6 months to create the core as a platform and test the pilot. During this time, we tried to create a platform on which the pilot skated. Then the hypothesis was confirmed, and since it works, then we can continue. They began to replicate and strengthened the team - they brought it to a separate unit, which does just that.
Next comes systematic work based on the needs of the business, opportunities, availability of resources and everything that is currently in the works.
Dmitriy:
OK. Alexander, what do you say?
Alexander:
Our microservices were born from the "foam of the sea" - due to saving resources, due to some leftovers in the form of server capacities and the redistribution of forces within the team. Initially, we did not sell this project to the business. It was a project where we both researched and developed accordingly. We started at the beginning of 2018 and developed this direction simply on enthusiasm. Sales have just started and we are in the process.
Dmitriy:
Does it happen that a business allows you to do such things according to the Google principle - on one free day a week? Do you have such direction?
Alexander:
Simultaneously with research, we also dealt with business problems, therefore all our microservices are a solution to business problems. Only at the beginning we built microservices that cover a small part of the subscriber base, and now we are in almost all flagship products.
And the material impact is already clear - we can already be counted, we can estimate the speed of output of products and lost revenue if we went the old way. This is where we build the case.
Microservices: hype or necessity?
Dmitriy:
Numbers are numbers. And the revenue or money saved is very important. What if you look at the other side? It seems that microservices are a trend, a hype and many companies are abusing this? How clearly do you distinguish between what you are translating and not translating into microservices? If legacy is now, will you still have a legacy in 5 years? What will be the age of the information systems that work in M.Video-Eldorado and MegaFon in 5 years? Will there be ten-year-old, fifteen-year-old information systems or will it be a new generation? How do you see it?
Sergei:
It seems to me that it is very difficult to guess far. If you look back, then who expected that the market for technologies and the same machine learning, user identification by face would develop in this way? But if you look at the coming years, it seems to me that core-systems, enterprise ERP-class systems in companies - they have been working for a long time.
Our companies are collectively 25 years old, where classic ERP is very deep in the systems landscape. It is clear that we take out some pieces from there and try to aggregate them into microservices, but the core will remain. It’s hard for me to imagine now that we will replace all the core systems there and quickly move to the other, bright side of the new systems.
I am a supporter of the fact that everything that is closer to the client and the consumer is where the greatest business benefit and value is, where adaptability and focus on speed, change, “try, cancel, reuse, do something else” are needed. ' - that's where the landscape will change. And boxed products don't fit in very well. At least we don't see it. It requires the most light, simple solutions.
We see this development:
- core information systems (mostly back office);
- middle layers in the form of microservices connect the core, aggregate, create a cache, and so on;
- front-line systems are aimed at the consumer;
- an integration layer that is generally integrated into marketplaces, other systems and ecosystems. This layer is as light as possible, simple, it has a minimum of business logic.
But at the same time, I am a supporter of continuing to use the old principles, if they are used in the right place.
Let's say you have a classic enterprise system. It is located in the landscape of one vendor, consists of two modules that work with each other. There is also a standard integration interface. Why redo it and bring a microservice there?
But when there are 5 modules in the back office, from which pieces of information are collected into a business process, which is then used by 8-10 front-line systems, the benefit is immediately noticeable here. You take from five back-office systems, create a service, and an isolated one, which is focused on a business process. You make the service technologically advanced - so that it caches information and is fault-tolerant, and also works with documents or business entities. And you integrate it according to a single principle with all front-line products. They canceled the front-end product - they just turned off the integration. Tomorrow you need to write a mobile application or make a small website and just land one part into functionality - everything is simple: you put it together like a constructor. I see development in this direction more, at least in our country.
Alexander:
Sergey fully described our approach, thanks. I will only say where we will definitely not go - to the core part, to the topic of online billing. That is, the rating and charging will remain, in fact, a "sish" thresher, which will reliably write off money. And this system will continue to be certified by our regulatory authorities. Everything else that looks towards customers, of course, is microservices.
Dmitriy:
Here, just certification is one story. Probably more support. If you spend a little on support or the system does not require support and improvement, it is better not to touch it. Reasonable compromise.
How to develop robust microservices
Dmitriy:
Fine. But I'm still interested. Now you are telling a success story: everything was fine, we switched to microservices, defended the idea in front of the business, and everything worked out. But I've heard other stories.
A couple of years ago, a Swiss company that had invested two years in developing a new microservices platform for banks ended up shutting down the project. Completely turned around. Many millions of Swiss francs were spent, but in the end they dispersed the team - it didn’t work.
Have you had similar stories? Were there any difficulties? For example, maintenance of microservices, the same monitoring is also a headache in the company's operations. After all, the number of components increases tenfold. How do you see it, have there been unsuccessful examples of investments here? And what can be advised to people so that they do not face similar problems?
Alexander:
Unfortunate examples were that the business changed priorities and canceled projects. When at a good stage of readiness (in fact, the MVP is ready), the business said: “We have new priorities, we are going to another project, and we are closing this one.”
We did not have global fails with microservices. We sleep well, we have a 24/7 duty shift that serves the entire BSS [business support system].
And one more thing - we rent out microservices according to the rules by which boxed products are rented out. The key to success is that you need, firstly, to assemble a team that will fully prepare the microservice for production. The development itself is, conditionally, 40%. The rest is analytics, DevSecOps methodology, the right integrations, and the right architecture. We pay special attention to the principles of building secure applications. Representatives of information security participate in each project both at the stage of planning the architecture and during the implementation process. They also manage code analysis systems for vulnerabilities.
Let's say we deploy our stateless services - we have them in Kubernetes. This allows everyone to sleep peacefully due to autoscaling and auto-raising services, and the shift on duty picks up incidents.
During the entire existence of our microservices, there was only one or two incidents that reached our line. Now there are no operational problems. Of course, we have not 200, but about 50 microservices, but they are used in flagship products. If they failed, we would be the first to know.
Microservices and HR
Sergei:
I agree with my colleague about the transfer to support - that you need to properly organize the work. But I'll tell you about the problems, which, of course, there are.
First, the technology is new. This is a hype in a good sense, and finding a specialist who will understand and be able to create this is a big challenge. The competition for resources is crazy, so experts are worth their weight in gold.
Secondly, with the creation of certain landscapes and a growing number of services, the problem of reuse must be constantly solved. As developers like to do: "Let's write a lot of interesting things here now ..." Because of this, the system grows and loses its effectiveness in terms of money, cost of ownership, and so on. That is, it is necessary to lay reuse in the architecture of systems, include in the road map the implementation of services and the transfer of legacy to a new architecture.
Another problem - although this is good in its own way - is internal competition. “Oh, new fashionable guys have appeared with us, they speak a new language.” People, of course, are different. There are those who are used to writing in Java and those who write and use Docker and Kubernetes. These are completely different people, they speak differently, use different terms and sometimes do not understand each other. The ability or inability to share practice, knowledge sharing, in this sense, is also a problem.
Well, scaling resources. "Hey, let's go! And now we want faster, more. What, you can't? Isn't it possible to bet twice as much in a year? And why?" Such growth problems are probably standard for many things, many approaches, and they are felt.
About monitoring. It seems to me that services or industrial monitoring tools are already learning or able to work with both Docker and Kubernetes in a different, non-standard mode. So that, for example, you do not have 500 Java machines under which all this is running, namely, it aggregates. But these products still lack maturity, they have to go through this. The topic is really new, it will still be developed.
Dmitriy:
Yes, very interesting. And that goes for HR. Perhaps your HR process and HR brand have changed a bit in these 3 years. You began to recruit other people with different competencies. And there are probably pluses and minuses. Previously, blockchain and data science were hype, and specialists in them cost just millions. Now the cost is falling, the market is saturated, and there is a similar trend in microservices.
Sergei:
Yes, absolutely.
Alexander:
HR asks the question: “Where is your pink unicorn between backend and frontend?” HR doesn't understand what a microservice is. We revealed the secret to them and said that it was the backend who did everything, and there is no unicorn there. But HR is changing, learning quickly and recruiting people who have basic IT knowledge.
The evolution of microservices
Dmitriy:
If you look at the target architecture, then microservices look like such a monster. Your path took several years. Others have a year, others three years. Did you foresee all the problems, the target architecture, did anything change? For example, in the case of microservices, gateways, service meshes are now appearing again. Did you use them in the beginning or did you change the architecture itself? Do you have such challenges?
Sergei:
We have already rewritten several interaction protocols. At first, one protocol, now they switched to another. We improve safety and reliability. We started with enterprise technologies - Oracle, Web Logic. Now we are moving away from technological enterprise products in microservices and moving to open source or completely open technologies. We refuse databases, we pass to what works more effectively for us in this model. We no longer need Oracle technologies.
We started just as a service, without thinking about how much we need a cache, what we will do when there is no connection with the microservice, but data is needed, etc. Now we are developing the platform so that the architecture can be described not in the language of services, and in business language, take business logic to the next level when we start talking in words. Now we have learned to speak in letters, and the next level is when services will be assembled into some kind of aggregate, when this is already a word - for example, a whole product card. It is assembled from microservices, but this is such an API built on top of this.
Security is very important. As soon as you start to be available and you have a service through which you can get a lot of interesting things, and very quickly, in a split second, then there is a desire to get it in a not the safest way. To get away from this, we had to change approaches to testing and monitoring. I had to change the team, the supply management structure, CI / CD.
This is an evolution, just like with phones, only much faster: first there were push-button phones, then smartphones appeared. They rewrote, redesigned the product because the market had a different need. This is how we evolve: first grade, tenth grade, work.
Iteratively, in a year, something is laid in terms of technology, something else in terms of backlog and needs. We connect one with the other. The team spends 20% on the technical debt and technical support of the team, 80% on the business entity. And we move, with an understanding of why we are doing it, why we are doing these technological improvements, what they will lead to. Like that.
Dmitriy:
Cool. And what about MegaFon?
Alexander:
The main challenge during our arrival in microservices is not to fall into chaos. The architectural office of MegaFon immediately joined us, even became the initiator and driver - now we have a very strong architecture. His task was to understand what target model we are moving into and what technologies need to be piloted. With architecture, we ourselves conducted these pilotings.
The next question was: “And then how to exploit all this?”. And one more: “How to ensure transparent interaction between microservices?”. The service mesh helped us answer the last question. We piloted Istio and we liked the results. Now we are at the stage of rolling into productive zones. We have a positive attitude to all challenges - to the fact that you need to constantly change the stack, learn something new. We are interested in developing, not exploiting old solutions.
Dmitriy:
Gold words! Such challenges keep the team and business in good shape and create the future. GDPR created chief data protection officers, and current challenges create chief microservices and architecture. And it pleases.
We discussed a lot. The main thing is that a good study of microservices and the architecture itself allows you to avoid many mistakes. Of course, the process is iterative and evolutionary, but it is the future.
Thanks to all the participants, thanks to Sergey and Alexander!
Questions from the audience
Question from the audience (1):
Sergey, how has IT management changed in your company? I understand that when there is a large stack of several systems, how it is managed is a fairly understandable and logical process. How did you rebuild the management in the IT component after a very large number of microservices were integrated in such a short time?
Sergei:
I will join my colleague in saying that architecture is very important as a driver of change. We started with the fact that we had an architectural division. Architects are at the same time owners of the distribution of functionality and requirements for how it will look in the landscape. So they also act as coordinators of these changes. As a result, there were specific changes in the specific delivery process when we created the CI/CD platform.
But the standard, basic principles of development, business analysis, testing and development - no one has canceled. We just added speed. Previously, the cycle took so much, installation on test environments took so much more. Now business sees the benefit and says: “Why can’t you do the same in other places?”
It's like, in a good way, an injection in the form of a vaccine, which showed: you can do it this way, but you can do it differently. Of course, there is a problem in personnel, in competencies, in knowledge, in resistance.
Question from the audience (2):
Critics of microservice architecture say that there are difficulties with testing and development. This is logical where things get complicated. What challenges did your team face and how did you overcome them? Question to everyone.
Alexander:
There are difficulties when moving from microservices to a platform, but they are solvable.
For example, we are making a product that consists of 5-7 microservices. We need to provide integration tests across the microservices scope to give the green light to the master branch. This task was not new for us: we have been doing this for a long time at BSS, when the vendor supplied us with already shipped solutions.
And our problem is only in a small team. One conditional product requires one QA engineer. And so, we ship a product from 5-7 microservices, of which 2-3 can be developed by third-party people. For example, we have a product developed by our billing system vendor, Mail.ru Group and MegaFon R&D. We need to cover this with tests before shipping it to production. A QA engineer has been working on this product for a month and a half, and the rest of the team is left without his support.
This complexity is caused only by scaling. We understand that microservices cannot exist in a vacuum, there is no absolute isolation. When changing one service, we always try to keep the API contract. If something changes under the hood, the service front remains. If the changes are fatal, there is some kind of architectural transformation and we generally switch to a different data metamodel that is completely incompatible - only then we say that the v2 service API specification appears. We support the first and second versions at the same time, and after the transition of all consumers to the second version, we simply close the first one.
Sergei:
I want to add. I absolutely agree about the complications - they happen. The landscape is becoming more complex, and overheads are rising, especially for testing. How to deal with this: switch to autotesting. Yes, you will have to additionally invest in writing autotests and unit tests. So that developers could not commit without passing the test, they could not change the code. So that even the push button does not work without an autotest, a unit test.
It is important to maintain the previous functionality, and this is an additional overhead. If you rewrite technology to another protocol, then you rewrite it until you close everything entirely.
Sometimes we don’t do end-to-end testing on purpose, because we don’t want to stop development, although we also cling to one another. The landscape is very large, complex, there are many systems. Sometimes just stubs - yes, you lower the security margin, more risks appear. But at the same time, you release the supply.
Alexander:
Yes, autotests and unit tests allow you to make a quality service. We are for a pipeline that cannot be passed without unit and integration tests. We often have to drag emulators, systems from the sale to test zones and development environments, because not all systems can be placed in test zones. Moreover, they do not just get wet - we generate a full-fledged response from the system. This is a serious part of working with microservices, and we are also investing in it. Without it, there will be chaos.
Question from the audience (3):
As far as I understand, initially microservices grew from a separate team and now exist in this model. What are her pros and cons?
It’s just that we have a similar story: a certain microservices factory has emerged. Now we have conceptually approached the fact that we are extending this approach to production by streams and by systems. In other words, we are moving away from the centralized development of exactly microservices, microservice models, and are getting closer to systems.
Accordingly, our operation also goes to the systems, that is, we decentralize this topic. What is your approach and what is the target story for you?
Alexander:
You took the name “microservice factory” off the tongue - we also want to scale. Firstly, we really have one team now. We want to give all the development teams that MegaFon has the opportunity to work in a common ecosystem. We do not want to completely take over all the development functionality that we have now. The local task is to scale, the global task is to develop all teams in the microservice layer.
Sergei:
Let me tell you the path we have taken. We really started working as one team, but now it is not alone. I am a supporter of the following: there must be an owner of the process. Someone needs to understand, manage, control and build the microservices development process. He must own resources and engage in resource management.
These resources who know the technology, know the specifics, and understand how to build microservices can be on product teams. We have a mix where people from the microservice platform are on the product team that makes the mobile app. They are there, but they work on the process of the microservice platform management department with their development manager. Within this division there is a separate team that deals with technology. That is, we mix together a common pool of resources and share them, giving them inside the commands.
At the same time, the process remains general, controlled, it proceeds according to general technological principles, with unit testing, and so on - everything that is built on top. There may be columns in the form of resources collected from different departments of the product approach.
Alexander:
Sergey, you are actually the owner of the process, right? Backlog tasks shared? Who is distributing it?
Sergei:
Look: here is the mix again. There is a backlog that is formed based on technological improvements - this is one story. There is a backlog that is formulated from projects, and there is a backlog from products. But the sequence of bringing inside each of the products of the service or the creation of this service is developed by the product manager. He is not in the IT department, he was specially removed from it. But my people definitely work on the same process.
The owner of the backlog in different directions - backlog of changes - will be different people. Connectivity of technological services, their principle of organization - all this will be in IT. I own the platform, I own the resources too. At the top is what concerns the backlog and functional changes, and the architecture in this sense.
Suppose a business says: “We want such a function, we want to create a new product - remake a loan.” We answer: "Yes, we will redo it." Architects say: “Let's think: where can we write microservices on credit and how can we do it?”. Then we decompose it into projects, products or a technological stack, lower it into teams and implement it. Have you created a product inside and decided to use microservices in this product? We say: “Now the legacy systems that we had, or front-line ones, should switch to these microservices.” Architects say: “So: in the technological backlog inside the front-end products - the transition to microservices. Go". And product specialists or business owners understand how much capacity is allotted, when it will be done and why.
End of discussion, but not yet
The mailto:CLOUD conference was organized .
We also do other activities - for example, @Kubernetes Meetupwhere we are always looking for great speakers:
- Follow @Kubernetes and other @Meetup news in our Telegram channel
- Interested in performing at one of the @Meetups? Leave a request for
Source: habr.com