History of the Dodo IS Architecture: The Back Office Path

Habr is changing the world. We have been blogging for over a year now. About six months ago, we received a completely logical feedback from Khabrovites: “Dodo, you say everywhere that you have your own system. And what is this system? And why does a pizza chain need it?

We sat, thought and realized that you are right. We try to explain everything on our fingers, but it comes out in torn pieces and nowhere is there a complete description of the system. Thus began a long journey of collecting information, searching for authors and writing a series of articles about Dodo IS. Let's go!

Acknowledgments: Thank you for sharing your feedback with us. Thanks to him, we finally described the system, compiled a technical radar and will soon roll out a large description of our processes. Without you, we would have been sitting there for another 5 years.

Series of articles "What is Dodo IS?" tells about:

1. Early monolith in Dodo IS (2011-2015). (In progress...)
2. The back office path: separate bases and bus. (you are here)
3. The client side path: facade over the base (2016-2017). (In progress...)
4. The history of real microservices. (2018-2019). (In progress...)
5. Finished sawing of the monolith and stabilization of the architecture. (In progress...)

If you are interested to know something else - write in the comments.

Opinion on the chronological description from the author
I regularly hold a meeting for new employees on the topic "System Architecture". We call it “Intro to Dodo IS Architecture” and it is part of the onboarding process for new developers. Telling in one form or another about our architecture, about its features, I have born a certain historical approach to the description.

Traditionally, we look at the system as a set of components (technical or higher-level), business modules that interact with each other to achieve some goal. And if such a view is justified for design, then it is not quite suitable for description and understanding. There are several reasons here:

• Reality is different from what is on paper. Not everything works out as intended. And we are interested in how it actually turned out and works.
• Consistent presentation of information. In fact, you can go chronologically from the beginning to the current state.
• From simple to complex. Not universally, but in our case it is. The architecture moved from simpler approaches to more complex ones. Often through complication, problems of implementation speed and stability were solved, as well as dozens of other properties from the list of non-functional requirements (here well told about contrasting complexity with other requirements).

In 2011, the Dodo IS architecture looked like this:

By 2020, it has become a little more complicated and has become like this:

How did this evolution take place? Why are different parts of the system needed? What architectural decisions were made and why? Let's find out in this series of articles.

The first problems of 2016: why should services leave the monolith

The first articles from the cycle will be about the services that were the first to separate from the monolith. To put you in context, I'll tell you what problems we had in the system by the beginning of 2016, that we had to deal with the separation of services.

A single MySql database, in which all applications that existed at that time in Dodo IS wrote their records. The consequences were:

• Heavy load (with 85% of requests accounted for reading).
• The base has grown. Because of this, its cost and support became a problem.
• Single point of failure. If one application writing to the database suddenly began to do it more actively, then other applications felt it on themselves.
• Inefficiency in storage and queries. Often the data was stored in some structure that was convenient for some scenarios but not suitable for others. Indexes speed up some operations, but can slow down others.
• Some of the problems were removed by hastily made caches and read-replicas to the bases (this will be a separate article), but they only allowed them to gain time and did not fundamentally solve the problem.

The problem was the presence of the monolith itself. The consequences were:

• Single and rare releases.
• Difficulty in joint development of a large number of people.
• Inability to bring in new technologies, new frameworks and libraries.

Problems with the base and the monolith have been described many times, for example, in the context of crashes in early 2018 (Be like Munch, or a few words about technical debt, The day Dodo IS stopped. Asynchronous Script и The story of the Dodo bird from the Phoenix family. The Great Fall of Dodo IS), so I won't dwell too much. Let me just say that we wanted to give more flexibility when developing services. First of all, this concerned those that were the most loaded and root in the entire system - Auth and Tracker.

The Back Office Path: Separate Bases and Bus

Chapter Navigation

1. Monolith scheme 2016
2. Starting to Unload the Monolith: Auth and Tracker Separation
   
3. What does Auth do?
4. Where are the loads from?
5. Unloading Auth
6. What does Tracker do?
7. Where are the loads from?
8. Unloading Tracker

Monolith scheme 2016

Here are the main blocks of the Dodo IS 2016 monolith, and just below is a transcript of their main tasks.

Cashier Delivery. Accounting for couriers, issuing orders to couriers.
Contact center. Acceptance of orders through the operator.
Site. Our websites (dodopizza.ru, dodopizza.co.uk, dodopizza.by, etc.).
Auth. Authorization and authentication service for the back office.
Tracker. Order tracker in the kitchen. Service for marking readiness statuses when preparing an order.
Cash desk of the Restaurant. Taking orders in a restaurant, cashier interfaces.
Export. Uploading reports in 1C for accounting.
Notifications and invoices. Voice commands in the kitchen (for example, “New pizza arrived”) + invoice printing for couriers.
Shift Manager. Interfaces for the work of the shift manager: list of orders, performance graphs, transfer of employees to the shift.
Office Manager. Interfaces for the work of the franchisee and the manager: reception of employees, reports on the work of the pizzeria.
Restaurant Scoreboard. Menu display on TVs in pizzerias.
Admin panel. Settings in a specific pizzeria: menu, prices, accounting, promo codes, promotions, website banners, etc.
Employee's Personal Account. Work schedules of employees, information about employees.
Kitchen Motivation Board. A separate screen that hangs in the kitchen and displays the speed of the pizza makers.
Communication. Sending sms and email.
FileStorage. Own service for receiving and issuing static files.

The first attempts to solve the problems helped us, but they were only a temporary respite. They did not become system solutions, so it was clear that something had to be done with the bases. For example, to divide the general database into several more specialized ones.

Starting to Unload the Monolith: Auth and Tracker Separation

The main services that then recorded and read from the database more than others:

1. Auth. Authorization and authentication service for the back office.
2. Tracker. Order tracker in the kitchen. Service for marking readiness statuses when preparing an order.

What does Auth do?

Auth is a service through which users log in to the back office (there is a separate independent entrance on the client side). It is also called upon in the request to make sure that the required access rights are present and that these rights have not changed since the last login. Through it, devices enter the pizzeria.

For example, we want to open a display with the statuses of finished orders on the TV hanging in the hall. Then we open auth.dodopizza.ru, select "Login as a device", a code appears that can be entered in a special page on the shift manager's computer, indicating the type of device (device). The TV itself will switch to the desired interface of its pizzeria and begin to display the names of customers whose orders are ready there.

Where are the loads from?

Each logged in user of the back office goes to the database, to the user table for each request, pulls out the user through an sql query and checks if he has the necessary access and rights to this page.

Each of the devices does the same only with the device table, checking its role and its access. A large number of requests to the master database leads to its loading and the waste of resources of the common database for these operations.

Unloading Auth

Auth has an isolated domain, that is, data about users, logins or devices enters the service (for the time being) and remains there. If someone needs them, then he will go to this service for data.

IT WAS. The original scheme of work was as follows:

I would like to explain a little how it worked:

1. A request from the outside comes to the backend (there is Asp.Net MVC), brings with it a session cookie, which is used to get session data from Redis(1). It either contains information about accesses, and then access to the controller is open (3,4), or not.
2. If there is no access, you need to go through the authorization procedure. Here, for simplicity, it is shown as part of the path in the same attribute, although it is a transition to the login page. In the case of a positive scenario, we will get a correctly completed session and go to the Backoffice Controller.
3. If there is data, then you need to check it for relevance in the user base. Has his role changed, should he not be allowed on the page now? In this case, after receiving the session (1), you need to go directly to the database and check the user's access using the authentication logic layer (2). Next, either to the login page, or go to the controller. Such a simple system, but not quite standard.
4. If all procedures are passed, then we skip further in the logic in controllers and methods.

User data is separated from all other data, it is stored in a separate membership table, functions from the AuthService logic layer may well become api methods. Domain boundaries are defined quite clearly: users, their roles, access data, granting and revoking access. Everything looks so that it can be taken out in a separate service.

BECOME. So they did:

This approach has a number of problems. For example, calling a method inside a process is not the same as calling an external service via http. Latency, reliability, maintainability, transparency of the operation are completely different. Andrey Morevskiy spoke in more detail about such problems in his report. "50 Shades of Microservices".

The authentication service and, with it, the device service are used for the back office, that is, for services and interfaces used in production. Authentication for client services (like a website or a mobile application) occurs separately without using Auth. The separation took about a year, and now we are again dealing with this topic, transferring the system to new authentication services (with standard protocols).

Why did the separation take so long?
There were many problems along the way that slowed us down:

1. We wanted to move user, device, and authentication data from country-specific databases into one. To do this, we had to translate all tables and usage from the int identifier to the global UUId identifier (recently reworked this code Roman Bukin "Uuid - a big story of a small structure" and open source project Primitives). Storing user data (since it is personal information) has its limitations and for some countries it is necessary to store them separately. But the user's global id must be.
2. Many tables in the database have audit information about the user who performed the operation. This required an additional mechanism for consistency.
3. After the creation of api-services, there was a long and gradual period of transition to another system. Switching had to be seamless for users and required manual work.

Device registration scheme in a pizzeria:

General architecture after the extraction of Auth and Devices service:

Note. For 2020, we are working on a new version of Auth, which is based on the OAuth 2.0 authorization standard. This standard is quite complex, but it is useful for developing a pass-through authentication service. In the article "Subtleties of authorization: an overview of OAuth 2.0 technology» we Alexey Chernyaev tried to tell about the standard as simply and clearly as possible so that you save time on studying it.

What does Tracker do?

Now about the second of the loaded services. The tracker performs a dual role:

• On the one hand, its task is to show the employees in the kitchen what orders are currently in work, what products need to be cooked now.
• On the other hand, to digitize all the processes in the kitchen.

When a new product appears in an order (for example, pizza), it goes to the Rolling out tracker station. At this station, there is a pizza maker who takes a bun of the required size and rolls it out, after which he notes on the tracker tablet that he has completed his task and transfers the rolled dough base to the next station - “Initiation”.

There, the next pizza maker fills the pizza, then notes on the tablet that he has completed his task and puts the pizza in the oven (this is also a separate station that must be noted on the tablet). Such a system was from the very beginning in Dodo and from the very beginning of the existence of Dodo IS. It allows you to fully track and digitize all transactions. In addition, the tracker suggests how to cook a particular product, guides each type of product according to its manufacturing schemes, stores the optimal cooking time for the product, and tracks all operations on the product.

This is how the screen of the tablet looks like at the station of the tracker "Raskatka"

Where are the loads from?

Each of the pizzerias has about five tablets with a tracker. In 2016, we had more than 100 pizzerias (and now more than 600). Each of the tablets makes a request to the backend once every 10 seconds and scrapes data from the order table (connection with the client and address), order composition (connection with the product and indication of the quantity), the motivation accounting table (the time of pressing is tracked in it). When a pizza maker clicks on a product on the tracker, the entries in all of these tables are updated. The order table is general, it also contains inserts when accepting an order, updates from other parts of the system and numerous readings, for example, on a TV that hangs in a pizzeria and shows finished orders to customers.

During the period of struggle with loads, when everything and everything was cached and transferred to the asynchronous replica of the base, these operations with the tracker continued to go to the master base. There should not be any lag, the data should be up-to-date, out of sync is unacceptable.

Also, the lack of own tables and indexes on them did not allow writing more specific queries tailored for their use. For example, it might be efficient for a tracker to have an index for a pizzeria on an order table. We always remove pizzeria orders from the tracker database. At the same time, for receiving an order, it is not so important which pizzeria it falls into, it is more important which client made this order. And means there the index on the client is necessary. It is also not necessary for the tracker to store the id of the printed receipt or bonus promotions associated with the order in the order table. This information is not of interest to our tracker service. In a common monolithic database, tables could only be a compromise between all users. This was one of the original problems.

IT WAS. The original architecture was:

Even after being separated into separate processes, most of the code base remained common for different services. Everything below the controllers was single and lived in the same repository. We used common methods of services, repositories, a common base, in which common tables lay.

Unloading Tracker

The main problem with the tracker is that the data must be synchronized between different databases. This is also its main difference from the separation of the Auth service, the order and its status can change and should be displayed in different services.

We accept an order at the Restaurant's Checkout (this is a service), it is stored in the database in the "Accepted" status. After that, he should get to the tracker, where he will change his status several more times: from “Kitchen” to “Packed”. At the same time, some external influences from the Cashier or the Shift Manager interface may occur with the order. I will give the order statuses with their description in the table:

The scheme for changing order statuses looks like this:

Statuses change between different systems. And here the tracker is not a final system in which the data is closed. We have seen several possible approaches for partitioning in such a case:

1. We concentrate all order actions in one service. In our case, this option requires too much service to work with the order. If we stopped at it, we would get the second monolith. We would not solve the problem.
2. One system makes a call to another. The second option is already more interesting. But with it, chains of calls are possible (cascading failures), the connectivity of the components is higher, it is more difficult to manage.
3. We organize events, and each service communicates with another through these events. As a result, it was the third option that was chosen, according to which all services begin to exchange events with each other.

The fact that we chose the third option meant that the tracker would have its own database, and for each change in the order, it would send an event about this, to which other services subscribe and which also falls into the master database. To do this, we needed some service that would ensure the delivery of messages between services.

By that time, we already had RabbitMQ in the stack, hence the final decision to use it as a message broker. The diagram shows the transition of an order from the Restaurant Cashier through the Tracker, where it changes its status and its display on the Manager's Orders interface. STEEL:

Order path step by step
The order path starts at one of the order source services. Here is the Cashier of the Restaurant:

1. At the checkout, the order is completely ready, and it's time to send it to the tracker. The event to which the tracker is subscribed is thrown.
2. The tracker, accepting an order for itself, saves it to its own database, making the event “Order Accepted by the Tracker” and sending it to RMQ.
3. There are several handlers already subscribed to the event bus per order. For us, the one that makes synchronization with a monolithic base is important.
4. The handler receives an event, selects from it data that is significant for it: in our case, this is the status of the order "Accepted by the Tracker" and updates its order entity in the main database.

If someone needs an order from the monolithic table orders, then you can read it from there as well. For example, the Orders interface in the Shift Manager needs this:

All other services can also subscribe to order events from the tracker to use them for themselves.

If after a while the order is taken into work, then its status first changes in its database (Tracker database), and then the "OrderIn Progress" event is immediately generated. It also gets into RMQ, from where it is synchronized in a monolithic database and delivered to other services. There may be various problems along the way, more details about them can be found in the report of Zhenya Peshkov about the implementation details of Eventual Consistency in the Tracker.

Final architecture after changes in Auth and Tracker

Summing up the intermediate result: Initially, I had the idea to pack the nine-year history of the Dodo IS system into one article. I wanted to quickly and simply talk about the stages of evolution. However, sitting down for the material, I realized that everything is much more complicated and interesting than it seems.

Reflecting on the benefits (or lack thereof) of such material, I came to the conclusion that continuous development is impossible without full-fledged annals of events, detailed retrospectives and analysis of my past decisions.

I hope that it was useful and interesting for you to learn about our path. Now I am faced with a choice which part of the Dodo IS system to describe in the next article: write in the comments or vote.

Only registered users can participate in the survey. Sign in, you are welcome.

What part of Dodo IS would you like to know about in the next article?

• 24,1%Early monolith in Dodo IS (2011-2015)14

• 24,1%First problems and their solutions (2015-2016)14

• 20,7%The client-side path: facade over base (2016-2017)12

• 36,2%The history of real microservices (2018-2019)21

• 44,8%Complete sawing of the monolith and stabilization of the architecture26

• 29,3%About further plans for the development of the system17

• 19,0%I don't want to know anything about Dodo IS11

58 users voted. 6 users abstained.

Source: habr.com