Or every unhappy company with a monolith is unhappy in its own way.
The development of the Dodo IS system began right alongside the Dodo Pizza business in 2011. The underlying idea was complete and total digitalization of business processes, , which raised many questions and skepticism even back in 2011. Yet, for the past 9 years, we have been following this path—with our own development that started with a monolith.
This article is a "response" to the questions "Why rewrite the architecture and make such large and lengthy changes?" to the previous article . I will begin with how the development of Dodo IS started, what the initial architecture looked like, how new modules emerged, and the issues that necessitated large-scale changes.
The series of articles 'What is Dodo IS?' will cover:
The early monolith in Dodo IS (2011-2015). (You are here)
.
The client-side journey: facade over the database (2016-2017). (In progress...)
The history of real microservices. (2018-2019). (In progress...)
The completed slicing of the monolith and stabilization of the architecture. (In progress...)
Initial architecture
In 2011, the architecture of Dodo IS looked like this:
The first module in the architecture was order taking. The business process was as follows:
the customer calls the pizzeria;
the manager answers the phone;
takes the order over the phone;
simultaneously inputs it into the order-taking interface: customer information, order details, and delivery address are collected.
The information system interface looked something like this...
The first version from October 2011:
Resources for developing the first order-taking module were limited. A lot needed to be done quickly and with a small team. The small team consisted of 2 developers who laid the foundation for the entire future system.
Their first solution determined the future fate of the technology stack:
Backend on ASP.NET MVC, language C#. The developers were .NET specialists; this stack was familiar and comfortable for them.
Frontend on Bootstrap and JQuery: user interfaces on custom styles and scripts.
MySQL database: no licensing costs, easy to use.
Servers on Windows Server, because .NET could only run on Windows at that time (we won't discuss Mono).
Physically, all of this manifested as a "dedicated server from a hoster."
Order Taking Application Architecture
Back then, everyone was talking about microservices, and SOA had been used in large projects for about 5 years; for example, WCF was released in 2006. But at that time, a reliable and proven solution was chosen.
Here it is.
Asp.Net MVC is Razor that generates an HTML page with server-side rendering in response to a request from a form or client. On the client side, CSS and JS scripts display information and, if necessary, perform AJAX requests via JQuery.
Requests to the server go to classes *Controller, where processing and generation of the final HTML page occur in a method. Controllers make requests to a logic layer called *Services. Each service corresponds to a certain aspect of the business:
For instance, DepartmentStructureService provided information about pizzerias, regarding departments. A department is a group of pizzerias managed by a single franchisee.
ReceivingOrdersService accepted and calculated the order composition.
And SmsService sent SMS, invoking API services for SMS dispatch.
Services processed data from the database and stored business logic. Each service had one or more *Repository with the corresponding name. They contained queries to stored procedures in the database and a mapping layer. The business logic was in the stored procedures, especially in those that generated reporting data. ORM was not used; everything relied on manually written SQL.
There was also a domain model layer and common helper classes, for example, the Order class that stored the order. There was also a helper in the layer for converting display text based on the selected currency.
All this can be represented in the following model:
Order Path
Let's consider a simplified initial path for creating such an order.
Initially, the site was static. It displayed prices with a phone number and a message saying "Want pizza — call this number and place your order". To place an order, we need to implement a simple flow:
The client visits the static site with prices, selects products, and calls the number displayed on the site.
The client names the products they want to add to the order.
They provide their address and name.
The operator accepts the order.
The order is displayed in the accepted orders interface.
It all starts with the menu display. The logged-in operator user can only take one order at a time. Therefore, the draft cart can be kept in their session (the user's session is stored in memory). There is a Cart object in which the products and client information are held.
The client mentions the product, and the operator clicks on + next to the product, and a request is sent to the server. Information about the product is pulled from the database and added to the cart.
Note. Yes, here you can pass the product from the frontend instead of pulling it from the database. But for clarity, I showed the path from the database.
Next, we enter the customer's address and name.
When clicking "Create Order":
The request is sent to OrderController.SaveOrder().
We retrieve the Cart from the session, where the products are stored in the necessary quantity.
We supplement the Cart with customer information and pass it to the AddOrder method of the ReceivingOrderService class, where it is saved in the database.
The database has tables for the order, order components, the client, and they are all linked.
The order display interface fetches the latest orders and reflects them.
New modules
Receiving orders was important and necessary. You can't run a pizza selling business without an order reception for sales. Therefore, the system began to gain functionality — around 2012 to 2015. During this time, many different blocks of the system appeared, which I will refer to as modules, in contrast to the concept of a service or product.
A module is a set of functions combined for a common business goal. They physically reside in one application.
Modules can be called system blocks. For example, there is a reporting module, admin panel interfaces, , authorization. All of these are different interfaces for the user, some even have different visual styles. However, all are within the same application, within one working process.
Technically, the modules were designed as Areas (this idea still exists in ). There were separate files for the frontend, models, as well as their own controller classes. Eventually, the system transformed from this...
...to this:
Some modules are implemented as separate websites (executable projects), due to completely separate functionality and partly because of a more focused development approach. These include:
Site — of the site dodopizza.ru.
Export: report export from Dodo IS for 1C.
Personal — employee personal account. Developed separately and has its own entry point and separate design.
fs — a project for hosting static content. Later, we moved away from it, transferring all static files to CDN Akamai.
The remaining blocks were located in the BackOffice application.
Explanation of names:
Cashier — Restaurant cash register.
ShiftManager — interfaces for the role of 'Shift Manager': real-time sales statistics for the pizzeria, the ability to add products to a stop list, and modify orders.
OfficeManager — interfaces for the roles of 'Pizzeria Manager' and 'Franchisee'. This is where functions for setting up the pizzeria, its bonus promotions, employee management, and reporting are gathered.
PublicScreens — interfaces for televisions and tablets hanging in pizzerias. Menus, promotional information, and order status are displayed on the televisions during delivery.
They used a shared service layer, a common block of domain classes Dodo.Core, as well as a common database. Sometimes they could also interact through transitions to each other. Separate sites like dodopizza.ru or personal.dodopizza.ru also accessed the shared services.
When new modules appeared, they tried to maximize the reuse of already created service codes, stored procedures, and tables in the database.
To better understand the scope of modules developed in the system, here's a diagram from 2012 outlining the development plans:
By 2015, everything on the diagram and even more was in production.
Order acceptance evolved into a separate Contact Center block, where an operator takes the order.
Public screens with menus and information appeared, hanging in pizzerias.
In the kitchen, there is a module that automatically plays a voice message 'New Pizza' upon receiving a new order, and also prints a receipt for the courier. This significantly simplifies processes in the kitchen, allowing staff to avoid distractions from many simple operations.
The delivery block became a separate Delivery Checkout, where the order was handed over to a courier, who had previously clocked in. Their working hours were recorded for payroll calculations.
From 2012 to 2015, more than 10 developers appeared, 35 pizzerias were opened, the system was deployed in Romania, and preparations were made for opening locations in the USA. Developers no longer handled all tasks but were divided into teams, each specializing in their part of the system.
Issues
This was due in part to the architecture (but not only).
Chaos in the database
One database is convenient. It allows for consistency, especially with means built into relational databases. Working with it is familiar and convenient, especially if there are few tables and little data.
However, after 4 years of development, there were about 600 tables and 1,500 stored procedures, many of which also contained logic. Unfortunately, stored procedures do not provide significant advantages when working with MySQL. They are not cached by the database, and embedding logic in them complicates development and debugging. Code reuse is also difficult.
Many tables lacked appropriate indexes, while in some cases there were too many indexes, which hindered insertion. About 20 tables needed to be modified — the transaction for creating an order could take about 3-5 seconds.
Data in tables was not always in the most suitable form. In some cases, denormalization was necessary. Some regularly received data was stored in a column as an XML structure, which increased execution time, elongated queries, and complicated development.
There were very varied queries to the same tables. Popular tables, like the mentioned table orders or the table pizzeria, were affected. They were used for displaying operational interfaces in the kitchen and analytics. The website,, also accessed them, which could unexpectedly receive many requests at any given time.
The data was not aggregated and many calculations occurred on the fly using the database. This created unnecessary computations and additional load.
Often the code accessed the database when it could have avoided it. In some areas, bulk operations were lacking, and in others, it was necessary to split one request into several through the code to speed up and improve reliability.
Cohesion and complexity in the code
Modules that were supposed to manage their business area did not do so effectively.Some of them had function duplication across roles. For example, the local marketer responsible for the network's marketing activities in their city had to use both the 'Admin' interface (for creating promotions) and the 'Office Manager' interface (to view the impact of promotions on the business). Of course, both modules internally used the same service that handled bonus promotions.
Services (classes within a large monolithic project) could call each other to enrich their data.
The model classes that stored the data were handled differently in the code.There were builders somewhere that allowed specifying mandatory fields. In some cases, this was done through public properties. Of course, retrieving and transforming data from the database varied.
The logic was either in the controllers or in the service classes.
These may seem like minor issues, but they significantly slowed down development and compromised quality, leading to instability and errors.
The complexity of large-scale development.
Difficulties also arose in the development itself.Different components of the system needed to be created, and concurrently. It became increasingly difficult to accommodate the needs of each component within a single codebase. It wasn't just challenging to reach an agreement and satisfy all components at once. Additionally, there were constraints in the technologies used, particularly regarding the database and frontend. There was a need to move away from jQuery towards higher-level frameworks, especially in client services (the website).
In some parts of the system, databases more suitable for this could have been used.For example, we later had a precedent of transitioning from Redis to CosmosDB for order cart storage.
Teams and developers focused on their areas clearly desired more autonomy for their services, both in terms of development and deployment. Conflicts during merges, issues during releases. If this problem was insignificant for 5 developers, with 10, especially with the planned growth, it would become more serious. Ahead was the development of a mobile application (which started in 2017, and in 2018 there was) ).
Different parts of the system required different stability metrics., but due to the strong coupling of the system, we could not provide that. An error in developing a new feature in the admin panel could easily affect the order process on the website, as the code is shared and reusable, and the database and data are also unified.
Perhaps it would have been possible, even within such a monolithic-modular architecture, to avoid these errors and issues by establishing clear separations of responsibility, conducting refactoring of both the code and the database, distinctly separating layers from one another, and monitoring quality daily. However, the architectural decisions made and the focus on rapid expansion of system functionality led to issues regarding stability.
How the blog Power of Mind impacted restaurant cash registers
If the growth of the pizza chain (and workload) had continued at the same pace, the drop would have reached a point where the system wouldn't recover. A story that well illustrates the problems we started facing by 2015 is as follows.
In the blog “”, there was a widget that displayed revenue data for the entire network over the year. The widget accessed the public Dodo API, which provides this data. This statistics is now available at . The widget was displayed on every page and made requests on a timer every 20 seconds. The request went to api.dodopizza.ru and queried:
the number of pizzerias in the network;
the total revenue of the network since the beginning of the year;
the revenue for today.
The revenue statistics request went straight to the database, starting to fetch order data, aggregating it live and outputting the sum.
The cash registers in the restaurants would also access the same order table, exporting the list of accepted orders for the day, with new orders being added as well. The registers made their requests every 5 seconds or upon page refresh.
The scheme looked like this:
One autumn day, Fyodor Ovchinnikov wrote a long and popular article in his blog. A lot of people visited the blog and paid close attention to everything. While each visitor read the article, the revenue widget was successfully operational, querying the API every 20 seconds.
The API called a stored procedure to calculate the total amount of all orders since the beginning of the year across all pizzerias in the network. The aggregation was done by the orders table, which is highly popular. All cash registers from all open restaurants at that moment accessed it. The registers stopped responding, and new orders were not being accepted. Additionally, orders were not being accepted from the website, they did not appear on the tracker, and the shift manager could not view them in their interface.
This is not the only story. By the autumn of 2015, the system was under critical load every Friday. A few times we had to switch off the public API, and once, we even had to take the website down because nothing else helped. There was even a list of services with an order of shutdown during serious load.
From this time, our struggle with loads and the stabilization of the system began (from autumn 2015 to autumn 2018). It was during this time that the “». There were also failures that sometimes occurred later, some were quite sensitive, but the overall period of instability can now be considered past.
Rapid business growth
Why couldn't we 'get it right the first time'? Just look at the following charts.
Additionally, there was an opening in Romania in 2014-2015 and preparations were underway for an opening in the USA.
The network grew very quickly, new countries were opened, and new pizza formats appeared, for example, a pizzeria opened in a food court. All this required significant attention to the expansion of the Dodo IS functions. Without all these features, such as kitchen tracking, product and loss accounting in the system, and displaying order outputs in the food court, it is unlikely we would be discussing a 'correct' architecture and a 'proper' approach to development now.
Another obstacle to timely revising the architecture and addressing technical issues in general was the crisis of 2014. Such events severely impact growth opportunities for teams, especially for a young business like Dodo Pizza.
Quick solutions that helped
Problems needed solutions. Conventionally, solutions can be divided into two groups:
Quick solutions that put out the fire and provide a small margin of safety, buying us time for changes.
Systemic and, therefore, long-term. Reengineering several modules, separating the monolithic architecture into distinct services (most of which are not micro, but rather macro services, and there is a ).
The dry list of quick changes is as follows:
Scale up the master database
Of course, the first step taken to combat the load is to increase server capacity. This was done for the master database and for the web servers. Unfortunately, this is only possible up to a certain limit; beyond that, it becomes too expensive.
Since 2014, we have transitioned to Azure, and we also wrote about this back then in the article “”. However, after a series of server increases for the database, we hit a cost ceiling.
Read replicas of the database
We created two replicas for the database:
ReadReplica for queries on directories. It is used for reading directories such as cities, streets, pizzerias, products (slowly changed domain), and in those interfaces where a slight delay is acceptable. There were 2 of these replicas, and we ensured their availability just like the master.. Used for reading references such as cities, streets, pizzerias, and products (slowly changed domain), and in interfaces where a slight delay is acceptable. There were 2 of these replicas, and we ensured their availability just like the master.
ReadReplica for report queries. This database had lower availability, but all reports accessed it. While they may have heavy requests for large data recalculations, they do not affect the main database and operational interfaces.
Caches in code
There were no caches in the code at all. This led to additional, not always necessary, queries to the overloaded database. Caches were initially implemented both in-memory and on an external caching service, which was Redis. Everything was invalidated based on time, with settings specified in the code.
Multiple servers for the backend
The application's backend also needed to be scaled to handle increased loads. We needed to turn a single IIS server into a cluster. We migrated from memory to RedisCache, allowing for multiple servers behind a simple round-robin load balancer. Initially, the same Redis was used as for the caches, but later it was distributed across several instances.
As a result, the architecture became more complex…
…but some of the stress was alleviated.
Next, we needed to redesign the overloaded components, which we began to tackle. We will discuss this in the next part.
Source: habr.com
