Hello, Habr! At Dodo Pizza Engineering, we really love data (who doesn't these days?). Here’s a story about how to gather all the data of the Dodo Pizza world and provide any company employee convenient access to this wealth of information. The task at hand: keep the Data Engineering team’s nerves intact.
Like true collectors, we accumulate all sorts of information about the operations of our pizzerias:
- we remember all user orders;
- we know how long it took to prepare the very first pizza in Syktyvkar;
- we can see how long a pizza has been cooling on the heat shelf in Voronezh right now;
- we store data on product write-offs;
- and much more.
Currently, several teams handle data at Dodo Pizza, one of which is the Data Engineering team. Now we have a task: to provide any company employee with convenient access to this wealth of data.
When we started thinking about how to accomplish this and discussing the task, we discovered a very interesting approach to data management – (you'll find an enormous, excellent article at the link). Its ideas fit perfectly with our vision of how we want to build our system. The following in the article will be our reinterpretation of this approach and how we see its implementation in Dodo Pizza Engineering.
What do we mean by 'data'
First, let's clarify what we mean by data at Dodo Pizza Engineering:
- Events sent by services (we have a common bus built using RabbitMQ);
- Records within a database (for us, this includes MySQL and CosmosDB);
- Clickstream data from the mobile app and website.
For Dodo Pizza's business to utilize and trust this data, the following conditions must be met:
- They must be complete. We must ensure that we do not alter the data during processing, storage, and display. If the business cannot trust our data, then it serves no purpose.
- They must be time-stamped and not overwritten. This means that at any point in time, we want to be able to roll back and view the data from that time frame. For example, to find out how many pizzas were sold on July 8, 2018.
- They must be reliable. In the process of collecting and storing data, we must not only maintain integrity but also reliability. We cannot afford data loss or time slices because in doing so, we lose the trust of our clients (both external and internal).
- They should have a stable schema – we write queries for this data. We really don’t want the application code changes, with refactoring, to alter them so much that our queries stop working. Those who write queries will never know that you made refactoring until everything breaks down completely. It would be unfortunate to find out about this from clients.
Considering all these requirements, we concluded that the data at Dodo is a product. Just like the public API of the service. Consequently, the same team that owns the service should also own the data. Additionally, changes to the data schema must always be backward compatible.
Traditional approach – Data Lake
To solve the challenges of reliable storage and processing of large data, there is a traditional approach adopted by many companies that work with such pools of information – Data Lake. Within this approach, data engineers collect information from all system components and store it in one large repository (this can be, for example, Hadoop, Azure Kusto, Apache Cassandra, or even a MySQL replica if the data fits in it).
Then the same engineers write queries to this repository. The implementation of this approach at Dodo Pizza Engineering implies that the Data Engineering team will own the data schema in the analytical storage.
In this scenario, the team becomes very sad kittens, and here's why:
- They have to monitor changes in ALL services within the company. And there are many, with numerous changes (on average, we merge ~100 pull requests a week, while many services do not make pull requests at all).
- When there is a change in the data schema, the product owner and the team changing the data schema must wait for the Data Engineering team to write the necessary code to support the changes. Meanwhile, we have long had features in the pipeline and situations where one team waits for another are very rare. We don’t want this to become a 'normal' part of the development process.
- They need to be immersed in ALL the business of the company. A network of pizzerias may seem like a simple business, but that's just an illusion. It is quite difficult to gather enough competencies in one team to build a proper data model for the entire company.
- It serves as a single point of failure. Every time there's a need to change the data that the service returns or to write a query, all these tasks fall to the Data Engineering team. As a result, the team becomes overwhelmed with their backlog.
This means that the team is at the intersection of an enormous number of demands and is unlikely to meet them all. They will be under constant time pressure and stress. We definitely want to avoid that. Therefore, we need to think about how to solve these problems while also gaining the ability to analyze the data.
Transitioning from Data Lake to Data Mesh
Fortunately, this question has been asked by others as well. In fact, a similar problem has already been solved in the industry (hallelujah!). Just in a different area: application deployment. Yes, I’m talking about the DevOps approach, where the team determines how to deploy the product they are creating.
A similar approach to solving Data Lake problems was proposed by Zhamak Dehghani, a consultant at ThoughtWorks. Observing how companies like Netflix and Spotify address these challenges, she wrote an amazing article (the link to it was at the beginning of the article). The main ideas that we took from it are:
- Divide the large Data Lake into data domains, which are quite similar to domain-driven design domains. Each domain is a small bounded context.
- Feature Teams, which are responsible for DDD domains, are also in charge of the respective data domains. They maintain the schema, make changes to it, and load the data into it. They know exactly how to change data loading without breaking anything when the application changes. The knowledge does not go away. To access the data, they do not need to go anywhere. The team manages the entire development cycle from changing operational data to providing analytical data to third parties. One team owns everything related to the domain (both the business domain and the data domain).
- Data Engineer – a role within the Feature Team. This does not necessarily have to be a separate person, but it is essential for the team to possess this competence.
And meanwhile, the Data Engineering team...
If we imagine that all of this happens at the snap of a finger, we are left with two questions to answer:
What will the Data Engineering team focus on now? In Dodo Pizza Engineering, there is already a platform/SRE team. Their task is to provide developers with tools for easily deploying services. The Data Engineering team will perform a similar role, but for data.
Transforming operational data into analytical data is a complex process. Making analytical data accessible to the entire company is even more difficult. The Data Engineering team will be focused on solving these problems.
We plan to provide the Feature Team with a convenient set of tools and practices that will enable them to publish data from their service for the rest of the company. We will also be responsible for the common infrastructural parts of the data pipeline (queues, reliable storage, clusters for performing transformations on data).
How will Data Engineer skills develop within the Feature Team? Things are more complicated with the Feature Team. Of course, we could try to hire one Data Engineer for each of our teams. But this is very challenging. Finding a person with a solid background in data processing and convincing them to work within a product team is tough.
A major advantage of Dodo is that we value internal training. So our current plan is as follows: the Data Engineering team begins to publish data from some services, struggles and adapts, but continues to move forward. Once we understand that we have a ready process for publishing, we will start communicating about it to the Feature Team.
We have several ways to accomplish this:
- where we will discuss the process we have created, the tools available, and how to use them most effectively.
- Presenting at DevForum will help us gather feedback from product developers. After that, we will be able to join product teams and assist them in addressing data publishing issues, and organize training sessions for teams.
Data Consumption
I've talked a lot about data publishing. But there is also data consumption. What about this?
We have an amazing BI team that creates very complex reports for the management company. Inside Dodo IS, there are many reports for our partners that help them manage pizza restaurants. In our new model, we think of them as data consumers who have their own data domains. And it's the consumers who will be responsible for their own domains. Sometimes a consumer's domain can be described by a single query in the analytics repository — and that's fine. But we understand that it won't always work. That's why we want the platform we create for product teams to also be usable by data consumers (after all, in the case of reports within Dodo IS, these will be the same teams).
This is how we see data work at Dodo Pizza Engineering. We would love to hear your thoughts on this in the comments.
Source: habr.com
