Since 2019, a law has been in effect in Russia mandating product labeling. This law does not apply to all product groups, and the timelines for the enforcement of mandatory labeling differ among these groups. The first products to require labeling include tobacco, footwear, and pharmaceuticals, while others, such as perfumes, textiles, and milk, will follow later. This legislative change has spurred the development of new IT solutions that will allow tracking the entire lifecycle of a product from production to purchase by the end consumer, involving all participants in the process, including the government and all organizations selling products with mandatory labeling.
At X5, the system designed to track labeled products and exchange data with the government and suppliers is called "Markus." Letβs explain in order how it was developed, the technology stack it uses, and why we take pride in it.
A True HighLoad
"Markus" addresses numerous tasks, the primary one being the integration interaction between X5's information systems and the governmentβs information system for labeled products (GIS MP) to monitor the movement of labeled goods. The platform also stores all the received labeling codes and the entire history of these codes' movements across entities, helping to eliminate mislabeling of labeled products. For instance, in the case of tobacco products, which were among the first labeled goods, a single truck carrying cigarettes contains around 600,000 packs, each with its own unique code. Our system's task is to track and verify the legality of each such pack's movement between warehouses and stores, ultimately confirming their sale to the end consumer. We record about 125,000 cash transactions per hour, and we also need to document how each pack reached the store. Considering all movements between entities, we expect to have dozens of billions of records per year.
The M Team
Although 'Marcus' is considered a project within X5, it is implemented using a product approach. The team works according to Scrum. The project started in the summer of last year, but the first results only came in October β the dedicated team was fully assembled, the system architecture developed, and equipment purchased. Currently, the team consists of 16 people, six of whom are engaged in backend and frontend development, while three focus on system analysis. Another six people handle manual, load, automated testing, and product support. In addition, we have an SRE specialist.
In our team, not only developers write code; practically everyone is capable of programming and contributes to writing automated tests, load scripts, and automation scripts. We pay special attention to this, as even product support requires a high level of automation. We always try to advise and assist colleagues who haven't programmed before, assigning them some small tasks.
Due to the coronavirus pandemic, we transitioned the entire team to remote work. The availability of all tools for managing development, along with the established workflow in Jira and GitLab, made it easy to navigate this phase. The months spent working remotely demonstrated that the team's productivity was not negatively affected; in fact, many experienced increased comfort in their work, though there is a lack of in-person communication.
Team meeting before remote work
Meetings during remote work
Technology stack of the solution
GitLab serves as the standard repository and CI/CD tool for X5. We use it for code storage, continuous testing, and deployment on testing and production servers. We also practice code review, which requires approval from at least two colleagues for any code changes made by the developer. Static code analyzers like SonarQube and JaCoCo help us maintain code quality and ensure the required level of unit test coverage. All code changes must pass through these checks. All test scenarios run manually are later automated.
To successfully execute the business processes with 'Marcus', we had to solve a number of technological challenges, which I will address in order.
Task 1. The Need for Horizontal Scalability of the System
To address this task, we opted for a microservices architecture. It was crucial to understand the responsibilities of the services. We aimed to separate them by business operations, taking into account process specifics. For instance, warehouse acceptance is not a very frequent but highly intensive operation, during which we need to quickly obtain information from the government regulator regarding the received product units, with quantities in a single shipment reaching up to 600,000, verify the permissibility of accepting this product into the warehouse, and provide all necessary information to the warehouse automation system. On the other hand, shipping from warehouses is much more frequent but involves processing smaller data volumes.
All services are built on a stateless principle, and we strive to break down even internal operations into steps, using what we call self-topics in Kafka. This means the microservice sends messages to itself, which helps balance the load on more resource-intensive operations and simplifies product maintenance, but more on that later.
We decided to separate the modules interacting with external systems into distinct services. This approach allowed us to address the issue of frequently changing external system APIs with minimal impact on the services that handle business functionalities.
All microservices are deployed in an OpenShift cluster, which solves both the scalability issue for each microservice and also allows us to avoid using third-party Service Discovery tools.
Task 2. The Need to Maintain High Load and Very Intensive Data Exchange Between Platform Services: only at the project launch phase, about 600 operations per second are performed. We expect this value to increase to 5000 op/sec as more trading entities connect to our platform.
This issue was addressed by deploying a Kafka cluster and virtually eliminating synchronous interactions between the platform's microservices. This requires very careful analysis of system requirements, as not all operations can be asynchronous. Moreover, we do not just transmit events through the broker, but we also pass all the necessary business information in the message. Consequently, the message size can reach several hundred kilobytes. The limitation on message volume in Kafka demands accurate forecasting of message sizes; if necessary, we split them, but this division is logical, related to business operations.
For example, goods arriving in a vehicle are divided by boxes. Separate microservices are allocated for synchronous operations, and thorough load testing is conducted. The use of Kafka posed another challenge for us β ensuring our service operates correctly with Kafka integration makes all our unit tests asynchronous. We solved this issue by writing our own utility methods using the Embedded Kafka Broker. This does not negate the need to write unit tests for separate methods, but we prefer to test complex cases using Kafka.
We paid a lot of attention to log tracing to ensure that their TraceId do not get lost when exceptions occur during service operations or when working with Kafka batches. While there were no particular issues with the first case, in the second case, we had to log all the TraceId that came with the batch and choose one to continue tracing. Thus, when searching by the original TraceId, the user can easily discover which tracing continued.
Task 3. The need to store a large amount of data: over 1 billion tags per year for tobacco alone arrives at X5. They require constant and quick access. Overall, the system should process about 10 billion records related to the movement history of tagged goods.
To solve the third task, the NoSQL database MongoDB was chosen. We have built a shard consisting of 5 nodes, with each node having a Replica Set of 3 servers. This allows the system to scale horizontally by adding. new servers to ensure its fault tolerance. Here we encountered another issue β ensuring transactional integrity in the MongoDB cluster while using horizontally scalable microservices. For instance, one of the tasks of our system is to detect attempts to resell items with identical marking codes. This leads to overlaps with erroneous scans or incorrect cashier operations. We found that such duplicates could arise both within a single processed Kafka batch and between two parallel batches. Thus, checking for duplicates by querying the database yielded no results. We addressed the problem for each microservice separately, based on the business logic of that service. For instance, for receipts, we added a check within the batch and a separate process for identifying duplicates during insertion.
To ensure that user operations with the transaction history did not impact the most critical aspect β the functioning of our business processes, we separated all historical data into a distinct service with its own database, which also receives information through Kafka. This way, users work with an isolated service, having no effect on the services processing current operations.
Task 4. Retrying queues and monitoring:
In distributed systems, problems and errors with the availability of databases, queues, and external data sources inevitably arise. In the case of 'Markus', the source of these errors is the integration with external systems. We needed to find a solution that would allow for retries on erroneous responses with a specified timeout, without halting the processing of successful requests in the main queue. To address this, the so-called 'topic-based retry' concept was chosen. For each main topic, one or more retry topics are created to which erroneous messages are directed, thereby eliminating delays in processing messages from the main topic. The interaction scheme β
To implement such a scheme, we needed to integrate this solution with Spring and avoid code duplication. We came across a similar solution based on Spring BeanPostProcessor, but it seemed overly cumbersome to us. Our team created a simpler solution that fits into the Spring cycle for creating consumers and additionally adds Retry Consumers. We proposed our prototype of the solution to the Spring team, which can be viewed. . The number of Retry Consumers and the number of attempts for each consumer can be configured through parameters, depending on the needs of the business process, and to make everything work, all that remains is to place the familiar annotation to all Spring developers: org.springframework.kafka.annotation.KafkaListener.
If a message cannot be processed after all retry attempts, it lands in the DLT (dead letter topic) through the Spring DeadLetterPublishingRecoverer. At the request of support, we expanded this functionality and created a separate service that allows viewing the messages that reached the DLT, along with stack traces, trace IDs, and other useful information about them. In addition, we added monitoring and alerts for all DLT topics, and now, in essence, the appearance of a message in the DLT topic serves as a reason for review and defect creation. This is very convenient β by the topic name, we immediately understand at which step of the process the problem occurred, significantly speeding up the search for its root cause.
Recently, we implemented an interface that allows messages to be resent by our support team after resolving their issues (for example, restoring the functionality of an external system) and, of course, creating the corresponding defect for analysis. Here our self-topics came in handy; to avoid restarting a lengthy processing chain, we can restart it from the needed step.
Platform Operation
The platform is already in productive operation; we conduct deliveries and ship goods every day, connecting new distribution centers and stores. As part of the pilot, the system operates with product groups 'Tobacco' and 'Footwear.'
Our entire team is involved in conducting pilots, analyzing emerging issues, and making suggestions for product improvements, ranging from log enhancements to process changes.
To avoid repeating past mistakes, all cases identified during the pilot are reflected in automated tests. A large number of automated tests and unit tests enable us to conduct regression testing and apply hotfixes literally within a few hours.
We are currently continuing to develop and enhance our platform, consistently encountering new challenges. If you're interested, we will share our solutions in upcoming articles.
Source: habr.com
