Hello!
My name is Mikhail, I am the Deputy Director of IT at Sportmaster. I want to share a story about how we dealt with the challenges that arose during the pandemic.
In the early days of the new reality, the familiar format of offline commerce at Sportmaster froze, and the load on our online channel, particularly in terms of client delivery, increased tenfold. Within a few weeks, we transformed our giant offline business into online, adapting our service to meet the needs of our customers.
In summary, what was essentially our secondary operation became our main business. The importance of each online order increased exponentially. We needed to retain every ruble that a customer brought to the company.
To swiftly respond to customer requests, we opened an additional contact center at the company headquarters, and now we can handle about 285,000 calls per week. At the same time, we transitioned 270 stores to a new format of contactless and safe operation, allowing customers to receive orders and employees to keep their jobs.
During the transformation process, we faced two main problems. Firstly, the load on our online resources significantly increased (how we tackled this will be explained by Sergey). Secondly, the flow of rare (pre-COVID) operations increased multiple times, which in turn required a large volume of rapid automation. To solve this problem, we had to quickly redirect resources from areas that were previously primary. How we managed this will be explained by Elena.
Operation of online services
Sergey Kolesnikov, responsible for the operation of the online store and microservices
Since the moment our retail stores began to close to visitors, we started recording an increase in metrics such as the number of users, the number of orders placed in our app, and the number of requests to the applications.
Number of orders from March 18 to March 31
Number of requests to the online payment microservices
Number of orders placed on the website
In the first graph, we see that the growth was approximately 14 times, while in the second, it was 4 times. We consider the response time metric of our applications to be particularly indicative.
In this graph, we see the responses from the front end and applications, and we've determined that we did not notice any significant growth.
This is primarily due to the fact that we began preparatory work at the end of 2019. Our services are now reserved, providing redundancy at the level of physical servers, virtualization systems, and containers. At the same time, our server resources are capable of handling multiple loads.
The main tool that helped us throughout this process has been our monitoring system. However, until recently, we did not have a unified system that could collect metrics at all levels, from physical hardware to business metrics.
Formally, there was monitoring in the company, but it was usually scattered and the responsibility lay with specific departments. In fact, when an incident occurred, we often did not have a unified understanding of what had happened, there was a lack of communication, and this often led to a repetitive cycle of searching for and localizing the problem for subsequent resolution.
At some point, we thought it was time to stop tolerating this — we needed a unified system to see the full picture. The main technologies in our stack include Zabbix as the alerting and metric storage center, Prometheus for collecting and storing application metrics, the ELK stack for logging and storing data for the entire monitoring system, as well as Grafana for visualization, Swagger, Docker, and other useful tools you're familiar with.
Moreover, we not only utilize available market technologies but also develop some solutions ourselves. For instance, we create services for system integration, which is essentially an API for collecting metrics. Additionally, we are working on our own monitoring systems — at the level of business metrics, we use UI tests. We also have a bot in Telegram for team notifications.
Moreover, we strive to make the monitoring system accessible for teams, allowing them to independently store their metrics and work with them, including setting alerts for specific metrics that have limited applications.
Throughout the entire system, we aim for proactivity and the fastest possible localization of incidents. Additionally, the number of our microservices and systems has significantly increased lately, which in turn raises the number of integrations. To optimize the incident diagnosis process at the integration level, we are developing a system that enables cross-system checks and outputs results, helping to identify the main issues related to imports and system interactions.
Naturally, we still have room for growth and development in system operations, and we are actively working on this. You can read more about our monitoring system here. .
Technical Trials
Sergey Orlov, heads the web and mobile development competency center
Since the closure of physical stores, we have faced various challenges in terms of development. The primary concern was the surge in load itself. It is clear that if appropriate measures are not taken, when the system is subjected to high loads, it could sadly turn into a pumpkin with a loud crash, either completely degrading in performance or losing its functionality altogether.
The second aspect, which is slightly less obvious, is that the system under high load needed to be changed very quickly to adapt to changes in business processes—sometimes several times a day. Many companies have a rule that during significant marketing activities, no changes should be made to the system at all. Just let it run if it's working.
In our case, we effectively had an endless Black Friday, during which we still needed to change the system. Any mistake, issue, or failure in the system would have been extremely costly for the business.
Looking ahead, I can say that we managed to meet these challenges; all systems withstood the load, scaled easily, and we experienced no significant technical failures.
There are four pillars that support the system's ability to withstand high burst loads. The first is monitoring, which you read about earlier. Without a well-established monitoring system, it is nearly impossible to identify bottlenecks in the system. A good monitoring system is like home clothing; it should be comfortable and tailored to you.
The second aspect is testing. We take this very seriously: we write both classic unit tests and integration tests, load tests, and many others for each system. We also draft a testing strategy, aiming to reach a level of testing where manual checks are no longer necessary.
The third pillar is the CI/CD Pipeline. The processes of building, testing, and deploying applications must be as automated as possible, with no manual interventions. The subject of CI/CD Pipeline is quite deep, and I will only touch on it briefly. It’s worth mentioning that we have a CI/CD Pipeline checklist, which each product team goes through with the help of competence centers.
Here is the checklist
This achieves many goals. It includes API versioning, feature toggles to avoid release train issues, and ensuring that various tests are covered to the extent that testing is fully automated, deployments are seamless, and more.
The fourth pillar consists of architectural principles and technical solutions. There is much to be said about architecture, but I want to highlight a couple of principles that I would like to focus on.
First of all, you need to choose specialized tools for specific tasks. Yes, it sounds obvious, and it's clear that nails should be hammered with a hammer, while wristwatches should be disassembled with special screwdrivers. However, in our age, many tools aim for universality to cover the maximum segment of users: databases, caches, frameworks, and others. For example, if we take the MongoDB database, it works with multi-document transactions, while Oracle database works with JSON. It may seem like everything can be used for anything. But if we advocate for performance, we need to clearly understand the strengths and weaknesses of each tool and use the ones that are necessary for our class of tasks.
Secondly, in system design, every increase in complexity must be justified. We must constantly keep this in mind; the principle of low coupling is well-known. I believe it should be applied at the level of individual services, throughout the entire system, and across the architectural landscape. The ability for horizontal scaling of each component under load is also important. If we possess this ability, scaling will be straightforward.
When it comes to technical solutions, we asked product teams to prepare a fresh set of recommendations, ideas, and solutions that they implemented in the preparation for the next wave of load.
Caches
We need to consciously approach the choice between local and distributed caches. Sometimes it makes sense to use both within a single system. For example, we have systems where part of the data essentially serves as a display cache, meaning the source of updates is outside the system itself, and these systems do not change that data. For such an approach, we use the local Caffeine Cache.
On the other hand, there are data that the system actively modifies during operation, and here we apply a distributed cache with Hazelcast. This approach allows us to leverage the benefits of distributed caches where they are truly needed and minimize service costs associated with data circulation in the Hazelcast cluster where we can do without it. We have written a lot about caches. and .
Additionally, switching the serializer to Kryo in Hazelcast has given us a considerable boost. The transition from ReplicatedMap to IMap + Near Cache in Hazelcast allowed us to minimize data movement across the cluster.
A small tip: when mass invalidating cache, sometimes the strategy of warming up a secondary cache before switching to it can be applicable. It may seem that with this approach we should experience double memory consumption, but in practice, in systems where this has been implemented, memory usage decreased.
Reactive stack
We use the reactive stack in quite a number of systems already. In our case, it's Webflux or Kotlin with coroutines. The reactive stack works especially well where we anticipate slow input-output operations. For example, calls to slow services, file system operations, or storage systems.
The most important principle is to avoid blocking calls. Under the hood of reactive frameworks, there is a small number of live service threads. If we recklessly allow a direct blocking call, such as a JDBC driver call, the system will simply halt.
Try to turn errors into your own runtime exceptions. The real execution flow of the program shifts to reactive frameworks, making code execution nonlinear. As a result, it becomes very difficult to diagnose issues through stack traces. A solution here would be to create clear and objective runtime exceptions for each error.
Elasticsearch
When using Elasticsearch, do not choose unused data. This is, in principle, a very simple advice, but it is often forgotten. If you need to select more than 10,000 records at once, you should use Scroll. If we draw an analogy, it resembles a cursor in a relational database.
Do not use postfilter unnecessarily. With large datasets in the main selection, this operation significantly burdens the database.
Use bulk operations where applicable.
API
When designing the API, consider the requirements for minimizing data transmission. This is especially relevant in conjunction with the front end: at this junction, we go beyond our data center channels and operate on the connection that links us to the client. If there are any minor issues on this channel, excessive traffic can lead to negative user experience.
And finally, don't throw out a pile of data; approach the contract between consumers and providers with clarity.
Organizational transformation
Elena Eroshkina, Deputy Director for IT
At the moment when the quarantine occurred and there was a need to rapidly increase the pace of online development and implement omnichannel services, we were already in the process of organizational transformation.
Part of our structure was transitioned to work according to the principles and practices of a product approach. Teams have been formed that are now responsible for the functioning and development of each product. Employees in such teams are engaged 100% and organize their work according to Scrum or Kanban, depending on which is preferable for them, set up deployment pipelines, implement technical practices, quality assurance practices, and much more.
By happy coincidence, most of these product teams were indeed focused on online and omnichannel services. This allowed us to quickly transition to remote work without losing efficiency (seriously, literally within two days). The established process facilitated rapid adaptation to new working conditions and sustained a sufficiently high pace of delivering new functionality.
Moreover, we found the need to strengthen those teams that are on the frontier of online business. At that moment, it became clear that we could only do this using internal resources. About 50 people changed the area they were working in before and integrated into the work on a new product within two weeks.
This did not require any special managerial efforts, because alongside organizing our own processes, enhancing the product technically, and ensuring quality, we teach our teams self-organization — to manage their own production process without involving administrative resources.
We were able to focus our managerial resources precisely where it was necessary at that moment — on coordinating with the business: What is currently important for our client, which functionalities should be implemented first, and what needs to be done to increase our throughput for delivery and order processing. All of this, along with a clear role model, allowed us during this period to overload our production flows with creating value based on what is truly important and needed.
It’s clear that, with remote work and a high pace of change, when each person's participation affects business metrics, we cannot rely solely on internal feelings like 'Is everything going well? It seems okay.' Objective metrics of the production process are necessary. We have them available to anyone interested in the metrics of product teams. Primarily for the team itself, the business, related parties, and management.
Every two weeks, a status meeting is held with each team, where metrics are analyzed for 10 minutes, bottlenecks in the production process are identified, and a joint solution is developed: what can be done to eliminate these bottlenecks. Here, one can also immediately seek assistance from management if any identified problem lies outside the teams' influence or requires the expertise of colleagues who may have encountered a similar issue.
Nevertheless, we understand that in order to achieve significant acceleration (which is our goal), we still need to learn a lot and implement it into our daily work. Right now, we are continuing to scale our product approach to other teams and new products. For this, we had to master a new format for us, which is an online school for methodologists.
Methodologists, the people who help teams establish processes, improve communication, and enhance work efficiency, are essentially change agents. Right now, graduates from our first cohort are working with teams, assisting them in becoming successful.
I believe that the current situation presents us with opportunities and prospects that, perhaps, we haven't fully grasped yet. However, the experience and practice we are gaining right now confirm that we have chosen the right path for development, and we won't miss out on these new opportunities in the future while effectively responding to the challenges that 'Sportmaster' will face.
Conclusions
During this challenging time, we have formulated the key principles that underpin software development, which I believe will be relevant for every company engaged in this field.
People. This is the foundation of everything. Employees must enjoy their work, understand the company's goals, and the objectives of the products they are involved with. And of course, they should have the opportunity for professional growth.
Technology. A company must approach its technology stack maturely and build competencies where they are truly necessary. This sounds very simple and obvious, yet it is often ignored.
Processes. It's crucial to properly organize the work of product teams and centers of excellence, establishing collaboration with the business to operate as partners.
In general, that's about how we survived. The main thesis of our times has been confirmed once again, with a resounding slap to the forehead.
Even if you are a huge offline business with many stores and numerous cities present, develop your online presence. It's not just an additional sales channel or a pretty app through which something can also be purchased (especially since competitors have pretty ones too). It's not a spare tire just in case, to help weather the storm.
It's an absolute necessity. One that not only your technical capabilities and infrastructure must be ready for, but also your people and processes. After all, quickly acquiring memory, space, spinning up new instances, and other tasks can be done in a couple of hours. However, preparing people and processes for this must be done in advance.
Source: habr.com
