We all love stories. We enjoy sitting around the campfire sharing tales of our past victories, battles, or simply our work experiences.
Today is one of those days. And while you may not be around a campfire right now, we have a story for you. A story about how we started working with storage on Tarantool.
Once upon a time, our company had a couple of 'monoliths' and one shared 'ceiling' to which these monoliths slowly but surely drew closer, limiting our company's flight and development. It was clear: one day we would hit that ceiling hard.
Now we embrace the ideology of separating everything, from hardware to business logic. As a result, we have two data centers that are practically independent at the network level. Back then, however, everything was quite different.
Today, there are countless tools and means for making changes, like CI/CD, K8S, and so on. During the 'monolithic' times, we didn't need so many foreign terms. We just needed to tweak the 'storage' in the database.
But time moved on, and the number of requests advanced along with it, pushing RPS sometimes beyond our capabilities. When we entered the CIS market, the load on the DB processor of the first monolith never dropped below 90%, while RPS hovered around 2400. And these were not just small selects, but hefty requests with numerous checks and JOINs that could span almost half the data against a backdrop of high IO.
When full-fledged sales began to appear on 'Black Friday'āwith Wildberries being one of the first to conduct them in Russiaāthe situation became quite dire. After all, the load on such days triples.
Ah, those 'monolithic times'! I'm sure you've encountered something similar and still can't comprehend how that could happen to you.
What can you doātrends belong to technology too. About five years ago, we had to rethink one of those trends involving an existing site on .NET and MS SQL Server, which meticulously stored all the site's operational logic. It was kept so meticulously that dismantling such a monolith turned out to be a long and quite challenging endeavor.
A brief digression.
At various events, I say, "if you haven't split the monolith, then you haven't grown!" Iām interested in your opinion on this matter, please share it in the comments.
And the thunder struck
Letās return to our 'fire'. To distribute the load of 'monolithic' functionality, we decided to split the system into microservices based on open-source technologies. Because, at the very least, their scaling is cheaper. And we knew for 100% that scaling would be necessary (and significantly so). By that time, we had already managed to enter the markets of neighboring countries, and the number of registrations, as well as orders, began to grow even more rapidly.
After analyzing the first candidates for the transition from the monolith to microservices, we realized that in 80% of cases, data entry into them is 99% coming from back office systems, while reading is from the front office. This primarily concerned a couple of critical subsystems for us ā user data and the system for calculating the final cost of goods based on information about additional customer discounts and coupons.
As a side note. It's hard to imagine now, but in addition to the aforementioned subsystems, our monolith also housed product catalogs, user carts, product search systems, product catalog filtering systems, and various recommendation systems. Each of them has separate classes of narrowly-focused systems, but once they all lived in one 'dacha'.
We planned to move our client data to a sharded system right away. The transfer of functionality for calculating the final cost of goods required good scalability for reading, as it created the highest load in terms of RPS and was the most complex to execute for the database (a lot of data is involved in the calculation process).
As a result, we came up with a scheme that works well with Tarantool.
At that time, the microservices operated using schemes with multiple data centers on virtual and physical machines. As shown in the diagrams, Tarantool replication options were applied in both master-master and master-slave modes.
Architecture. Option 1. User service
Currently, there are 24 shards, each with 2 instances (one in each data center), all in master-master mode.
Applications that interact with database replicas are built on top of the database. These applications communicate with Tarantool through our custom library, which implements the Go driver interface for Tarantool. It recognizes all replicas and can operate with the master for both reading and writing. Essentially, it implements a replica set model, enriched with logic for replica selection, retry execution, circuit breaker, and rate limiting.
Additionally, there's the option to configure the replica selection policy at the shard level. For instance, using round-robin.
Architecture. Option 2. Service for calculating the final cost of a product.
A few months ago, the majority of requests for calculating the final cost of products transitioned to a new service, which operates fundamentally without databases. However, some time ago, all 100% of these calculations were handled by the service with Tarantool in the background.
The database of the service consists of 4 masters, where the synchronizer collects data, and each of these masters distributes data to readonly replicas via replication. Each master has approximately 15 such replicas.
In both the first and second schemas, if one data center becomes unavailable, the application can still retrieve data from the second one.
It's worth noting that Tarantool's replication is quite flexible and can be configured at runtime. In other systems, there have been challenges. For instance, in PostgreSQL, changing the parameters max_wal_senders and max_replication_slots requires a master restart, which in some cases can lead to connection failures between the application and the database management system.
Seek and you shall find!
Why didnāt we do it 'like normal people' and instead choose an atypical approach? It depends on what you consider normal. Many people create clusters out of Mongo and distribute them across three geographically distributed data centers.
At that time, we already had two projects on Redis. The first was a cache, and the second served as a persistent storage for less critical data. This second one was quite challenging for us, partly due to our own faults. Sometimes, large volumes were stored in a single key, and occasionally the site would experience issues. This system was employed in a master-slave configuration, and there were numerous cases where something went wrong with the master, causing replication to break.
In other words, Redis is good for stateless tasks, not for stateful ones. In principle, it could solve most problems, but only if they were key-value solutions with a couple of indexes. However, at that time, Redis had quite poor persistence and replication capabilities. Additionally, there were concerns regarding its performance.
We considered MySQL and PostgreSQL. However, the former didn't quite fit with us, while the latter is a rather complex product, making it impractical to build simple services on it.
We tried RIAK, Cassandra, even a graph database. All of these are fairly niche solutions that were not suitable as a universal tool for building services.
Ultimately, we decided on Tarantool.
We approached it when it was at version 1.6. We were interested in its combination of key-value capabilities and the functionality of a relational database. It has secondary indexes, transactions, and spacesālike tables, but not simple ones, allowing for a varying number of columns. However, the killer feature of Tarantool was the secondary indexes combined with key-value and transaction support.
The responsive Russian-speaking community, ready to help in the chat, also played a role. We actively utilized it and essentially lived in the chat. One should not forget about its decent persistence without any obvious flaws. Looking at our history with Tarantool, we had many pains and screw-ups with replication, but we never lost data because of it!
The implementation started off difficult.
At that time, our main development stack was .NET, for which there was no connector for Tarantool. We immediately started doing something in Go. Lua also worked reasonably well. The main issue at that time was debugging: .NET had excellent debugging capabilities, but then transitioning to the world of embedded Lua, where you only have logs and no real debugging, was quite challenging. Moreover, for some reason replication periodically broke down, and we had to delve into the structure of the Tarantool engine. The chat helped with this, to a lesser extent the documentation, and sometimes we looked at the code. At that time, the documentation was rather lacking.
Over the course of several months, we managed to learn from our mistakes and achieve respectable results working with Tarantool. We documented our standard practices in Git, which assisted in the development of new microservices. For instance, when tasked with creating another microservice, the developer would refer to the source code of the reference solution in the repository, and the creation of a new service would take no more than a week.
Those were special times. At that time, you could just walk up to the admin at the neighboring desk and ask, "Give me a virtual machine." Within about thirty minutes, the machine would be yours. You would connect, install everything yourself, and they would route traffic to it.
Today thatās no longer possible: you have to set up monitoring for the service, logging, cover functionality with tests, order a virtual machine or deploy in Kubernetes, etc. Overall, it's better this way, although itās more time-consuming and cumbersome.
Divide and conquer. How are things with Lua?
There was a serious dilemma: some teams were unable to reliably deploy updates in services with heavy logic coded in Lua. This often led to service failures.
In other words, developers prepare a certain change. Tarantool starts to execute a migration, while the replica still has the old code; some DDL is replicated to it, and the code just breaks because it wasn't accounted for. As a result, the update procedure for admins was outlined on an A4 sheet: stop replication, update this, enable replication, disable here, update there. It was a nightmare!
As a result, we most often now try not to do anything in Lua. We simply use iproto (a binary protocol for server interaction), and thatās it. Perhaps this is a lack of knowledge among the developers, but from that perspective, the system is complex.
We don't always blindly follow this script. Today we have no black and white: either everything in Lua or everything in Go. We already understand how to combine them to avoid migration problems later.
Where is Tarantool now?
Tarantool is used in the service for calculating the final cost of products, including discount coupons, known as the 'Promotizer'. As mentioned earlier, it is now being phased out: it is being replaced by a new catalog service with pre-calculated prices. However, until six months ago, all calculations were done in the 'Promotizer'. Previously, half of its logic was written in Lua. Two years ago, the service was made into a storage solution, and the logic was rewritten in Go because the discount mechanics changed slightly and the service lacked performance.
One of the most critical services is the user profile. This means that all Wildberries users are stored in Tarantool, numbering around 50 million. It features a sharded system based on user ID, distributed across several data centers with a Go service wrapper.
At one time, 'Promotizer' was the leader in RPS, reaching up to 6,000 requests. At one point, we had 50-60 instances. Currently, the leader in RPS is the user profiles, handling around 12,000 requests. This service uses custom sharding, dividing by ranges of user IDs. It serves over 20 machines, but that's too many; we plan to reduce the allocated resources since it only requires the capabilities of 4-5 machines.
The sessions service was our first service built on vshard and Cartridge. The configuration of vshard and the update of Cartridge required a certain amount of effort, but in the end, everything turned out well.
The service for displaying different banners on the website and in the mobile app was one of the first released directly on Tarantool. This service is notable because it is about 6-7 years old, still operational, and has never been restarted. It utilized master-master replication. Nothing has ever broken.
There is an example of using Tarantool for quick reference functionality in the warehouse system to swiftly verify information in certain cases. We tried using Redis for this, but the data in memory took up more space than in Tarantool.
The services for waiting lists, client subscriptions, the currently trendy stories, and deferred products also operate with Tarantool. The latter service occupies about 120 GB in memory, making it the most resource-intensive one among those mentioned.
Conclusion
Thanks to secondary indexes combined with key-value and transactional capabilities, Tarantool is well-suited for microservices-based architectures. However, we encountered difficulties when deploying changes to services with extensive Lua logicāthese services often stopped functioning. We could not overcome this challenge, and over time, we arrived at various combinations of Lua and Go: we understand where to use one language and where to use the other.
What else to read on the topic
- Creating a high-load application on Tarantool from scratch
- A reliable choice for leaders in Tarantool Cartridge
- Tarantool Telegram channel with product news
- Discuss Tarantool in the community chat
Source: habr.com
