Hello! My name is Alexey Pyankov, I am a developer at Sportmaster. In this I shared how the work on the Sportmaster website began in 2012, what initiatives we managed to "push through" and conversely, what pitfalls we encountered.
Today I want to share my thoughts following another storyline – the choice of a caching system for the Java backend in the website admin panel. This topic is particularly significant for me – although the story unfolded over just 2 months, we worked 12-16 hours a day without a single day off. I had never before thought or imagined that one could work so much.
Therefore, I break the text into 2 parts, to avoid overwhelming you. On the contrary, the first part will be very light – a preparation, an introduction, some thoughts on what caching is. If you are already an experienced developer or have worked with caches – there probably won’t be anything new technically in this article. But for a junior developer, such a brief overview might hint at which direction to look if they find themselves at such a crossroads.
When the new version of the Sportmaster website was launched in production, the data was being transmitted in a manner that was, to put it mildly, not very convenient. The foundation consisted of tables prepared for the previous version of the website (Bitrix), which had to be pulled into ETL, transformed into a new format, and enriched with various embellishments from about a dozen other systems. For a new image or product description to appear on the site, one had to wait until the next day – updates were only done overnight, once a day.
Initially, there were so many concerns during the first weeks after going live that the content managers' inconveniences seemed trivial. But once everything settled down, project development continued — a few months later, at the beginning of 2015, we started actively developing the admin panel. In 2015 and 2016, everything was going well; we released updates regularly, and the admin panel began to cover an increasing amount of data preparation, preparing us for the time when our team would be entrusted with the most important and complex task — the product management (complete preparation and maintenance of data for all products). However, in the summer of 2017, just before launching the product management, the project found itself in a very difficult situation — precisely due to caching issues. This is the episode I want to talk about in the second part of this two-part publication.
But in this post, I will start from afar, summarizing some thoughts – ideas about caching that would have been a good step to consider beforehand in a major project.
When the task of caching arises
The task of caching doesn't come out of nowhere. We developers create software products and want them to be in demand. If the product is in demand and successful — users come in. More and more users keep arriving, and soon there are so many of them that the product becomes highly loaded.
At the early stages, we don't think much about code optimization and performance. The main focus is functionality: quickly rolling out a pilot version and testing hypotheses. And if the load increases — we upgrade the hardware. We double, triple, or increase it fivefold, even tenfold. There is a limit to finances, though. And how much will the number of users actually increase? It won't just be 2-5-10; in the event of success, it could be from 100 to 1,000 and up to 100,000 times. So sooner or later, optimization will have to be addressed.
Let's assume that a certain part of the code (let's call this part a function) takes an unreasonably long time to execute, and we want to reduce the execution time. A function could be database access or executing some complex logic — the main point is that it takes a long time. How much can we reduce the execution time? In theory, it can be reduced to zero, but not further. And how can we reduce the execution time to zero? The answer: by completely excluding execution. Instead, we simply return the result. But how do we know the result? The answer: either compute it, which takes time, or look it up somewhere. Computing is slow. Looking it up means, for example, remembering the result that the function returned the last time it was called with the same parameters.
This means that the implementation of the function is not important to us. It's sufficient to know which parameters determine the result. Then, if we represent the parameter values in an object that can be used as a key in some storage, we can save the computed result and retrieve it the next time we call it. If these read-write operations are faster than executing the function, we gain speed. The speed gain can reach 100, 1000, or even 100,000 times (10^5 is more of an exception, but in cases with significantly lagging databases, it is quite possible).
Key requirements for a caching system
The first requirement for a caching system can be fast read speeds and, to a slightly lesser extent, write speeds. This is true, but only until we roll out the system into production.
Let's explore a case.
Suppose we have equipped the hardware for the current load and are gradually implementing caching. The number of users is slowly increasing, the load is growing — we add caches here and there. This continues for some time, and now heavy functions are rarely called — all the main load falls on the cache. The number of users has increased by N times during this period.
If the initial hardware capacity could handle 2-5 times the load, then with caching we could improve performance by 10 times or, in a good scenario, 100 times, and possibly even 1000 times in certain cases. This means that on the same hardware, we process 100 times more requests. Fantastic, we deserve a reward!
But then, at one point, by chance, the system crashed and the cache failed. Nothing special – after all, the cache was chosen based on the requirement of 'high read and write speed, everything else doesn’t matter.'
In terms of initial load, we had 2-5 times hardware headroom, but during that time the load increased by 10-100 times. With the cache, we excluded calls for heavy functions, so everything was running smoothly. But now, without the cache – how much will our system drop? What will happen to us? The system will crash.
Even if our cache hasn't crashed, but has only been cleared temporarily – it will need to be warmed up, and this will take some time. During this time, the main load will fall on functionality.
Conclusion: high-load projects in production require the caching system not only to have high read and write speeds, but also to ensure data integrity and fault tolerance.
The agony of choice
In a project with an admin panel, the selection went as follows: we initially set up Hazelcast, as we were already familiar with this product from our experience with the main site. However, this choice turned out to be unsuccessful – under our load profile, Hazelcast operates not just slowly, but horrendously slowly. And by the deadlines for going live, we had already committed.
Spoiler: I will tell you in the second part how the circumstances unfolded that we missed such a blunder and ended up in a tense situation – how we got there and how we got out. But for now, let me just say that it was a strong stress, and 'thinking – somehow doesn’t come to mind, let’s shake the bottle.' 'Shaking the bottle' is also a spoiler, about that a bit later.
What we did:
- We compiled a list of all the systems suggested by Google and StackOverflow. A little over 30.
- We wrote load tests characteristic of production. For this, we recorded the data that passes through the system in a production environment – a sort of sniffer for data not on the network, but within the system. We ran these specific data in the tests.
- As a team, each person chooses the next system from the list, sets it up, and runs tests. If a test doesn't pass, doesn't handle the load – we discard it and move on to the next in line.
- At the 17th system, it became clear that all hope was lost. Enough with the 'shaking the bottle', it was time to think seriously.
But this is a scenario where you need to choose a system that will meet the speed requirements in pre-prepared tests. What if there are no such tests yet and you want to choose something faster?
Let's model this scenario (it's hard to imagine that a mid-level developer lives in a vacuum and hasn't already formed a preference for which product to try first — thus, the following discussions are more theoretical/philosophical/about juniors).
Having determined the requirements, we will start selecting an out-of-the-box solution. Why reinvent the wheel: we will take an existing caching system.
If you are just starting out and will be Googling, the order will be roughly similar, but generally, the targets will be as follows. First, you'll come across Redis, which is well-known everywhere. Then you'll learn that there’s EhCache, the oldest and most trusted system. Next will be information about Tarantool — a domestic development with a unique aspect to the solution. Also Ignite, because it's currently gaining popularity and is supported by SberTech. Finally, there’s Hazelcast, which often appears in the enterprise world among large companies.
This list is not exhaustive; there are dozens of systems. We will only attach one. Let’s take the five selected systems for a 'beauty contest' and conduct a selection. Who will be the winner?
Redis
Let's read what is written on the official website.
— an open-source project. Offers in-memory data storage, the possibility of on-disk saving, automatic partitioning, high availability, and recovery after network outages.
Everything seems excellent; you can take it and integrate it — it does everything you need. But let's take a look at the other candidates just out of curiosity.
EhCache
— 'the most widely used cache for Java' (translation of the slogan from the official site). Also open-source. And here we understand that Redis is not specifically for Java; it's a general system, and a wrapper is needed for interaction. EhCache will be more convenient. What else does the system promise? Reliability, proven effectiveness, full functionality. Oh, and it also caches terabytes of data.
Redis is forgotten; I’m ready to choose EhCache.
But my sense of patriotism urges me to see what’s good about Tarantool.
Tarantool
— is referred to as the "Real-time Data Integration Platform." It sounds very complex, so let's read the page thoroughly and find a bold statement: "Caches 100% of data in RAM." This raises questions — after all, data can significantly exceed memory capacity. The clarification here is that Tarantool does not perform serialization for writing data to disk from memory. Instead, it utilizes low-level system features where memory is simply mapped to the filesystem with quite impressive I/O performance. Overall, they've done something remarkably great.
Let's look at the implementations: Mail.ru corporate backbone, Avito, Beeline, Megafon, Alfa-Bank, Gazprom…
If there were still any doubts about Tarantool, the implementation case at Mastercard seals the deal for me. I choose Tarantool.
But still…
Ignite
… there’s also , claimed as "an in-memory computing platform… in-memory speeds at petabytes of data." There are many advantages here: a distributed in-memory cache, the fastest key-value storage and cache, horizontal scalability, high availability, strict integrity. So, it turns out that the fastest one is Ignite.
Implementations: Sberbank, American Airlines, Yahoo! Japan. Then I learn that Ignite is not just implemented at Sberbank, but the SberTech team sends their staff to the Ignite team to enhance the product. This completely wins me over, and I’m ready to take Ignite.
Completely unclear why, I look at the fifth point.
Hazelcast
I visit the site , read it. And it turns out that the fastest solution for distributed caching is Hazelcast. It is orders of magnitude faster than all other solutions and is, in fact, a leader in in-memory data grids. Against this backdrop, choosing something else would mean disrespecting oneself. Plus, it uses redundant data storage for continuous cluster operation without data loss.
That's it, I'm ready to choose Hazelcast.
Comparison
But upon reviewing, all five candidates are described in such a way that each of them is the best. How to choose? We can look at which one is the most popular, search for comparisons, and the headache will go away.
We find such a , selecting our 5 systems.
They are sorted here: at the top is Redis, in second place — Hazelcast, gaining popularity are Tarantool and Ignite, and EhCache remains as it was.
But let’s look at the : links to websites, general interest in the system, job offers — great! So, when my system fails, I will say: “No, it’s reliable! There are many job offers…”. Such a simple comparison won't work.
All these systems are not just caching systems. They have a lot more functionality, including the fact that instead of client data being sent for processing, code that needs to be executed on the data moves to the server, is executed there, and the result is returned. They are not often considered purely as caching systems.
Alright, we won’t give up, let's find a direct comparison of the systems. We’ll take the top two options — Redis and Hazelcast. We are interested in speed, so we'll compare them based on this parameter.
Hz vs Redis
We find this :
Blue is Redis, red is Hazelcast. Hazelcast wins everywhere, and there’s justification for this: it’s multi-threaded, highly optimized, and each thread works with its own partition, so there are no locks. Redis is single-threaded and does not benefit from modern multi-core CPUs. Hazelcast uses asynchronous I/O, while Redis-Jedis uses blocking sockets. Ultimately, Hazelcast employs a binary protocol, whereas Redis is text-oriented, meaning it is inefficient.
Just in case, let’s refer to another source of comparison. What will it show us?
Redis vs Hz
Another one :
Here, it's the opposite; red is Redis. This means that Redis outperforms Hazelcast. In the first comparison, Hazelcast won; in the second, Redis. It turns out the first result was actually manipulated: Redis was taken in its basic form, while Hazelcast was fine-tuned for the test case. So, firstly, you can’t trust anyone, and secondly, when we do choose a system, we still need to configure it properly. These configurations include dozens, almost hundreds of parameters.
Shaking the bottle
And the whole process we’ve just gone through can be explained with the metaphor “Shaking the bottle.” That is, right now programming isn’t the main thing; the key is to know how to read stackoverflow. And I have a person on my team, a professional, who operates just like that in critical moments.
And the whole process we've just gone through can be explained by the metaphor of 'Shaking the Bottle.' In other words, right now, it's not about programming; it's primarily about being able to read Stack Overflow. And I have a professional in my team who works exactly like that in critical moments.
What does he do? He sees a broken thing, looks at the stack trace, picks out some words from it (which ones is his expertise in the program), searches on Google, finds Stack Overflow among the answers. Without reading or thinking deeply, among the responses to the question, he chooses something that closely resembles the phrase 'do this and that' (choosing such an answer is his talent, as it's not always the one that received the most likes), applies it, and checks: if something changed, great. If nothing changed — we revert. And we repeat the run-check-search process. In this intuitive way, he eventually gets the code to work. He doesn’t know why, doesn’t know what he did, can’t explain. But! It works. And 'the fire is put out.' Now let's figure out what we did. When the program works — it’s much easier. And it saves a lot of time.
This method is very well explained through the following example.
Once, it was very popular to build a sailing ship in a bottle. The ship is large and fragile, while the neck of the bottle is very narrow, making it impossible to push the ship inside. How do you assemble it?
There is a method for this, very quick and very effective.
The ship consists of a bunch of small parts: sticks, ropes, sails, glue. We put all of this in the bottle.
We take the bottle with both hands and start shaking it. We shake it and shake it. And typically — it results in a complete mess, of course. But sometimes. Sometimes, it results in a ship! More precisely, something that resembles a ship.
We show this something to someone: 'Sergey, do you see!?'. And indeed, from a distance — it looks like a ship. But we can't let it go further.
There is another method. More advanced guys, such as hackers, use it.
I gave such a guy a task, he finished it and left. And you look — it seems completed. But some time later, when it's time to refine the code — it starts to unravel because of him… Good thing he was able to run away far. These are the guys who will take the bottle as an example and do this: you see, where the bottom is — the glass bends. And it’s not quite clear whether it’s transparent or not. So the 'hackers' saw off this bottom, insert the ship inside, reattach the bottom, and it looks as if that’s how it should be.
From the perspective of task definition, everything seems correct. But take the example of ships: why even create this ship, who really needs it? It serves no functionality. Typically, such ships are gifts for very high-ranking individuals, who display them as a symbol or a sign. And if such a person, a leader of a large business or a high-level official, has a shoddy piece with a broken neck as a flag, it's better if he never finds out. So, how are these ships made that can be gifted to important people?
The only place, a key aspect, that really cannot be altered is the hull. And the hull of the ship passes through the neck. Meanwhile, the ship is assembled outside of the bottle. But it's not just about assembling the ship; it's a true artisan craft. Special levers are added to the components, allowing them to be lifted later. For instance, the sails are folded, carefully inserted inside, and then with the help of tweezers, they are meticulously pulled and raised. As a result, it becomes a work of art that can be gifted with a clear conscience and pride.
And if we want the project to be successful, there must be at least one artisan in the team. Someone who cares about the quality of the product and considers all aspects without sacrificing any, even in times of stress when circumstances demand delivering quickly at the expense of what is important. All successful projects that are resilient and have stood the test of time are built on this principle. They possess something very precise and unique, utilizing every available opportunity. In the example of a ship in a bottle, the hull of the ship passing through the neck is played with.
Returning to the task of choosing our caching server, how could this approach be applied? I suggest a selection method from all available systems — rather than shaking the bottle, we should look into what exactly is available, focusing on the key aspects to consider when choosing a system.
Where to look for the bottle-neck
Let's try not to shake the bottle, not to go through everything in order, but see what tasks arise if, suddenly, we need to design such a system independently. We certainly won't reinvent the wheel, but we'll use this scheme to orient ourselves regarding which aspects to focus on in product descriptions. Let's sketch out such a scheme.
If the system is distributed, it means we will have several servers (6). Let's say four (it's easy to place them in the picture, but, of course, there can be as many as needed). If the servers are on different nodes, it means there’s some code running on them that ensures these nodes form a cluster and reconnect to recognize each other in case of a break.
We also need the code-logic (2) specifically for caching. This code interacts with clients via some API. The client code (1) can either run within the same JVM or connect to it over the network. The logic implemented internally determines which objects to keep in the cache and which to discard. We use memory (3) for cache storage, but if necessary, we can save some data on disk (4).
Let's look at where the load will arise. Each arrow and each node will indeed be under load. Firstly, between the client code and the API, if it's a network interaction, the drop might be quite noticeable. Secondly, within the API itself—if we overdo it with complex logic, we might run into CPU issues. It would be good if the logic doesn't unnecessarily push memory. Finally, we have interactions with the file system—typically this involves serialization/reconstruction and writing/reading.
Next is the interaction with the cluster. Most likely it will be within this same system but can also be separate. Here, we must consider data transfer to it, serialization speed, and interactions within the cluster.
Now, on one hand, we can visualize "which gears will turn" in the caching system when processing requests from our code, and on the other hand, we can estimate which and how many requests our code will generate to this system. This is enough to make a reasonably sober choice—selecting a system tailored to our use case.
Hazelcast
Let's see how such a breakdown can be applied to our list. For example, Hazelcast.
To put/get data from Hazelcast, the client code interacts (1) with the API. Hz allows the server to run as embedded, and in this case, calling the API is a method invocation within the JVM, which can be considered free of cost.
For the logic to work in (2), Hz relies on a hash from the byte array of the serialized key — that is, key serialization will occur in any case. This is an unavoidable overhead for Hz.
Eviction strategies are well implemented, but for special cases, you can connect your own. You don't need to worry about this part.
The storage (4) can be connected. Excellent. Interaction (5) for embedded can be considered instantaneous. Data exchange between nodes in the cluster (6) — yes, it exists. This contributes to fault tolerance at the cost of speed. The Hz feature Near-cache allows lowering the cost — data obtained from other nodes in the cluster will be cached.
What can be done in such conditions to increase speed?
For example, to avoid serializing the key in (2) — on top of Hazelcast, attach another cache for the hottest data. At Sportmaster, Caffeine was chosen for this purpose.
To tweak at level (6), Hz offers two types of storage: IMap and ReplicatedMap.
It's worth mentioning how Hazelcast ended up in the technology stack of Sportmaster.
In 2012, when we were working on the very first pilot of the future website, Hazelcast turned out to be the first link provided by the search engine. The acquaintance started 'at first sight' — we were impressed that just two hours after we integrated Hz into the system, it was running. And it was running well. By the end of the day, we wrote some tests and were pleased. And this burst of energy was enough to overcome the surprises that Hz threw at us over time. Currently, the Sportmaster team has no reason to abandon Hazelcast.
However, arguments like 'the first link in the search engine' and 'quickly assembled HelloWorld' are, of course, exceptions and characteristics of the moment during the selection process. The real tests for the chosen system begin with going live, and this stage should be given attention when choosing any system, including caches. In our case, it can be said that we selected Hazelcast by chance, but then it turned out that we made the right choice.
For production, much more important are: monitoring, handling failures on individual nodes, data replication, the cost of scaling. This means it is worth paying attention to the tasks that will arise during system maintenance—when the load exceeds the planned capacity by several times, when we accidentally upload something wrong or to the wrong place, and when we need to roll out a new version of the code, replace data, and do this unnoticed by clients.
For all these requirements, Hazelcast is undoubtedly suitable.
To be continued
But Hazelcast is not a panacea. In 2017, we chose Hazelcast for caching in the admin panel, simply based on the good impression from past experience. This played a key role in a very bad joke, which left us in a difficult situation and we "heroically" pulled ourselves out of it over 60 days. But more on that in the next part.
And for now… Happy New Code!
Source: habr.com
