Hello! My name is Alexey Pyankov, I'm a developer at Sportmaster. In that I told how work on the Sportmaster website began in 2012, what initiatives we managed to “push through” and vice versa, what kind of rake we collected.
Today I want to share thoughts that follow another story - choosing a caching system for the java backend in the site admin. This plot is of particular importance to me - although the story unfolded for only 2 months, but during these 60 days we worked 12-16 hours and without a single day off. I never thought or imagined that it was possible to work so hard before.
Therefore, the text is divided into 2 parts, so as not to download in full. On the contrary, the first part will be very easy - preparation, introduction, some considerations, what is caching. If you are already an experienced developer or have worked with caches, there will most likely be nothing new in this article from the technical side. But for a junior, such a small review can tell you which way to look, if he finds himself at such a crossroads.
When the new version of the Sportmaster website was launched into production, the data was received in a way that was, to put it mildly, not very convenient. The tables prepared for the previous version of the site (Bitrix) served as the basis, which had to be tightened into ETL, brought to a new look and enriched with various little things from a dozen more systems. In order for a new picture or product description to appear on the site, it was necessary to wait until the next day - the update is only at night, once a day.
At first, there were so many worries from the first weeks of release in the production that such inconvenience of content managers was a trifle. But, as soon as everything settled down, the development of the project continued - a few months later, at the beginning of 2015, we began to actively develop the admin panel. In 2015 and 2016 everything is going well, we regularly release, the admin panel covers more and more data preparation and we are preparing for the fact that soon our team will be entrusted with the most important and complex - the product outline (full preparation and maintenance of data for all products). But in the summer of 2017, just before the launch of the product circuit, the project will find itself in a very difficult situation - precisely because of caching problems. I want to talk about this episode in the second part of this two-part publication.
But in this post, I'll start from afar, like some thoughts - ideas about caching, which it would be a good step to scroll through before a big project.
When the task of caching occurs
The caching task does not just appear. We are developers, we are writing a software product and we want it to be in demand. If the product is in demand and successful, users arrive. And more and more are coming. And there are a lot of users and then the product becomes highly loaded.
At the first stages, we do not think about code optimization and performance. The main thing is functionality, quickly rolling out a pilot and testing hypotheses. And if the load grows, we pump iron. We increase it two or three times, five times, even 10 times. Somewhere here - finances will no longer allow. How many times will the number of users grow? It will not be like 2-5-10, but if successful, it will be from 100-1000 to 100 thousand times. That is, sooner or later, but optimization will have to be done.
Let's say some part of the code (let's call this part a function) takes an obscenely long time, and we want to reduce the execution time. A function - it can be access to a database, it can be the execution of some complex logic - the main thing is that it takes a long time. By how much can the execution time be reduced? In the limit - you can reduce to zero, no further. And how can you reduce the execution time to zero? Answer: exclude execution altogether. Instead, immediately return the result. And how to find out the result? Answer: either calculate, or peep somewhere. Computing is a long time. And to peep is, for example, to remember the result that the function produced last time when called with the same parameters.
That is, the implementation of the function is unimportant to us. It is enough to know on what parameters the result depends. Then, if the parameter values are represented as an object that can be used as a key in some storage, then we can save the result of the calculation and read it at the next call. If these write-read results are faster than the execution of the function, we have a profit in terms of speed. The amount of profit can reach 100, and 1000, and 100 thousand times (10 ^ 5 is rather an exception, but in the case of a decently lagging base, it is quite possible).
Basic requirements for a caching system
The first thing that may become a requirement for a caching system is a fast read speed and, to a slightly lesser extent, a write speed. This is true, but only until we roll out the system to production.
Let's play this case.
Let's say we have provided the current load with iron and now we are gradually introducing caching. Users grow a little, the load grows - we add a little caches, fasten it here and there. This continues for some time, and now heavy functions are practically not called anymore - all the main load falls on the cache. The number of users during this time has grown N times.
And if the initial supply of hardware could be 2-5 times, then with the help of the cache we could increase the performance by 10 times or, in a good case, by 100 times, in some places, perhaps by 1000. That is, on the same hardware, we process 100 times more requests. Wonderful, you deserve a gingerbread!
But now, at one fine moment, by accident, the system crashed and the cache collapsed. Nothing special - after all, the cache was chosen on the basis of the requirement "high read and write speed, the rest does not matter."
Regarding the starting load, we had 2-5 times the reserve for iron, and the load during this time has grown by 10-100 times. With the help of the cache, we excluded calls for heavy functions and therefore everything flew. And now, without cache - how many times will our system sag? What will happen to us? The system will fall.
Even if our cache has not collapsed, but only cleared for a while, it will need to be warmed up, and this will take some time. And at this time - the main burden will fall on the functionality.
Conclusion: highly loaded projects in production require from the caching system not only high read and write speeds, but also data integrity and failure tolerance.
Flour choice
In a project with an admin panel, the choice went like this: Hazelcast was installed first, because. were already familiar with this product from the experience of the main site. But, here such a choice turned out to be unsuccessful - under our load profile, Hazelcast is not just slow, but terribly slow. And at that time we had already signed under the terms of withdrawal to the prod.
Spoiler: how exactly the circumstances developed that we missed such a splash and got an acute and tense situation - I will tell you in the second part - and how we ended up and how we got out. But now - I can only say that it was a lot of stress, and "thinking - somehow I don’t think, we shake the bottle." “Shaking the bottle” is also a spoiler, more about that a little further.
What we did:
- We make a list of all the systems that google and StackOverflow suggest. A little over 30
- We write tests with a load typical for production. To do this, we recorded the data that passes through the system in a production environment - a kind of sniffer for data not on the network, but inside the system. Exactly this data was run in tests
- With the whole team, everyone chooses the next system from the list, sets up, runs the tests. Does not pass the test, does not pull the load - throw it away, go to the next one in the queue.
- On the 17th system, it became clear that everything was hopeless. Enough "shaking the bottle", it's time to seriously think.
But this is an option when you need to choose a system that will “crawl in speed” in pre-prepared tests. And if there are no such tests yet and you want to choose quickly?
Let's simulate such an option (it's hard to imagine that a middle + developer lives in a vacuum, and at the time of the choice has not yet formalized a preference for which product to try in the first place - therefore, further reasoning is more of a theorist / philosophy / pro junior).
Having decided on the requirements, we will begin to choose a solution out of the box. Why reinvent the wheel: we'll go ahead and grab a ready-made caching system.
If you are just starting and will google, then plus or minus the order, but in general, the guidelines will be like this. First of all, you will stumble upon Redis, it is all over the place. Then you will find out that there is EhCache as the oldest and most proven system. Further, it will be written about Tarantool - a domestic development, in which there is a unique aspect of the solution. And also Ignite, because it is now on the rise in popularity and enjoys the support of SberTech. At the end, there is Hazelcast, because in the enterprise world it often flickers among large companies.
This list is not exhausted, there are dozens of systems. And we will screw only one. Let's take the selected 5 systems to the "beauty contest" and make a selection. Who will be the winner?
Redis
We read what they write on the official website.
- open source project. Offers in-memory data storage, on-disk persistence, auto-partitioning, high availability, and network break recovery.
It seems that everything is fine, you can take it and screw it on - everything you need, it does. But just for the sake of interest, let's look at the other candidates.
EhCache
- "the most widely used cache for Java" (translation of the slogan from the official site). Also opensource. And here we understand that Redis is not for java, but general, and a wrapper is needed to interact with it. And EhCache will be more convenient. What else does the system promise? Reliability, verification, full functionality. Well, it's also the most common. And caches terabytes of data.
Redis is forgotten, I'm ready to choose EhCache.
But the feeling of patriotism pushes me to see what Tarantool is good for.
Tarantool
- meets the designation "Platform for real-time data integration." It sounds very complicated, so we read the page in detail and find a loud statement: "Caches 100% of data in RAM." This should raise questions - after all, there can be much more data than memory. The explanation is that it is implied here that Tarantool does not run serialization to write data to disk from memory. Instead, it uses the low-level features of the system, where memory is simply mapped onto a file system with very good I/O performance. In general, they did something wonderful and cool.
Let's look at implementations: Mail.ru corporate backbone, Avito, Beeline, Megafon, Alfa-Bank, Gazprom...
If there were still some doubts about Tarantool, then the implementation case in Mastercard finishes me off. I take Tarantool.
But anyway…
Ignite
… is there some more , is billed as "an in-memory computing platform ... in-memory speeds on petabytes of data." There are also many advantages here: distributed in-memory cache, the fastest key-value storage and cache, horizontal scaling, high availability, strict integrity. In general, it turns out that the fastest is Ignite.
Implementations: Sberbank, American Airlines, Yahoo! Japan. And then I also find out that Ignite is not just implemented in Sberbank, but the SberTech team sends its people to the Ignite team to finalize the product. It is completely captivating and I am ready to take Ignite.
It is completely incomprehensible why, I look at the fifth point.
hazelcast
I go to the site , reading. And it turns out that the fastest solution for distributed caching is Hazelcast. It is orders of magnitude faster than all other solutions and in general it is a leader in the field of in-memory data grid. Against this background, take something else - do not respect yourself. It also uses redundant data storage for continuous operation of the cluster without data loss.
That's it, I'm ready to take the Hazelcast.
Comparison
But if you look, then all five candidates are scheduled in such a way that each of them is the best. How to choose? We can see which one is the most popular, look for comparisons, and the headache will pass.
We find such , choose our 5 systems.
Here they are sorted: Redis is at the top, Hazelcast is in second place, Tarantool and Ignite are gaining popularity, EhCache has been and remains.
But let's look at : links to websites, general interest in the system, job offers - great! That is, when my system falls, I will say: “No, it is reliable! Here are a lot of job offers ... ". Such a simple comparison will not work.
All of these systems are not just caching systems. They still have a lot of functionality, including when not the data is transferred to the client for processing, but vice versa: the code that needs to be executed on the data moves to the server, it is executed there, and the result is returned. And they are not often considered as a separate system for caching.
Okay, let's not give up, let's find a direct comparison of systems. Let's take the top two options - Redis and Hazelcast. We are interested in speed, according to this parameter we will compare them.
hz vs redis
We find this :
Blue is Redis, red is Hazelcast. Hazelcast wins everywhere, and this is justified: it is multi-threaded, highly optimized, each thread works with its own partition, so there are no locks. And Redis is single-threaded, it does not take advantage of modern multi-core CPUs. Hazelcast has asynchronous I/O, Redis-Jedis has blocking sockets. After all, Hazelcast uses a binary protocol and Redis is text oriented, which means it's inefficient.
Just in case, let's turn to another source of comparison. What will he show us?
Redis vs Hz
Another :
Here, on the contrary, red is Redis. That is, Redis outperforms Hazelcast in terms of performance. In the first comparison, Hazelcast won, in the second - Redis. explained very precisely why Hazelcast won in the previous comparison.
It turns out that the result of the first one was actually rigged: Redis was taken in the base box, and Hazelcast was sharpened for a test case. Then it turns out: firstly, no one can be trusted, and secondly, when we do choose a system, we still need to configure it correctly. These settings include dozens, almost hundreds of parameters.
Shaking the bottle
And the whole process that we have now done, I can explain with such a metaphor "Shaking the bottle." That is, now you don’t have to program, now the main thing is to be able to read stackoverflow. And in my team there is a person, a professional, who works exactly in this way at critical moments.
What is he doing? He sees a non-working contraption, sees a stack trace, takes some words from it (which ones are his expertise in the program), searches on Google, finds stackoverflow among the answers. Without reading, without thinking, among the answers to the question - he chooses something that is most similar to the proposal "to do this and that" (choosing such an answer is his talent, because it is not always the answer that collected more likes), applies , looks: if something has changed, then great. If it hasn't changed, roll back. And repeat the launch-check-search. And in such an intuitive way, he achieves that after a while the code works. He doesn't know why, he doesn't know what he did, he can't explain. But! This infection works. And the fire is extinguished. Now let's see what we did. When the program works, it's much easier. And it saves a lot of time.
This method is very well explained with an example.
It was once very popular to collect a sailboat in a bottle. At the same time, the sailboat is large and fragile, and the neck of the bottle is very narrow, it cannot be pushed inside. How to assemble it?
There is such a method, very fast and very effective.
The ship consists of a bunch of little things: sticks, ropes, sails, glue. We put it all in a bottle.
We take the bottle with both hands and start shaking. We shake it, shake it. And usually - it turns out complete garbage, of course. But sometimes. Sometimes you get a ship! More precisely, something like a ship.
We show this something to someone: “Serge, see!?”. And indeed, from afar - like a ship. But it can't go any further.
There is another way. The guys are more advanced, such hackers.
I gave this guy a task, he did everything and left. And look - it seems to be done. And after a while, when it is necessary to finalize the code, this starts because of him ... It’s good that he has already managed to run far away. These are the guys who, using the example of a bottle, will do this: you see, where the bottom is, the glass bends. And it is not entirely clear whether it is transparent or not. Then the "hackers" saw off this bottom, insert a ship there, then glue the bottom again, and as if it were necessary.
From the point of view of setting the task, everything seems to be correct. But on the example of ships: why make this ship at all, who needs it at all? It doesn't carry any functionality. Usually such ships are gifts to very high-ranking people who put it on a shelf above them, as some kind of symbol, as a sign. And now, if such a person, the head of a large business or a high-ranking official, how will such a hack, whose neck has been cut off, have a flag? It would be better if he never knew about it. So, how do these ships, which can be presented to an important person, be made in the end?
The only place, the key, with which nothing can really be done, is the body. And the hull of the ship just passes through the neck. Whereas the ship is going outside the bottle. But it's not just to assemble a ship, it's a real jewelry craft. Special levers are added to the components, which then allow them to be lifted. For example, sails are folded, carefully brought inside, and then with the help of tweezers, very jewelry, exactly, they are pulled up and raised. The result is a work of art that can be presented with a clear conscience and pride.
And if we want the project to be successful, there must be at least one jeweler in the team. The one who cares about the quality of the product and takes into account all aspects, without sacrificing any even in moments of stress, when circumstances require the urgent to be done at the expense of the important. All successful projects that are sustainable, that have stood the test of time, they are built on this principle. There is something very precise and unique about them, something that takes advantage of all the possibilities available. In the example with a ship in a bottle, the fact that the ship's hull passes through the neck is played up.
Returning to the task of choosing our caching server, how could this method be applied? I offer such an option of choosing from all the systems that exist - do not shake the bottle, do not choose, but see what, in principle, they have, what to look for when choosing a system.
Where to look for bottle neck
Let's try not to shake the bottle, not to go through everything in turn, but let's see what tasks will arise, if suddenly, for our task - to design such a system on our own. Of course, we will not assemble a bicycle, but we will use this scheme to find out what points to pay attention to in product descriptions. Let's sketch such a scheme.
If the system is distributed, then we will have several servers (6). Let's say four (it's convenient to place in the picture, but, of course, there can be as many as you like). If the servers are on different nodes, it means that some code is spinning on all of them, which is responsible for ensuring that these nodes form a cluster and, in the event of a break, connect and recognize each other.
We also need code-logic (2), which is actually about caching. Clients interact with this code via some API. The client code (1) can be both within the same JVM and access it over the network. The logic implemented inside is the decision which objects to leave in the cache, which ones to throw out. We use memory (3) to store the cache, but if necessary, we can save some of the data on disk (4).
Let's see in which parts the load will occur. Actually, each arrow and each node will be loaded. First, between the client code and the api, if it's a network interaction, the drawdown can be quite noticeable. Secondly, within the framework of the api itself - having overdone it with complex logic, we can run into the CPU. And it would be nice if logic did not drive memory once again. And the interaction with the file system remains - in the usual case, this is serialize / restore and write / read.
Further interaction with the cluster. Most likely, it will be in the same system, but it may be separate. Here, too, you need to take into account the transfer of data to it, the speed of data serialization and interaction between the cluster.
Now, on the one hand, we can imagine “what gears will spin” in the cache system when processing requests from our code, and on the other hand, we can estimate what and how many requests our code will generate to this system. This is enough to make a more or less sober choice - to choose a system for our use case.
hazelcast
Let's see how to apply this decomposition to our list. For example, Hazelcast.
In order to put/take data from Hazelcast, the client code calls (1) api. Hz allows you to run the server as embedded, and in this case, accessing the api is a method call inside the JVM, which can be considered free.
In order for the logic in (2) to work, Hz relies on the hash from the byte array of the serialized key - that is, the key will be serialized anyway. This is the inevitable overhead for Hz.
Eviction strategies are implemented well, but for special cases, you can use your own. You don't have to worry about this part.
Storage (4) can be connected. Great. Interaction (5) for embedded can be considered instantaneous. Data exchange between nodes in a cluster (6) - yes, it is. This is a contribution to fault tolerance at the cost of speed. The Hz-feature Near-cache allows you to reduce the price - the data received from other cluster nodes will be cached.
What can be done in such conditions to increase speed?
For example, to avoid serialization of the key in (2), add another cache on top of Hazelcast for the hottest data. Sportmaster chose Caffeine for this purpose.
For twisting at level (6), two types of storage are proposed in Hz: IMap and ReplicatedMap.
It is worth saying how Hazelcast got into the Sportmaster technology stack.
In 2012, when we were working on the very first pilot of the future site, it was Hazelcast that turned out to be the first link that the search engine gave out. The acquaintance began “from the first time” - we were bribed by the fact that just two hours later, when we screwed Hz into the system, it worked. And worked well. Until the end of the day, we added some tests, we were glad. And this reserve of vivacity was enough to overcome those surprises that Hz threw up over time. Now the Sportmaster team has no reason to refuse Hazelcast.
But such arguments as “the first link in the search engine” and “HelloWorld quickly assembled” are, of course, an exception and a feature of the moment in which the choice took place. The real tests for the selected system begin with the release, and it is this stage that you should pay attention to when choosing any system, including the cache. Actually, in our case, we can say that we chose Hazelcast by chance, but then it turned out that we chose the right one.
For production, it is much more important: monitoring, handling failures on individual nodes, data replication, the cost of scaling. That is, you should pay attention to the tasks that will arise just in support of the system - when the load is ten times higher than planned, when you accidentally fill in something wrong and in the wrong place, when you need to roll out a new version of the code, replace the data and do it unnoticed for clients.
For all these requirements, Hazelcast certainly fits.
To be continued
But Hazelcast is not a panacea. In 2017, we chose Hazelcast for cache in the admin panel, simply based on a good impression from past experience. This played a key role in a very cruel joke, because of which we found ourselves in a difficult situation and "heroically" got out of it for 60 days. But more on that in the next section.
In the meantime... Happy New Code!
Source: habr.com