In the summer, both consumer activity and the pace of changes in web project infrastructure traditionally decrease, as Captain Obvious tells us. This is simply because even IT professionals take vacations sometimes. And so do CTOs. This makes it tougher for those who remain on duty, but thatâs not the topic right now: perhaps thatâs exactly why summer is the best time to thoughtfully consider the existing backup strategy and develop a plan for its improvement. For this, the experience of Yegor Andreev from , which he shared at the conference .
When building backup sites, there are several traps one can fall into during the backup process. Falling into these traps is completely unacceptable. Perfectionism and⊠laziness ruin everything, just like in many other areas. We try to do everything perfectly, but perfection is not necessary! We only need to do certain things, and we must do them correctly and see them through to the end so that they function well.
Failover is not just some fun, trivial thing 'to have'; it is something that should accomplish one specific goal â to minimize downtime so that the service and the company lose less money. In all backup methods, I suggest thinking in the following context: where is the money?
The first trap: when we build large, reliable systems and engage in backups â we reduce the number of failures. This is a dreadful misconception. When we engage in backups, the number of failures is likely to increase. If we do everything right, we will cumulatively reduce downtime. There will be more failures, but they will occur with lower costs. After all, what is backup? â itâs a complication of the system. Any complication is detrimental: we have more screws, more gears, in short, more elements â and, consequently, a higher chance of failure. And they will indeed break. And they will break more often. A simple example: letâs say we have a certain site running on PHP and MySQL. And it urgently needs to be backed up.
Well, we're taking on a second platform and building an identical system... The complexity doubles â now we have two entities. Additionally, we layer on a certain logic for transferring data from one platform to the other â meaning data replication, static file copying, and so on. The fact is, replication logic is usually very complex, so the overall system complexity may not just be 2, but 3, 5, or even 10 times greater.
The second trap: when we construct truly large complex systems, we fantasize about what we ultimately want to achieve. VoilĂ : we want to have a super-reliable system that operates with zero downtime, switches in half a second (or better yet, instantly), and we start bringing our dreams to life. However, there's a catch: the shorter the desired switching time, the more complicated the system logic becomes. The more complex this logic needs to be, the more frequently the system will break. You can find yourself in a very unpleasant situation: we are trying our hardest to reduce downtime, but in reality, we are complicating everything, and when something goes wrong, the downtime ends up being greater. Often, you catch yourself thinking: it would be better not to have a backup. It would be better if it worked on its own with a clear downtime.
How can we combat this? We need to stop lying to ourselves, stop flattering ourselves that we are going to build a spaceship here, and realistically understand how long the project can be down. For this maximum time, we will choose the methods we use to enhance the reliability of our system.
It's time for 'stories from life'... from life, of course.
Example number one
Imagine a business card website for the pipe manufacturing plant No. 1 in City N. It boldly states â PIPE MANUFACTURING PLANT NO. 1. Just below â a slogan: "Our pipes are the roundest pipes in N." And at the bottom, the phone number of the CEO and his name. We understand the need for reservation â it's a very important thing! We start figuring out what it consists of. Html static â that is, a couple of pictures, where the CEO is, actually, at a table in a bathhouse with his partner discussing some deal. We begin to think about downtime. It comes to mind: it needs to lie there for five minutes, no more. And then the question arises: how many sales came from our website at all? How many? What do you mean 'zero'? It indeed means: because the CEO made all four deals last year at that same table, with the same people he goes to the bathhouse with. And we realize that even if the website lies idle for a day â nothing terrible will happen.
Based on the inputs, there is a day to set this up. We begin to think about the reservation scheme. We choose the most ideal reservation scheme for this example: we do not use any reservation. This whole thing can be set up by any admin in half an hour with breaks. Set up the web server, place the files â that's it. It will work. There is nothing to monitor, nothing to pay special attention to. Thus, the conclusion from the first example is quite obvious: services that do not need reservation â do not need reservation.
Example number two
Company Blog: specially trained people write news there, like our participation in a certain exhibition, or the release of a new product, and so on. Let's say it's standard PHP with WordPress, a small database, and a bit of static content. Of course, it comes to mind that we can't let it just sit there â 'no more than five minutes!' and all that. But let's think further. What does this blog do? People come from Yandex, Google for some queries, organically. Great. But how are sales related at all? The realization: not much. Paid traffic goes to the main site, which is on another server. We begin to think about the backup scheme for the blog. Ideally, it should be up in a couple of hours, and we should prepare for that. It would be wise to take a server in another data center, set up the environment there, meaning the web server, PHP, WordPress, MySQL, and leave it in a dormant state. The moment we realize something is broken, we need to do two things â roll out a 50MB MySQL dump, which will zip through in a minute, and roll out some images from the backup. That's not a huge amount either. Thus, this whole setup comes up in half an hour. No replications, or heaven forbid, automatic failover. Conclusion: what we can quickly restore from backup doesn't need reserving.
Example number three, a bit more complicated
Online Store. PHP with open heart slightly modified, MySQL with a solid database. Quite a lot of static content (after all, an online store has beautiful HD images and all that), Redis for sessions, and Elasticsearch for search. We start thinking about downtime. And here, of course, it is obvious that a day without the online store is simply not acceptable. The longer it's down, the more money we lose. We need to speed up. But by how much? I suppose if we are down for an hour, no one will go crazy. Yes, we will lose something, but if we start to push harder â it will only get worse. Let's define the downtime scheme acceptable in an hour.
How can all of this be reserved? A machine is needed in any case: one hour is quite limited. MySQL: here we need replication, live replication, because in an hour, 100 GB may not fit into the dump. Static files, images: again, 500 GB may not load in an hour. Therefore, itâs better to start copying the images right away. Redis: this is where it gets interesting. Sessions are stored in Redis â we can't just take and erase it. Because that wouldnât be good: all users would get logged out, carts would be emptied, and so on. People would have to re-enter their login and password, and many might drop off and not complete their purchase. Again, conversion would fall. On the other hand, having Redis exactly with the last logged-in users probably isnât necessary either. A good compromise would be to take Redis and restore it from a backup, whether from yesterday or, if you have hourly backups, from an hour ago. Fortunately, restoring it from a backup is just copying one file. And the most interesting story is Elasticsearch. Who has ever set up MySQL replication? Who has ever set up Elasticsearch replication? And who has had it working properly afterward? What Iâm getting at is that we see some entity in our system. It seems useful â but itâs complex.
It's complicated in the sense that our engineering colleagues have no experience working with it. Either they have had negative experiences, or we understand that this technology is still relatively new with its nuances or lack of maturity. We think... Damn, Elasticsearch is also hefty; restoring it from a backup takes time, what should we do? We understand that Elasticsearch is being used for search in our case. But how does our online store sell? We go to the marketers and ask where the customers are coming from. They reply, "90% come directly from Yandex.Market to the product page." They either buy or they don't. Therefore, search is only needed for 10% of users. And maintaining Elasticsearch replication, especially between different data centers in various zones, indeed has many intricacies. Whatâs the solution? We take Elasticsearch to a reserved environment and donât do anything with it. If the matter drags on, we might bring it up later, but thatâs uncertain. In fact, the takeaway is pretty much the same: we donât reserve services that donât have a financial impact.
Example number four, even more complex.
Integrator: flower sales, taxi calls, product sales, basically anything. A serious thing that operates 24/7 for a large number of users. With a fully interesting stack, where there are intriguing databases, solutions, high loads, and most importantly, it can't be down for more than 5 minutes. Not only because people wonât buy, but because theyâll see that this thing isnât working, get upset and may never come back.
Okay. Five minutes. What are we going to do about this? In this case, we seriously build a genuine backup environment with complete replication of everything and possibly even automate the switch to this environment as much as we can. And in addition to this, one important thing should not be forgotten: to actually write the switching protocol. The protocol, even if everything is automated, can be very simple. Like "run this particular Ansible script," "click this checkbox in Route 53," and so on â but it needs to be a precise list of actions.
Everything seems clear. Switching replication is a trivial task, or it may switch itself. Rewriting the domain name in DNS is from the same category. The trouble is, when a project like this fails, panic sets in, and even the toughest, most seasoned admins can succumb to it. Without a clear instruction like "open the terminal, go here, the address of our server is still this one," it's hard to stick to a five-minute timeframe for recovery. Furthermore, when we use this regulation, it's easy to fix certain changes in the infrastructure, for instance â and accordingly adjust the guidelines.
Well, if the backup system is very complex and at some point we made a mistake, we might crash our backup site as well and turn the data into a pumpkin on both sites â that would be truly unfortunate.
Example number five, full hardcore.
An international service with hundreds of millions of users worldwide. All time zones imaginable, high load to the max, absolutely cannot go down. One minute â and it will be sad. What to do? Back up everything, again, fully. We did everything mentioned in the previous example and just a bit more. An ideal world, and our infrastructure â it conforms to all DevOps principles of IaaC. In other words, everything is in Git, just press a button.
Whatâs missing? One thing â drills. We canât do without them. It seems like everything is perfect, like we have everything under control. We press a button, and everything happens. Even if thatâs the case â and we understand that it isnât so straightforward â our system interacts with some other systems. For example, it involves DNS from Route 53, S3 storage, integration with some APIs. We wonât be able to anticipate everything in this theoretical experiment. And until we really pull the switch â we wonât know if it works or not.
Thatâs probably all. Donât be lazy and donât overdo it. And may uptime be with you!
Source: habr.com
