{"id":36393,"date":"2019-10-31T22:11:24","date_gmt":"2019-10-31T19:11:24","guid":{"rendered":"https:\/\/prohoster.info\/blog\/failover-nas-gubit-perfektsionizm-i-len\/"},"modified":"2019-10-31T22:11:24","modified_gmt":"2019-10-31T19:11:24","slug":"failover-nas-gubit-perfektsionizm-i-len","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/failover-nas-gubit-perfektsionizm-i-len","title":{"rendered":"Failover: perfectionism and\u2026 laziness ruin us","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

In the summer, both consumer activity and the pace of changes in web project infrastructure traditionally decrease, as Captain Obvious tells us. This is simply because even IT professionals take vacations sometimes. And so do CTOs. This makes it tougher for those who remain on duty, but that\u2019s not the topic right now: perhaps that\u2019s exactly why summer is the best time to thoughtfully consider the existing backup strategy and develop a plan for its improvement. For this, the experience of Yegor Andreev from AdminDivision<\/a><\/noindex>, which he shared at the conference Uptime day<\/a><\/noindex>.<\/i><\/p>\n

When building backup sites, there are several traps one can fall into during the backup process. Falling into these traps is completely unacceptable. Perfectionism and\u2026 laziness ruin everything, just like in many other areas. We try to do everything perfectly, but perfection is not necessary! We only need to do certain things, and we must do them correctly and see them through to the end so that they function well. <\/p>\n

Failover is not just some fun, trivial thing 'to have'; it is something that should accomplish one specific goal \u2014 to minimize downtime so that the service and the company lose less money. In all backup methods, I suggest thinking in the following context: where is the money?<\/p>\n

\"Failover:
\n<\/a><\/noindex>
\nThe first trap<\/b>: when we build large, reliable systems and engage in backups \u2014 we reduce the number of failures. This is a dreadful misconception. When we engage in backups, the number of failures is likely to increase. If we do everything right, we will cumulatively reduce downtime. There will be more failures, but they will occur with lower costs. After all, what is backup? \u2014 it\u2019s a complication of the system. Any complication is detrimental: we have more screws, more gears, in short, more elements \u2014 and, consequently, a higher chance of failure. And they will indeed break. And they will break more often. A simple example: let\u2019s say we have a certain site running on PHP and MySQL. And it urgently needs to be backed up. <\/p>\n

Well, we're taking on a second platform and building an identical system... The complexity doubles \u2014 now we have two entities. Additionally, we layer on a certain logic for transferring data from one platform to the other \u2014 meaning data replication, static file copying, and so on. The fact is, replication logic is usually very complex, so the overall system complexity may not just be 2, but 3, 5, or even 10 times greater. <\/p>\n

The second trap<\/b>: when we construct truly large complex systems, we fantasize about what we ultimately want to achieve. Voil\u00e0: we want to have a super-reliable system that operates with zero downtime, switches in half a second (or better yet, instantly), and we start bringing our dreams to life. However, there's a catch: the shorter the desired switching time, the more complicated the system logic becomes. The more complex this logic needs to be, the more frequently the system will break. You can find yourself in a very unpleasant situation: we are trying our hardest to reduce downtime, but in reality, we are complicating everything, and when something goes wrong, the downtime ends up being greater. Often, you catch yourself thinking: it would be better not to have a backup. It would be better if it worked on its own with a clear downtime. <\/p>\n

How can we combat this? We need to stop lying to ourselves, stop flattering ourselves that we are going to build a spaceship here, and realistically understand how long the project can be down. For this maximum time, we will choose the methods we use to enhance the reliability of our system. <\/p>\n

\"Failover:
\n
\nIt's time for 'stories from life'... from life, of course. <\/p>\n

Example number one<\/h4>\n

\nImagine a business card website for the pipe manufacturing plant No. 1 in City N. It boldly states \u2014 PIPE MANUFACTURING PLANT NO. 1. Just below \u2014 a slogan: \"Our pipes are the roundest pipes in N.\" And at the bottom, the phone number of the CEO and his name. We understand the need for reservation \u2014 it's a very important thing! We start figuring out what it consists of. Html static \u2014 that is, a couple of pictures, where the CEO is, actually, at a table in a bathhouse with his partner discussing some deal. We begin to think about downtime. It comes to mind: it needs to lie there for five minutes, no more. And then the question arises: how many sales came from our website at all? How many? What do you mean 'zero'? It indeed means: because the CEO made all four deals last year at that same table, with the same people he goes to the bathhouse with. And we realize that even if the website lies idle for a day \u2014 nothing terrible will happen. <\/p>\n

Based on the inputs, there is a day to set this up. We begin to think about the reservation scheme. We choose the most ideal reservation scheme for this example: we do not use any reservation. This whole thing can be set up by any admin in half an hour with breaks. Set up the web server, place the files \u2014 that's it. It will work. There is nothing to monitor, nothing to pay special attention to. Thus, the conclusion from the first example is quite obvious: services that do not need reservation \u2014 do not need reservation.<\/p>\n

\"Failover:
\n<\/p>\n

Example number two<\/h4>\n

\nCompany Blog: trained professionals write news there, such as our participation in a certain exhibition or the launch of a new product, and so on. Let's say this is standard PHP with WordPress, a small database, and a bit of static content. Of course, it comes to mind that we must not let it sit idle \u2014 \"no more than five minutes!\", all that. But let\u2019s think further. What does this blog do? It attracts visitors from Yandex and Google through various queries, through organic traffic. Great. But are sales actually connected to it? The realization: not really. Paid traffic goes to the main site, which is hosted on a different server. We start considering a backup plan for the blog. Ideally, it should be up and running within a couple of hours, and it would be wise to prepare for that. It would make sense to take a server from another data center, set up the environment \u2013 that is, web server, PHP, WordPress, MySQL \u2013 and keep it on standby. The moment we realize something has broken, we need to do two things \u2014 restore the MySQL dump, which is around 50 megabytes and will fly through in a minute, and restore some images from the backup. That shouldn\u2019t take too long. Thus, in half an hour this whole thing can be up and running. No replication, or heaven forbid, automatic failover. Conclusion: what we can quickly restore from backup \u2014 doesn\u2019t require reserving. <\/p>\n

\"Failover:
\n<\/p>\n

Example number three, a bit more complicated<\/h4>\n

\nOnline Store. PHP with open heart slightly modified, MySQL with a solid database. Quite a lot of static content (after all, an online store has beautiful HD images and all that), Redis for sessions, and Elasticsearch for search. We start thinking about downtime. And here, of course, it is obvious that a day without the online store is simply not acceptable. The longer it's down, the more money we lose. We need to speed up. But by how much? I suppose if we are down for an hour, no one will go crazy. Yes, we will lose something, but if we start to push harder \u2014 it will only get worse. Let's define the downtime scheme acceptable in an hour.<\/p>\n

How can all of this be reserved? A machine is needed in any case: one hour is quite limited. MySQL: here we need replication, live replication, because in an hour, 100 GB may not fit into the dump. Static files, images: again, 500 GB may not load in an hour. Therefore, it\u2019s better to start copying the images right away. Redis: this is where it gets interesting. Sessions are stored in Redis \u2014 we can't just take and erase it. Because that wouldn\u2019t be good: all users would get logged out, carts would be emptied, and so on. People would have to re-enter their login and password, and many might drop off and not complete their purchase. Again, conversion would fall. On the other hand, having Redis exactly with the last logged-in users probably isn\u2019t necessary either. A good compromise would be to take Redis and restore it from a backup, whether from yesterday or, if you have hourly backups, from an hour ago. Fortunately, restoring it from a backup is just copying one file. And the most interesting story is Elasticsearch. Who has ever set up MySQL replication? Who has ever set up Elasticsearch replication? And who has had it working properly afterward? What I\u2019m getting at is that we see some entity in our system. It seems useful \u2014 but it\u2019s complex.
\nIt's complicated in the sense that our engineering colleagues lack experience with it. Either they have had negative experiences or we realize that it is still quite a new technology with nuances or instability. We think... Man, Elastic is also large, restoring it from backup takes time too; what should we do? We understand that Elastic is used for searching in our case. But how does our online store sell? We go to the marketers and ask where the people are coming from. They answer: \"90% comes directly from Yandex.Market to the product card.\" And they either buy or they don't. Therefore, only 10% of users need the search. And maintaining the replication of Elastic, especially between different data centers in different regions, has many nuances. What is the solution? We take Elastic on a reserved site and do nothing with it. If the matter drags on, we might bring it up later, but that's not certain. The conclusion is pretty much the same: we do not reserve services that do not affect revenue. To keep the scheme simpler.<\/p>\n

\"Failover:
\n<\/p>\n

Example number four, even more complex.<\/h4>\n

\nIntegrator: flower sales, taxi calls, product sales, basically anything. A serious thing that operates 24\/7 for a large number of users. With a fully interesting stack, where there are intriguing databases, solutions, high loads, and most importantly, it can't be down for more than 5 minutes. Not only because people won\u2019t buy, but because they\u2019ll see that this thing isn\u2019t working, get upset and may never come back. <\/p>\n

Okay. Five minutes. What are we going to do about this? In this case, we seriously build a genuine backup environment with complete replication of everything and possibly even automate the switch to this environment as much as we can. And in addition to this, one important thing should not be forgotten: to actually write the switching protocol. The protocol, even if everything is automated, can be very simple. Like \"run this particular Ansible script,\" \"click this checkbox in Route 53,\" and so on \u2014 but it needs to be a precise list of actions. <\/p>\n

Everything seems clear. Switching replication is a trivial task, or it may switch itself. Rewriting the domain name in DNS is from the same category. The trouble is, when a project like this fails, panic sets in, and even the toughest, most seasoned admins can succumb to it. Without a clear instruction like \"open the terminal, go here, the address of our server is still this one,\" it's hard to stick to a five-minute timeframe for recovery. Furthermore, when we use this regulation, it's easy to fix certain changes in the infrastructure, for instance \u2014 and accordingly adjust the guidelines.
\nWell, if the backup system is very complex and at some point we made a mistake, we might crash our backup site as well and turn the data into a pumpkin on both sites \u2014 that would be truly unfortunate. <\/p>\n

\"Failover:
\n<\/p>\n

Example number five, full hardcore.<\/h4>\n

\nAn international service with hundreds of millions of users worldwide. All time zones imaginable, high load to the max, absolutely cannot go down. One minute \u2014 and it will be sad. What to do? Back up everything, again, fully. We did everything mentioned in the previous example and just a bit more. An ideal world, and our infrastructure \u2014 it conforms to all DevOps principles of IaaC. In other words, everything is in Git, just press a button. <\/p>\n

What\u2019s missing? One thing \u2014 drills. We can\u2019t do without them. It seems like everything is perfect, like we have everything under control. We press a button, and everything happens. Even if that\u2019s the case \u2014 and we understand that it isn\u2019t so straightforward \u2014 our system interacts with some other systems. For example, it involves DNS from Route 53, S3 storage, integration with some APIs. We won\u2019t be able to anticipate everything in this theoretical experiment. And until we really pull the switch \u2014 we won\u2019t know if it works or not. <\/p>\n

\"Failover:
\n
\nThat\u2019s probably all. Don\u2019t be lazy and don\u2019t overdo it. And may uptime be with you!
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041b\u0435\u0442\u043e\u043c \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u0441\u043d\u0438\u0436\u0430\u0435\u0442\u0441\u044f \u0438 \u043f\u043e\u043a\u0443\u043f\u0430\u0442\u0435\u043b\u044c\u0441\u043a\u0430\u044f \u0430\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u044c, \u0438 \u0438\u043d\u0442\u0435\u043d\u0441\u0438\u0432\u043d\u043e\u0441\u0442\u044c \u0438\u0437\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0432\u0435\u0431-\u043f\u0440\u043e\u0435\u043a\u0442\u043e\u0432, \u0433\u043e\u0432\u043e\u0440\u0438\u0442 \u043d\u0430\u043c \u041a\u0430\u043f\u0438\u0442\u0430\u043d \u041e\u0447\u0435\u0432\u0438\u0434\u043d\u043e\u0441\u0442\u044c. \u041f\u0440\u043e\u0441\u0442\u043e \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u0434\u0430\u0436\u0435 \u0430\u0439\u0442\u0438\u0448\u043d\u0438\u043a\u0438, \u0441\u043b\u0443\u0447\u0430\u0435\u0442\u0441\u044f, \u0445\u043e\u0434\u044f\u0442 \u0432 \u043e\u0442\u043f\u0443\u0441\u043a. \u0418 C\u0422\u041e \u0442\u043e\u0436\u0435. \u0422\u0435\u043c \u0442\u044f\u0436\u0435\u043b\u0435\u0435 \u0442\u0435\u043c, \u043a\u0442\u043e \u043e\u0441\u0442\u0430\u0451\u0442\u0441\u044f \u043d\u0430 \u043f\u043e\u0441\u0442\u0443, \u043d\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u043d\u0435 \u043e\u0431 \u044d\u0442\u043e\u043c: \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e, \u0438\u043c\u0435\u043d\u043d\u043e \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043b\u0435\u0442\u043e \u2014 \u043b\u0443\u0447\u0448\u0438\u0439 \u043f\u0435\u0440\u0438\u043e\u0434 \u0434\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u043d\u0435 \u0442\u043e\u0440\u043e\u043f\u044f\u0441\u044c \u043e\u0431\u0434\u0443\u043c\u0430\u0442\u044c \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0441\u0445\u0435\u043c\u0443 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":27235,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36393","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Failover: perfectionism and laziness are our downfall | ProHoster","description":"In summer, both consumer activity and the intensity of infrastructure changes for web projects traditionally decline, says Captain Obvious. Simply because even IT specialists, sometimes, go on vacation.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/failover-nas-gubit-perfektsionizm-i-len","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Failover: \u043d\u0430\u0441 \u0433\u0443\u0431\u0438\u0442 \u043f\u0435\u0440\u0444\u0435\u043a\u0446\u0438\u043e\u043d\u0438\u0437\u043c \u0438\u2026 \u043b\u0435\u043d\u044c | ProHoster","og:description":"\u041b\u0435\u0442\u043e\u043c \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u0441\u043d\u0438\u0436\u0430\u0435\u0442\u0441\u044f \u0438 \u043f\u043e\u043a\u0443\u043f\u0430\u0442\u0435\u043b\u044c\u0441\u043a\u0430\u044f \u0430\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u044c, \u0438 \u0438\u043d\u0442\u0435\u043d\u0441\u0438\u0432\u043d\u043e\u0441\u0442\u044c \u0438\u0437\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0432\u0435\u0431-\u043f\u0440\u043e\u0435\u043a\u0442\u043e\u0432, \u0433\u043e\u0432\u043e\u0440\u0438\u0442 \u043d\u0430\u043c \u041a\u0430\u043f\u0438\u0442\u0430\u043d \u041e\u0447\u0435\u0432\u0438\u0434\u043d\u043e\u0441\u0442\u044c. \u041f\u0440\u043e\u0441\u0442\u043e \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u0434\u0430\u0436\u0435 \u0430\u0439\u0442\u0438\u0448\u043d\u0438\u043a\u0438, \u0441\u043b\u0443\u0447\u0430\u0435\u0442\u0441\u044f, \u0445\u043e\u0434\u044f\u0442 \u0432 \u043e\u0442\u043f\u0443\u0441\u043a.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/failover-nas-gubit-perfektsionizm-i-len","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:11:24+00:00","article:modified_time":"2019-10-31T19:11:24+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36393","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 03:07:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:45:36","updated":"2026-01-22 03:07:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36393","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=36393"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36393\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/27235"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=36393"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=36393"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=36393"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}