🥇Preparing DRP — don't forget to consider meteoroids

Even during a disaster, there’s always time for a cup of tea

DRP (disaster recovery plan) is something that ideally should never be needed. But if, for some reason, migrating beavers chew through the main fiber optic line or a junior admin drops the production database, you definitely want to be sure that you have a pre-established plan for what to do with this mess.

While customers panic and flood the support lines, the junior looks for cyanides, you wisely open the red envelope and start bringing everything back in line.

In this post, I want to share recommendations on how to write a DRP and what it should contain. We will also cover the following topics:

Let's learn to think like a villain.
We'll discuss the benefit of a cup of tea during an apocalypse.
We'll come up with a convenient structure for the DRP
We'll see how to test it properly

For which companies this can be beneficial

It’s very difficult to draw the line when the IT department starts needing these things. I would say you definitely need a DRP if:

Server, application downtime, or the loss of any database will lead to significant business losses overall.
You have a full-fledged IT department. I mean a department that acts as a complete unit of the company, with its own budget, and not just a few exhausted employees patching networks, cleaning viruses, and refilling printers.
You have a real budget for at least partial redundancy in case of an emergency.

When the IT department spends months begging for even a couple of HDDs for the old server for backups, you’re unlikely to successfully organize a full migration of the fallen service to backup resources. But even here, documentation will not be superfluous.

Documentation is important

Start with documentation. Let's say your service is built on a Perl script that was written three generations of admins ago, and nobody knows how it works. The accumulated technical debt and lack of documentation will inevitably shoot not just your knee, but other limbs as well; it's just a matter of time.

Once you have a good description of the service components, gather statistics on failures. They will almost certainly be typical. For example, your disk might occasionally fill up, leading to node failures until manual cleanup is performed. Or the client service becomes unavailable because someone forgot to renew the certificate again, and didn't know how to configure Let’s Encrypt.

Think like a saboteur

The most challenging part is predicting those failures that have never happened before but could potentially take down your service entirely. At this point, my colleagues and I often play the role of villains. Take a lot of coffee and some snacks, and lock yourselves in a meeting room. Just make sure you have also locked in the engineers who set up the target service or regularly work with it. Then, either on a board or on paper, start sketching out all the possible disasters that could occur with your service. It's not necessary to detail down to a specific janitor and cable pulling; it's enough to discuss scenarios like 'Breach of local network integrity'.

Typically, most standard failure situations fall into the following categories:

Network failure
OS service failure
Application failure
Hardware failure
Virtualization failure

Simply go through each category and see what applies to your service. For example, the Nginx daemon may crash and not restart—this relates to OS failures. A rare situation that puts your web application inoperable is software failure. During this stage, it’s important to work on diagnosing problems. How to distinguish a hanging interface on virtualization from a failed switch and a network failure, for instance. This is crucial for quickly identifying those responsible and starting to chase them down before the incident is resolved.

Once typical problems are noted, pour another cup of coffee and start considering the strangest scenarios when certain parameters begin to significantly exceed the norm. For example:

What will happen if the time on an active node shifts back by one minute compared to others in the cluster?
What if the time shifts forward, or what if it shifts by 10 years?
What will happen if a cluster node suddenly loses network connectivity during synchronization?
What happens if two nodes do not agree on leadership due to temporary isolation from each other across the network?

At this stage, a reverse approach is very helpful. Take the most eccentric team member with a vivid imagination and give them the task of orchestrating a disruption that will take down the service in a short time frame. If it's hard to diagnose, even better. You won't believe the strange and creative ideas engineers come up with when given the challenge to break something. And if you promise them a test environment for this — that's even better.

What is this DRP of yours?!

So, you've defined the threat model. You've considered the locals who cut fiber optic cables in search of copper, and the military radar that drops the radio relay line strictly on Fridays at 16:46. Now you need to figure out what to do about all this.

Your task is to create those red envelopes that will be opened in an emergency. Assume that when (not if!) everything goes wrong, the only person nearby will be the most inexperienced intern, who will be trembling with fear. Look at how emergency signage is implemented in medical offices. For example, what to do in case of anaphylactic shock. Medical staff know all the protocols by heart, but when someone nearby begins to die, they often helplessly grab at everything. For this, there is a clear instruction on the wall with steps like 'open the package of this' and 'administer intravenously this many units of the drug.'

In an emergency situation, it's hard to think! There must be simple instructions for instinctive processing.

A good DRP consists of several simple blocks:

Whom to notify at the start of the emergency. This is important to maximize the parallelization of the recovery process.
How to properly diagnose — perform a trace, check systemctl status servicename, and so on.
How much time can be spent at each stage. If you can’t fix it manually within the SLA time — the virtual machine is terminated and restored from yesterday's backup.
How to ensure that the emergency is resolved.

Remember that DRP starts when the service has completely failed and ends with restoring functionality, even at reduced efficiency. A simple loss of redundancy should not trigger the DRP. You might even want to include making a cup of tea in the DRP. Seriously. Statistics show that many incidents turn catastrophic due to staff panicking and trying to fix something, inadvertently taking down the only live node with data or completely crippling the cluster. Generally, five minutes for a cup of tea can give you a bit of time to calm down and analyze what’s happening.

Do not confuse DRP with the system passport! Don’t overload it with excessive data. Just provide a way to quickly and conveniently navigate through hyperlinks to the relevant sections of the documentation and read in an expanded format about the necessary parts of the service architecture. The DRP itself should only contain direct instructions on where and how to connect with specific commands for copy-pasting.

How to properly test

Ensure that any responsible employee is able to complete all steps. At the most critical moment, it may turn out that the engineer lacks access rights to the required system, the necessary account passwords are missing, or they have no idea what "Connect to the service management console through the proxy at the headquarters" means. Each point should be extremely simple.

Incorrect — "Go to virtualization and reboot the dead node."
Correct — "Connect to virt.example.com via the web interface, in the node section perform a reboot of the node that is causing the error."

Avoid ambiguities. Remember about the frightened intern.

Be sure to test the DRP. This is not just a plan for the sake of formality — it is what will allow you and your clients to quickly emerge from a critical situation. Ideally, this should be done several times:

One expert and several interns work on a test setup that closely mimics the real service. The expert breaks the service in various ways and gives the interns an opportunity to restore it according to the DRP. All problems, uncertainties in the documentation, and errors are recorded. After training the interns, the DRP is supplemented and simplified in unclear areas.
Testing on a real service. In reality, you can never create a perfect copy of an actual service. Therefore, a few times a year, it's necessary to systematically shut down part of the servers, disrupt connections, and create other disruptions from the threat list to evaluate recovery procedures. A planned outage for 10 minutes in the middle of the night is better than a sudden failure for several hours during peak load with data loss.
Actual incident resolution. Yes, this is also part of testing. If an incident occurs that was not on the threat list, it is necessary to update and refine the DRP based on the results of its investigation.

Key points

If something can go wrong, it will not only go wrong but do so in the most catastrophic manner possible.
Ensure that you have resources for emergency load balancing.
Make sure you have backups that are created automatically and regularly checked for consistency.
Consider typical threat scenarios.
Give engineers the opportunity to devise unconventional ways to disrupt the service.
DRP should be a simple and straightforward instruction. All complicated diagnostics should come only after the service is restored for clients. Even if it’s on backup capacities.
Include key phone numbers and contacts in the DRP.
Regularly test employees on their understanding of the DRP.
Conduct planned outages in production. Staging environments cannot replace everything.