Hello everyone, my name is Igor Tyukachev and I am a Business Continuity Consultant. In today's post we are going to have a long and tedious discussion of common truths, I want to share my experience and talk about the main mistakes that companies make when developing a business continuity plan.
1. RTO and RPO at random
The most important mistake I've come across is that the recovery time (RTO) is taken from the ceiling. Well, as if from the ceiling - for example, there are some figures from two years ago from the SLA, which someone brought from a previous job. Why do they do it? After all, for all methods, you must first analyze the consequences for business processes, and based on this analysis, calculate the target recovery time and allowable data loss. But doing such an analysis is sometimes time consuming, sometimes costly, sometimes it is not very clear how—underline the necessary. And the first thing that comes to mind is: “We are all adults and we understand how business works. Let's not waste time and money! Let's take the plus or minus, as it should be. Out of my head, using proletarian ingenuity! Let RTO be equal to two hours.
What does this lead to? When you come to management for money for activities to ensure the required RTO / RPO with some numbers, it always requires justification. If there is no justification, then the question arises: where did you get it from? And there is nothing to answer. As a result, the credibility of your work is lost.
Also, sometimes those two hours of recovery are worth a million dollars. And the rationale for the duration of the RTO is a matter of money, and very large ones.
And finally, when you take your BCP and/or DR plan to the performers (who will actually be running around and waving their arms at the time of the accident), they will ask a similar question: where did these two hours come from? And if you cannot clearly explain it, then they will not have confidence in you or in your document.
It turns out a piece of paper for the sake of a piece of paper, an unsubscribe. By the way, some do it deliberately, just to meet the requirements of the regulator.
Well do you understand
2. The cure for everything
Some believe that the BCP is designed to protect all business processes from any threat. Recently, to the question “What do we want to protect ourselves from?” I heard the answer: "From everything and more."
But the fact is that the plan is designed to protect only specific key business processes of the company from specific threats. Therefore, before developing a plan, it is necessary to assess the occurrence of risks and analyze their consequences for the business. Risk assessment is needed in order to understand what kind of threats the company is afraid of. In the event of a building collapse, there will be one continuity plan, in the event of sanctions pressure - another, in case of flooding - a third. Even two identical venues in different cities can have significantly different plans.
It is impossible to protect the entire company with one BCP, especially a large one. For example, the huge X5 Retail Group began to deal with ensuring continuity with two key business processes (we wrote about this ). And to enclose the whole company with one plan is simply unrealistic, this is from the category of “collective responsibility”, when everyone is responsible and no one is responsible.
In the ISO 22301 standard there is a concept of policy, with which, in fact, the process of continuity in the company begins. It describes what we will protect and from what. If people come running and ask to add this and that, for example:
- And let's add the risk of being hacked to BCP?
Or
- Recently, during the rain, the last floor was flooded - let's add a scenario, what to do in case of flooding?
Then immediately refer them to this policy and say that we protect specific company assets and only from specific, pre-agreed threats, because they are now in priority.
And even if the proposals for changes are really appropriate, then offer to take them into account in the next version of the policy. Because protecting a company is a lot of money. So all changes to the BCP plan must go through the budget committee and planning. We recommend reviewing the company's business continuity policy once a year or immediately after significant changes in the company's structure or external conditions (readers will forgive me for such words).
3. Fantasy and reality
It often happens that when drawing up a BCP plan, the authors describe some ideal picture of the world. For example, “we don’t have a second data center, but we will write a plan as if we have one.” Or the business does not yet have some part of the infrastructure, but employees will put it into the plan anyway in the hope that it will appear in the future. And then the company will pull reality onto the plan: build a second data center, describe other changes.
On the left is the BCP-compliant infrastructure, on the right is the real infrastructure
All this is a mistake. Writing a BCP plan means spending money. If you write a plan that won't work right now, you'll pay for very expensive paper. It is impossible to recover from it, it is impossible to test it. It's work for the sake of work.
You can write a plan quite quickly, but building a backup infrastructure, spending money on all protection solutions is long and expensive. This may take more than one year. And it may turn out that you already have a plan, and the infrastructure for it will appear in two years. Why is such a plan needed? What will he protect you from?
Another from the category of fantasy, when the working team for the development of BCP begins to think out for the experts what they should do and in what time. It turns out from the category: “having seen a bear in the taiga, it is necessary to turn around in the opposite direction from the bear and run at a speed exceeding the speed of the bear. During the winter months, you need to cover your tracks.”
4. Tops and roots
The fourth most important mistake is that the plan is either too superficial or too detailed. We need a golden mean. The plan should not be too detailed, for idiots, but should not be too general, so that something like this does not turn out:
Easy in general
5. Caesar - Caesarean, locksmith - locksmith
The next error stems from the previous one: one plan cannot contain all the actions for all levels of management. BCP plans are usually developed for large companies with large financial flows (by the way, according to our , an average of 48% of large Russian companies faced emergency situations that entailed significant financial losses) and a multi-level management system. For such companies, you should not try to fit everything into one document. If the company is large and structured, then the plan should have three separate levels:
- strategic level - for top management;
- tactical level - for middle managers;
- and the operational level for direct implementers in the field.
For example, if we are talking about the restoration of a fallen infrastructure, then at the strategic level a decision is made to activate the recovery plan, at the tactical level, process procedures can be described, and at the operational level, instructions for commissioning specific pieces of equipment.
BCP no budget
Everyone sees his own area of responsibility and communication with other employees. At the moment of the accident, everyone opens the plan, quickly finds their part and follows it. Ideally, you need to remember by heart which pages to open, because it happens that the score goes on for minutes.
6. Role play
Another mistake when drawing up a BCP plan: you do not need to write down specific names, email addresses and other contact details in the plan. In the text of the document itself, only impersonal roles should be indicated, and the names of those responsible for specific tasks should be assigned to these roles and their contacts should be listed in the appendix to the plan.
Why?
Today, most people change jobs every two or three years. And if you write down all those responsible and their contacts in the text of the plan, then it will have to be constantly changed. And in large companies, and even more so state-owned, every change in any document requires a bunch of approvals.
Not to mention the fact that if an emergency occurs, and you have to frantically scroll through the plan and look for the right contact, then precious time will be lost.
Life hack: when you change an application, you often don't even need to approve it. Another tip: You can use plan update automation systems.
7. Lack of versioning
Usually, a version 1.0 plan is created, and then all changes are made without editing mode, and without changing the file name. At the same time, it is often not clear what has changed compared to the previous version. In the absence of versioning, the plan lives its own life, which is not tracked in any way. The second page of any BCP plan should include the version, the author of the changes, and a list of the changes themselves.
No one can figure it out
8. Who to ask?
Often, companies do not have a BCP person responsible for the plan and there is no separate unit that is responsible for business continuity. This honorable duty is assigned to the CIO, his deputy, or according to the principle "you are engaged in information security, here's BCP in addition to you." As a result, the plan is developed, coordinated and approved by everything, from top to bottom.
And who is responsible for keeping the plan, updating, and revising the information in it? This may or may not be assigned. It is wasteful to hire an individual employee for this, but it is possible, of course, to load one of the existing employees with an additional duty, because everyone is now striving for efficiency: “Let's hang a lantern on him so that he can mow at night,” but is it necessary?
We are looking for those responsible for BCP two years after its creation
Therefore, it often happens like this: a plan was developed and put on the back burner covered with dust. No one tests it, no one maintains its relevance. The most frequent phrase I hear when I come to a customer is: “There is a plan, but it was developed a long time ago, whether it was tested is unknown, there is a suspicion that it does not work.”
9. Too much water
There are plans in which the introduction of five pages, including a description of the prerequisites and gratitude to all project participants, with information about what the company does. While you are scrolling through the pages to the tenth, where there is useful information, your data center is already flooded.
When you try to read up to the moment, what to do when the data center is flooded
Take out all corporate "water" in a separate document. The plan itself should be extremely specific: the person responsible for this task does this, and so on.
10. At whose expense is the banquet?
Often the creators of the plan do not have support from the top management of the company. But there is support from middle management who do not manage or do not have the necessary budget and resources to organize business continuity. For example, the IT department creates its BCP plan within its budget, but the CIO does not see the whole picture in the company. My favorite example is video conferencing. When the general doesn't have video conferencing, who will he disembowel? CIO who "did not provide". Therefore, from the point of view of the CIO, what is the most important thing in the company? What he is constantly “loved” for: videoconferencing, which immediately turns into a business-critical system. And from the point of view of business - well, there is no videoconferencing, just think, we'll talk on the phone, as under Brezhnev ...
In addition, the IT department usually thinks that its main task in the event of a disaster is to restore the operation of corporate IT systems. But sometimes you don't have to! If there is a business process in the form of printing pieces of paper on a terribly expensive printer, then you should not buy a second such printer as a spare and put it next to it in case of a breakdown. It may be enough to temporarily color the pieces of paper by hand.
If we build continuous protection inside IT, we must enlist the support of senior management and business representatives. Otherwise, by pupating inside the IT department, you can solve a certain range of problems, but not all the necessary ones.
This is what it looks like when only the IT department has DR plans
10. No testing
If there is a plan, it must be tested. For those who are not familiar with the standards, this is completely non-obvious. For example, you have “emergency exit” signs hanging everywhere. But tell me where you have a fire bucket, a hook, a shovel? Where is the fire hydrant located? Where should the fire extinguisher be located? And everyone should know this. It does not seem logical to us at all when entering the office to look for a fire extinguisher with our eyes.
Perhaps the need to test the plan should be mentioned in itself, but this is a controversial decision. In any case, the plan can be considered working only when it has been tested at least once. As mentioned above, I often hear: “There is a plan, all the infrastructure has been prepared, but it’s not a fact that everything will work out as written in the plan. Because they haven't tested it. Never".
In conclusion
Some companies can analyze their history in order to understand what troubles and how likely they are to occur. Research and experience show that we cannot protect ourselves from everything. Shit, sooner or later, happens to any company. Another thing is how prepared you are for this or a similar situation and whether you can restore your business in time.
Some people think that continuity is about how to eliminate all kinds of risks so that they do not materialize. No, the point is that the risks are realized, and we will be ready for this. Soldiers train not to think in battle, but to act. It's the same with the BCP plan: it will allow you to get your business back on track as quickly as possible.
The only equipment that does not require BCP
Igor Tyukachev,
Business Continuity Consultant
Center for Designing Computer Complexes
Jet Infosystems
Source: habr.com