Hello readers of our blog! In part, we are already familiar - my English-language posts appeared here in the translation of my dear colleague . This time I decided to address the Russian-speaking audience directly.
For my debut, I wanted to find a topic that would be interesting to the widest possible audience and require detailed consideration. Daniel Defoe argued that death and taxes await any person. For my part, I can say that any support engineer is waiting for questions about the policies for storing recovery points (or, in simpler terms, retention). I started explaining how retention works 4 years ago as a Level XNUMX Junior Engineer, and I continue to explain now as the leader of the Spanish and Italian speaking team. I am sure that my colleagues from the second and even the third level of support also regularly answer the same questions.
In this light, I wanted to write a final, as detailed post as possible, to which Russian-speaking users could return again and again as a reference. The moment is right - the recently released tenth anniversary version added new features to the basic functionality that has not changed for years. My post is focused primarily on this version - although most of what is written is true for previous versions, you simply will not find some of the described functionality there. Finally, looking a bit into the future, I will say that some changes are expected in the next version, but we will tell you about it when the time comes. So let's get started.
Backup jobs
First, let's look at the part that has not changed in version 10. The retention policy is determined by several parameters. Let's open the window for creating a new task and go to the Storage tab. Here we will see a parameter that determines the desired number of restore points:
However, this is only part of the equation. The actual number of points is also determined by the backup mode set for the job. To select this option, click on the Advanced button on the same tab. This will open a new window with many options. Let's number them and consider them one by one:
If only option 1 is enabled, the job will run in "infinitely incremental" mode (forever forward incremental). There are no difficulties here - the task will store the set number of restore points from a full backup (file with the VBK extension) to the last increment (file with the VIB extension). When the number of points exceeds the set value, the oldest increment will be merged with the full backup. In other words, if the task is set to store 3 points, then immediately after the next session there will be 4 points on the repository, after which the full backup will be merged with the oldest increment and the total number of points will return to 3.
Also extremely simple is the retention for the “reverse incremental” (reverse incremental) mode (option 2). Since in this case the newest point will be a full backup, followed by a chain of so-called rollbacks (files with the VRB extension), to apply the retention, it is enough to simply delete the oldest rollback. The situation will be the same: immediately after the session, the number of points will exceed the set value by 1, after which it will return to the desired value.
Note that with reverse-incremental mode, you can also enable periodic full backups (option 4), but this does not change the essence. Yes, full restore points will appear in the chain, but we will still just delete the oldest points one at a time.
Finally, we come to the interesting part. If you enable incremental backup, but also enable options 3 or 4 (or both at the same time), the task will start creating periodic full backups using the "active" or synthetic method. The method of creating a full backup is not important - it will contain the same data, and the incremental chain will be divided into "sub-chains". This method is called forward incremental, and it is he who causes a significant part of the questions from our customers.
Retention is applied here by deleting the oldest part of the chain (from a full backup to an increment). At the same time, we will not delete only a hollow backup or only part of the increments. The entire "subchain" is removed completely at once. The meaning of setting the number of points also changes - if in other methods this is the maximum allowable number, after which retention must be applied, then here this setting determines the minimum number. In other words, after deleting the oldest "subchain", the number of points in the remaining part should not fall below this minimum.
I will try to depict this concept graphically. Let's say the retention is set to 3 points, the task runs every day with a full backup on Monday. In this case, the retention will be applied when the total number of points reaches 10:
Why already 10 when they put up 3? On Monday, a full backup was created. From Tuesday to Sunday, the job created increments. Finally, next Monday, a full backup is created again and only when 2 increments are created can finally the entire old part of the chain be deleted, because the remaining number of points will not fall below the set 3.
If the idea is clear, then I suggest you try to calculate the retention yourself. Let's take the following conditions: the task is launched for the first time on Thursday (naturally, a full backup will be made). The task is set to create a full backup on Wednesdays and Sundays and store 8 restore points. When will the retention be applied for the first time?
To answer this question, I recommend that you take a sheet of paper, draw it by day of the week and write down which point is created each day. The answer will become obvious
Response
Clarification: To answer, it is enough to ask yourself “when will the retention be applied”? The answer is when we can remove the first 3 points (VBK, VIB, VIB) and the rest of the chain does not fall below the required 8 points. It becomes clear that we will be able to do this when we have 11 points in total, i.e. on Sunday of the second week.
Some readers may object: “why all this, if there is ? Without a doubt, it is a very useful tool, and in some cases I would use it, but it also has limitations. First of all, it does not allow you to specify the initial conditions, and in many cases the question is exactly “we have such a chain, what will happen if we change such and such settings?”. Secondly, the tool is still somewhat lacking in visibility. Showing the RPS page to clients, I did not find understanding, but after painting it as in the example (even using the same Paint), day after day, everything became clear.
Finally, we have not considered the option “Transform previous backup chains into rollbacks” (marked with number 5). This option sometimes confuses customers who activate it “on the fly”, wanting to enable a simple synthetic backup. Meanwhile, this option activates a very special backup mode. Without going into details, I’ll say right away that at this stage of product development “Transform previous backup chains into rollbacks” is an outdated option, and I can’t think of a single scenario when it should be used. Its value is so dubious that for some time Anton Gostev himself sent a call through the forum, asking him to send him examples of its useful use (if you have any, write in the comments, I'm very interested). If there are none (I think they will), then the option will be removed in future versions.
The job will create increments (VIB) until the day the synthetic full backup is scheduled. On this day, VBK is indeed created, but all the points before this VBK are transformed into rollbacks (VRB). After that, the job will continue to create increments to the full backup until the next synthetic backup. As a result, an explosive mixture of VBK, VBR and VIB files is created in the chain. Retension is applied very simply - by removing the last VBR:
Problems
Apart from actually understanding how it works, most of the problems that arise when using incremental mode are usually associated with a full backup. A regular full backup is necessary for this mode, otherwise the repository will accumulate points until it overflows.
For example, a full backup may be created too infrequently. Let's say the task is set to store 10 points, and a full backup is created once a month. It is clear that the actual number of points here will be much larger than the set one. Or the task is generally set to work in an infinite-incremental mode and store 50 points. Then someone accidentally created a full backup. That's it, from now on the task will wait until the full point accumulates 49 increments, after which it will apply the retention and return to the infinite-full mode.
In other cases, a full backup is set to be created regularly, but for some reason does not. I will list the most popular reason here. Some customers prefer to use the “run after” scheduling option and set up jobs to run in a chain. Let's take this example: there are 3 jobs that run every day and create a full backup on Sunday. The first task starts at 22.30, the rest are launched in a chain. An incremental backup takes 10 minutes, and therefore, by 23.00 all tasks finish their work. But a full backup takes an hour, so on Sunday the following happens: the first task runs from 22.30 to 23.30. The next one is from 23.30 to 00.30. But the third task is launched on Monday. A full backup is configured for Sunday, so in this case it simply will not be. The task will wait for a full backup to apply the retention. So be careful when using the “run after” option or don't use it at all - just set the jobs to start at the same time and let the resource scheduler do its job.
The difficult option “Remove deleted items”
Going through the settings of the Storage - Advanced - Maintenance task, you can stumble upon the option “remove deleted items data after”, calculated in days.
Some clients expect this to be retention. In fact, this is a completely separate option, the misunderstanding of which can lead to unexpected consequences. However, first of all, I need to explain how B&R reacts to situations where only a few machines are successfully backed up during the session.
Imagine this scenario: an infinite-incremental job configured to store 6 points. There are 2 machines in the task, one always backed up successfully, the other sometimes gave errors. As a result, by the seventh point, the following situation has developed:
It's time to apply the retention, but one machine has 7 points, and the other only 4. Will the retention be applied here? The answer is yes, it will. If at least one object has been backed up, B&R considers that the point has been created.
A similar situation can arise if some machine was simply not included in the task during a certain session. This happens, for example, when machines are added to the task not individually, but as part of containers (folders, storages) and some machine temporarily migrates to another container. The job will then be considered successful, but you will find a message in the statistics telling you to pay attention to the fact that such and such a machine is no longer processed by the job.
What will happen if you do not pay attention to it? In the case of infinite-incremental or reverse-incremental modes, the number of restore points of the "problem" machine will decrease with each session until it reaches 1 saved in VBK. In other words, even if the machine is not backed up for a long time, one restore point will still remain. This is not the case if periodic full backups are enabled. If signals from B&R are ignored, the last point may eventually be deleted along with the old part of the chain.
Having understood these details, we can finally consider the option “Remove deleted items data after”. It will delete all points for a particular machine if that machine is not backed up for X days. Please note that this setting does not respond to errors (tried - did not work). There should not even be an attempt to backup the machine. It would seem that the option is useful and should always be kept enabled. If the administrator removed the machine from the task, then it is logical to clear the chain of unnecessary data after a while. However, tuning requires discipline and care.
Let me give you an example from practice: several containers were added to the task, the composition of which was quite dynamic. Due to the lack of RAM, the B&R server experienced problems that went unnoticed. The task started and tried to make a backup of the machines, except for one, which at that time was not present in the container. Since many machines have generated errors, by default B&R has to make 3 additional attempts to backup "problem" machines. Due to constant problems with the RAM, these attempts dragged on for several days. There was no second attempt to backup the missing VM (the absence of a VM is not an error). As a result, during one of the repeated attempts, the “Remove deleted items” condition was met and all points of the machine were deleted.
On this occasion, I can say the following: if you have notifications about the results of tasks set up, and even better, integration with Veeam ONE is used, then most likely this will not happen to you. If you look at the B&R server once a week to check that everything is working, then it is better to refuse options that could potentially lead to the deletion of backups.
What's new in v.10
What we've been talking about before has existed at B&R for many versions. Having understood these principles of work, let's now see what was added in the anniversary "top ten".
Daily retention
Above, we considered the "classic" storage policy based on the number of points. An alternative approach is to set “days” instead of “restore points” in the same menu.
The idea is clear from the name - the retention will store the set number of days, the number of points in each day does not matter. In doing so, remember the following:
- The current day is not taken into account when calculating the retention
- Days when the task did not work at all are also counted. This should be kept in mind so that you do not accidentally lose the points of those tasks that work irregularly.
- The restore point is counted from the day when it was created (i.e. if the task started on Monday and finished on Tuesday, then this is a point from Monday)
Otherwise, the principles for applying retention by tasks are also determined by the chosen backup method. Let's try another calculation task using the same incremental method. Let's say the retention is set to 8 days, the task runs every 6 hours with a full backup on Wednesday. In this case, the task does not work on Sunday. The job runs on Monday for the first time. When will the retention be applied?
Response
As usual, it's best to draw a sign. I will allow myself to simplify the task and will not draw all the points created for each day, because the number of points per day does not matter here. It is only important for us that on the first Monday and on Wednesdays the first point will be a full backup, on the other days the task will simply create 4 incremental points.
We understand for ourselves that the retention will be applied by deleting the Monday full backup and its increment. When will it happen? When the rest of the chain will contain 8 days. At the same time, we do not count the current day, but Sunday, on the contrary, we count. So the answer is Thursday of the second week.
GFS archiving for regular jobs
Prior to v.10, the Grandfather-Father-Son (GFS) storage method was only available for Backup copy jobs and tape copy jobs. Now it is also available for a regular backup.
Although this is not related to the current topic, I can not say that the new functionality does not mean a departure from the 3-2-1 strategy. The presence of archive points in the main repository does not affect its reliability in any way. It is understood that GFS will be used in conjunction with a scale-out repository to ship these points to S3 and similar storages. If you do not use it, then it is better to continue to store primary and archive points in different repositories.
Now let's look at the principles of creating GFS points. In the task settings, at the Storage step, a special button has appeared that calls the following menu:
The essence of GFS can be reduced to several points (note that GFS works differently in other types of tasks, but more on that later):
- The task does not create a separate full backup under the GFS point. Instead, the most suitable full backup available will be used. Therefore, the job must run in incremental mode with a periodic full backup, or a full backup must be created manually by the user.
- If only one period is enabled (for example, a weekly period), then at the beginning of the GFS period, the task will simply start waiting for a full backup and mark the first suitable one as GFS.
Example: A job is configured to store a weekly GFS using a Wednesday backup. The task runs every day, but the full backup is scheduled for Friday. In this case, the GFS period will start on Wednesday and the task will start waiting for a suitable point. It will appear on Friday and will be marked with the GFS flag.
- If multiple periods are enabled at once (for example, weekly and monthly), then B&R will apply a method that allows the same point to be used as the GFS of multiple intervals (to save space). The flags will be assigned in turn, starting with the youngest.
Example: weekly GFS is set to Wednesday, and monthly GFS is set to the last week of the month. The job runs every day and creates full backups on Mondays and Fridays.
For simplicity, let's start counting from the penultimate week of the month. This week a full backup will be created on Monday, but it will be ignored because the weekly GFS interval starts on Wednesday. But the Friday full backup is completely suitable for the GFS point. This system is already familiar to us.
Now consider what will happen in the last week of the month. The monthly GFS interval will start on Monday, but the Monday VBK will not be tagged as GFS because the job seeks to tag one VBK as both a monthly and a weekly GFS point. At the same time, the search begins with the weekly one, therefore, by definition, it can also become a monthly one.
However, if only weekly and yearly intervals are enabled, they will operate independently of each other and may mark 2 separate VBKs as corresponding GFS intervals.
Backup copy jobs
Another type of task, often requiring clarification on the job. To begin with, let's analyze the "classic" method of work, without innovations v.10
A simple retention method
By default, such jobs run in infinite-incremental mode. The creation of points is determined by two parameters - the copying interval and the desired number of restore points (there is no retention by day here). The copy interval is set on the first Job tab when creating a job:
The number of points is determined a little further on the Target tab
The job creates 1 new point per interval (it doesn't matter how many points were created for the VM by the original jobs). At the end of the interval, the new point is finalized and, if necessary, a retention is applied by concatenating the VBK and the oldest increment. This mechanism is already familiar to us.
Retention method using GFS
BCJ can also store archived points. This is configured on the same Target tab, just below the number of restore points settings:
GFS points can be created in two ways - synthetically, using the data on the secondary repository, or by simulating a full backup and reading all the data from the primary repository (activated by the option marked with number 3). The retention in both cases will be very different, so we will consider them separately.
Synthetic GFS
In this case, the GFS point is not created exactly on the appointed day. Instead, a GFS point will be created when the VIB of the day the GFS point was scheduled to be created is merged with the full backup. This sometimes causes misunderstanding, because time goes by, but there is still no GFS point. And only a powerful shaman from technical support can predict on what day the point will still appear. In fact, magic is not needed - just look at the set number of points and the synchronization interval (how many points are created every day). Try to calculate it yourself using this example: the task is set to store 7 points, the synchronization interval is 12 hours (i.e. 2 points per day). At the moment, there are already 7 points in the chain, today is Monday, and the creation of a GFS point is scheduled for this day. What day will it be created?
Response
Here it is better to describe how the chain will change in dynamics, by day:
So, on Monday, the last increment in the chain is marked as GFS, but no other visible changes occur. Every day the task creates 2 new points, and the retention moves the chain inexorably forward. Finally, on Thursday, it's time to apply the retention to that same increment. This session will take longer than usual - because the task will "pull" the necessary blocks from the chain and create a new full point. From now on, there will be 8 points in the chain - 7 in the main chain + GFS.
Creating GFS points with “Read entire point” option
Above I said that BCJ works in infinite-incremental mode. Now we will analyze the only exception to this rule. If you enable the “Read entire point” option, the GFS point will be created exactly on the scheduled day. The task itself will work in incremental mode with periodic full backups, which we discussed above. The retention will also be applied by removing the oldest part of the chain. However, in this case, only increments will be deleted, and the full backup will be left as a GFS point. Accordingly, points marked with GFS flags are not taken into account when calculating the retention.
Suppose the task is set to store 7 points and create a weekly GFS point on Monday. In this case, every Monday the job will indeed create a full backup and mark it as GFS. The retention will be applied when, after deleting increments from the oldest part, the number of remaining increments does not fall below 7. This is how it looks on the diagram:
So, by the end of the second week, there are a total of 14 points in the chain. During the second week, the task created 7 points. If it were a simple task, the retention would have already been applied. But this is a BCJ with GFS retention, so we do not count GFS points, which means there are only 6 of them. That is, we still cannot apply the retention. In the third week, we create another full backup with the GFS flag. 15 points, but again we do not count this one. And finally, on Tuesday of the third week, we create an increment. Now, if we remove the increments of the first week chain, the total number of increments will satisfy the set retention.
As mentioned above, in this method it is very important that full backups are created regularly. For example, if you set the main retention to 7 days, but only 1 annual point, it is easy to imagine that the increments will accumulate much, much more than 7. In such cases, it is better to use the synthetic method of creating GFS.
And again “Remove deleted items”
This option is also present for BCJ:
The logic of this option here is the same as in regular backup tasks - if the machine is not processed for the specified number of days, then its data is deleted from the chain. However, for BCJ this option is objectively more useful, and here's why.
In normal mode, BCJ works in an infinite-incremental mode, so if at some point the machine is removed from the task, then the retention will gradually delete all recovery points until there is only one left - in VBK. Now let's imagine that the job is also configured to create synthetic GFS points. When the time comes, the job will need to create a GFS for all the machines in the chain. If some machine has no new points at all - well, you have to use the one that is. And so every time. As a result, the following situation may arise:
Pay attention to the Files section: we have the main VBK and 2 weekly GFS points. And now to the Restore points section - in fact, these files contain the same image of the machine. Naturally, there is no point in such GFS points, they only take up space.
This situation is only possible when using synthetic GFS. To prevent this, use the “Remove deleted items” option. Just remember to set it for an adequate number of days. Tech support has seen cases where the option was set to less than the number of days than the synchronization interval - BCJ began to go on a rampage and delete points before they had time to create them.
Note also that this option does not affect existing GFS points. If you want to clean the archives, you need to do it manually - by right-clicking on the machine and selecting "Delete from disk" (in the window that appears, do not forget to check the "Remove GFS full backup" checkbox):
Innovation v.10 - immediate copy (immediate copy)
Having dealt with the "classic" functionality, let's move on to the new one. Innovation is one, but very important. This is a new mode of operation.
There is no such thing as a “sync interval”, the task will constantly monitor whether new points have appeared and copy them all, no matter how many there are. However, the job remains incremental, meaning that even if the main job creates a VBK or VRB, those points will be copied as a VIB. Otherwise, there are no surprises in this mode - both standard and GFS retention work according to the rules described above (although only synthetic GFS is available here).
Discs are spinning. Features of Rotated Drives Repositories
The constant threat of ransomware viruses has made it the de facto security standard to have a copy of the data on a medium where the virus cannot reach. One option is to use disk-rotating repositories, where disks are used in turn: while one disk is connected and writable, the rest are stored in a safe place.
To teach B&R to work with such repositories, in the repository settings, at the Repository step, click on the Advanced button and select the appropriate option:
After that, VBR will wait for the periodically existing chain to disappear from the repository, which means disk rotation. Depending on the type of repository and the type of job, B&R will behave differently. You can represent this with a table like this:
Let's consider each option.
Normal Job and Windows Repository
So, we have a task that saves chains to the first disk. During rotation, the created chain actually disappears, and the task needs to somehow survive this loss. It finds consolation in creating a full backup. Thus, each rotation means a full backup. But what happens to the dots on a disconnected drive? They are remembered and taken into account when calculating the retention. Thus, the set number of points in the task is how many points must be kept on all disks. Here's an example:
The job runs in infinite-incremental mode and is configured to store 3 restore points. But we also have a second disk, and we rotate it once a week (there may be more disks, this does not change the essence).
In the first week, the task will create points on the first disk and merge the extra ones. Thus, the total number of points will be three:
Then we connect the second disk. On startup, B&R will notice that the drive has changed. The chain on the first disk will disappear from the interface, but information about it will remain in the database. The job will now hold 3 dots on the second disk. The general situation will be like this:
Finally, we reattach the first drive. Before creating a new point, the task will check what is there with the retention. And the retention, I remind you, is set to store 3 points. In the meantime, we have 3 points on disk 2 (but it is offline and stored in a safe place where B&R cannot reach) and 3 points on disk 1 (but this one is connected). So, you can safely remove 3 points from disk 1, since they exceed the retention. After that, the task creates a full backup again, and our chain starts to look like this:
If the retention is configured to store days instead of the number of points, then the logic does not change. Also, GFS retention is not supported at all when using repositories with disk rotation.
Normal Job and Linux Repository Network Storage
This option is also possible, but in general is less recommended due to the restrictions imposed. The task will respond to disk rotation and the disappearance of the chain in the same way - by creating a full backup. The limitation is connected with the truncated retention mechanism.
Here, during rotation, the entire chain on the disconnected disk is simply deleted from the B&R database. Pay attention - from the database, the files themselves remain on the disk. They can be imported and used for recovery, but it is not difficult to guess that sooner or later such forgotten chains will fill the entire repository.
The solution is to add DWORD ForceDeleteBackupFiles as indicated on this page: . After that, the job will simply start deleting all the contents of the job folder or repository folder (depending on the value) on every rotation.
However, this is not an elegant retention, but rather a cleansing of all content. Unfortunately, technical support has encountered cases when the repository was simply the root directory of the disk, where, in addition to backups, there were other data. All this was destroyed during the rotation.
In addition, when ForceDeleteBackupFiles is enabled, it works for all types of repositories, that is, even repositories on Windows will stop applying retention and start deleting content. In other words, a local disk on Windows is the best choice for such a backup storage system.
Backup copy and Windows repository
With BCJ, things get even more interesting. Not only is there a full-fledged retention, but there is no need to make a full backup every time you change the disk! It works like this:
First, B&R starts making dots on the first disc. Let's say we set the retention to 3 points. The task will work in an infinite-incremental mode and merge everything superfluous (I remind you that GFS retention is not supported in this case).
Then we connect the second disk. Since there is no chain on it yet, we create a full backup, after which we have a second chain of three points:
Finally, it's time to reconnect the first drive. And that's where the magic comes in, since the task will not create a full backup, but instead just continue the incremental chain:
After that, in fact, each disk will have its own independent chain. Therefore, retention here does not mean the number of points on all disks, but the number of points on each disk separately.
Backup copy and Linux repository network storage
Again, all elegance is lost if the repository is not on a local Windows drive. This script works similarly to the simple task above. On each rotation, BCJ will create a full backup, and existing points will be forgotten. In order not to be left without free space, you need to use DWORD ForceDeleteBackupFiles.
Conclusion
So, as a result of such a long text, we have considered two types of tasks. Of course, there are many more tasks, but it will not be possible to consider them all in the format of one article. If after reading you still have any questions, then write them in the comments, I will be happy to answer personally.
Source: habr.com