This note discusses backup tools that perform backups by creating archives on a backup server.
Among those that meet the requirements are duplicity (which has a nice interface in the form of deja dup) and duplicati.
Another noteworthy backup tool is dar, but since it has a very extensive list of options — the testing method covers hardly 10% of what it is capable of — it will not be tested within the current cycle.
Expected Results
Since both candidates create archives one way or another, a standard tar can be used as a benchmark.
Additionally, we will assess how well data storage on the storage server is optimized by creating backups that contain only the differences between the full copy and the current state of files, or between past and current archives (incremental, differential, etc.).
Behavior during Backup Creation:
- A relatively small number of files on the backup storage server (comparable to the number of backups or the size of data in GB), but their size is quite large (dozens to hundreds of megabytes).
- The size of the repository will include only changes — duplicates will not be stored, thus the size of the repository will be smaller than when using software based on rsync.
- There is expected to be a high CPU load when using compression and/or encryption, as well as likely significant load on the network and disk subsystem if the archiving and/or encryption process runs on the backup storage server.
As a reference value, we will run the following command:
cd /src/dir; tar -cf - * | ssh backup_server "cat > /backup/dir/archive.tar"The results of the execution are as follows:
Execution time 3m12s. It is clear that the speed was limited by the disk subsystem of the backup storage server, as in the example with . Just a bit faster, as the write is going to a single file.
We will also run the same option for compression, but enable compression on the backup server side:
cd /src/dir; tar -cf - * | ssh backup_server "gzip > /backup/dir/archive.tgz"The results are as follows:
Execution time 10m11s. Most likely, the bottleneck is the single-threaded compressor on the receiving side.
The same command, but with compression offloaded to the server with the original data to test the hypothesis that the bottleneck is the single-threaded compressor.
cd /src/dir; tar -czf - * | ssh backup_server "cat > /backup/dir/archive.tgz"It turned out that:
The execution time was 9m37s. It is clearly visible that one core is being utilized by the compressor, as the network transmission speed and load on the source's disk subsystem are similar.
To assess encryption, you can use openssl or gpg, adding an additional command in the pipe. openssl or gpg For reference, the command would be:
cd /src/dir; tar -cf - * | ssh backup_server "gzip | openssl enc -e -aes256 -pass pass:somepassword -out /backup/dir/archive.tgz.enc"The results were as follows:
The execution time was 10m30s, with 2 processes running on the receiving side—once again, the bottleneck was the single-threaded compressor, plus some overhead for encryption.
UPD: At the request of bliznezz, I’m adding tests with pigz. If using the compressor alone, it took 6m30s; if encryption is also included, it’s about 7m. The drop on the lower graph is due to an unflushed disk cache:
Testing duplicity
Duplicity is Python software for backing up by creating encrypted archives in tar format.
For incremental archives, librsync is used, so one can expect the behavior described in .
Backups can be encrypted and signed with gnupg, which is important when using different providers for backup storage (s3, backblaze, gdrive, etc.).
Let's see what the results will be:
Here are the results obtained when running without encryption:
spoiler
Execution time of each test run:
Run 1
Run 2
Run 3
16m33s
17m20s
16m30s
8m29s
9m3s
8m45s
5m21s
6m04s
5m53s
And here are the results when gnupg encryption was enabled, with a key size of 2048 bits:
Running time on the same data, with encryption:
Run 1
Run 2
Run 3
17m22s
17m32s
17m28s
8m52s
9m13s
9m3s
5m48s
5m40s
5m30s
A block size of 512 megabytes was specified, which is clearly visible on the graphs; CPU usage was consistently around 50%, meaning the program utilizes no more than one CPU core.
The principle of how the program works is also quite clear: a chunk of data is taken, compressed, and sent to a backup storage server that may be relatively slow.
Another characteristic is the predictable runtime of the program, which depends only on the size of the changed data.
Enabling encryption did not significantly increase the program's runtime, but it increased CPU load by about 10%, which can be quite a nice bonus.
Unfortunately, this program was unable to correctly detect the situation with the directory renaming, resulting in the repository size being equal to the size of changes (i.e., all 18GB), but the ability to use an untrusted server for backups definitely outweighs this behavior.
Testing duplicati
This software is written in C# and runs using a set of Mono libraries. It has a GUI, as well as a CLI version.
The approximate list of main features is close to duplicity, including various providers for backup storage, however, unlike duplicity, most features are available without third-party tools. Whether this is a plus or a minus depends on the specific case, but for beginners, it is likely easier to have all features listed in front of them rather than having to install packages for Python, as is the case with duplicity.
One small nuance is that the program actively writes a local SQLite database under the name of the user who starts the backup, so you need to pay extra attention to correctly specifying the necessary database at each start of the process using the CLI. When working through GUI or WEBGUI, the details will be hidden from the user.
Let's take a look at the metrics this solution can output:
If you disable encryption (though WEBGUI does not recommend doing this), the results are as follows:
Running time:
Run 1
Run 2
Run 3
20m43s
20m13s
20m28s
5m21s
5m40s
5m35s
7m36s
7m54s
7m49s
With encryption enabled, using AES, the results are as follows:
Running time:
Run 1
Run 2
Run 3
29m9s
30m1s
29m54s
5m29s
6m2s
5m54s
8m44s
9m12s
9m1s
And if you use the external program GnuPG, the results are as follows:
Run 1
Run 2
Run 3
26m6s
26m35s
26m17s
5m20s
5m48s
5m40s
8m12s
8m42s
8m15s
As can be seen, the program can work in multiple threads, but that does not make it a more efficient solution, and when comparing encryption performance—starting an external program
proved to be faster than using the library from the Mono set. This may be due to the external program being more optimized.
A pleasant aspect was also the fact that the size of the repository corresponds exactly to the actual amount of changed data, i.e., Duplicati detected the renaming of the directory and handled this situation correctly. This can be seen during the second test run.
Overall, the impressions of the program are quite positive, including its friendliness to beginners.
Results
Both candidates worked rather slowly, but overall, compared to the usual tar, there is progress, at least with Duplicati. The cost of such progress is also clear — noticeable CPU load.
In general, there are no particular deviations when predicting results.
Conclusions
If there is no rush and there is spare CPU capacity, any of the considered solutions will work; in any case, a significant amount of work has been done that should not be repeated by writing wrapper scripts over tar. The presence of encryption is a very useful feature if the server used for backup storage cannot be fully trusted.
When compared to solutions based on — performance may be several times worse, despite the fact that plain tar worked faster than rsync by 20-30%.
There are savings on the size of the repository, but only with Duplicati.
Announcement
Backup, Part 3: Overview and Testing of Duplicity, Duplicati, Deja Dup
Backup, Part 4: Overview and Testing of zbackup, restic, borgbackup
Backup, Part 5: Testing Bacula and Veeam Backup for Linux
Backup, Part 6: Comparison of Backup Solutions
Backup, Part 7: Conclusions
Author: Pavel Demkovich
Source: habr.com
