{"id":35003,"date":"2019-10-31T22:01:46","date_gmt":"2019-10-31T19:01:46","guid":{"rendered":"https:\/\/prohoster.info\/blog\/rezervnoe-kopirovanie-chast-4-obzor-i-testirovanie-zbackup-restic-borgbackup\/"},"modified":"2019-10-31T22:01:46","modified_gmt":"2019-10-31T19:01:46","slug":"rezervnoe-kopirovanie-chast-4-obzor-i-testirovanie-zbackup-restic-borgbackup","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervnoe-kopirovanie-chast-4-obzor-i-testirovanie-zbackup-restic-borgbackup","title":{"rendered":"Backup, Part 4: Overview and Testing of zbackup, restic, borgbackup","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Backup,<\/p>\n

<\/p>\n

This article will cover software tools for backup that create a repository by splitting data streams into separate components (chunks).<\/p>\n

<\/p>\n

The repository components can be further compressed and encrypted, and most importantly \u2014 reused during subsequent backup processes.<\/p>\n

<\/p>\n

A backup in such a repository is a named chain of interrelated components, for example, based on various hash functions.<\/p>\n

<\/p>\n

There are several such solutions; I will focus on three: zbackup, borgbackup, and restic.<\/p>\n

<\/a><\/noindex><\/p>\n

Expected Results<\/h3>\n

<\/p>\n

Since all candidates require creating a repository in one way or another, one of the most important factors will be assessing the repository size. Ideally, its size should not exceed 13 GB according to the accepted methodology, or even less\u2014with good optimization.<\/p>\n

<\/p>\n

It is also highly desirable to have the ability to create backups of files directly, without using archivers like tar, as well as to work with ssh\/sftp without additional tools like rsync and sshfs.<\/p>\n

<\/p>\n

Behavior during Backup Creation:<\/strong><\/p>\n

<\/p>\n

    \n
  1. The size of the repository will equal the size of the changes, or be smaller.<\/li>\n
  2. A high CPU load is expected when using compression and\/or encryption, as well as potentially significant load on the network and disk subsystem if the archiving and\/or encryption process runs on the backup storage server.<\/li>\n
  3. If the repository is corrupted, a delayed error is likely both during the creation of new backups and during restoration attempts. Additional measures to ensure the integrity of the repository need to be planned, or built-in integrity checking tools should be used.<\/li>\n<\/ol>\n

    <\/p>\n

    The reference value is working with tar, as shown in a previous article.<\/p>\n

    <\/p>\n

    Testing zbackup<\/h3>\n

    <\/p>\n

    The overall mechanism of how zbackup works is that the program finds areas in the input data stream that contain identical data, then optionally compresses and encrypts them, saving each area only once.<\/p>\n

    <\/p>\n

    A 64-bit circular hash function with a sliding window is used for deduplication to check byte-by-byte for matches with existing data blocks (similar to how it is implemented in rsync).<\/p>\n

    <\/p>\n

    LZMA and LZO are applied for compression in a multithreaded manner, while AES is used for encryption. In the latest versions, there is an option to delete old data from the repository in the future.
    \nThe program is written in C++ with minimal dependencies. The author appears to have been inspired by the unix-way, thus the program takes data from stdin when creating backups, outputting a similar data stream to stdout during restoration. As a result, zbackup can be used as quite a good 'building block' for writing custom backup solutions. For instance, for the article's author, this program has been the primary backup tool for home machines since around 2014.<\/p>\n

    <\/p>\n

    The data stream will use the standard tar, unless specified otherwise.<\/p>\n

    \nLet's see what the results will be:<\/b><\/p>\n

    The testing was conducted in 2 variations:<\/p>\n

    <\/p>\n

      \n
    1. a repository is created and zbackup is run on the server with the source data, then the contents of the repository are transferred to the backup storage server.<\/li>\n
    2. a repository is created on the backup storage server, zbackup is launched via SSH on the backup storage server, receiving data from the pipe.<\/li>\n<\/ol>\n

      <\/p>\n

      The results of the first option were as follows: 43m11s \u2014 with an unencrypted repository and LZMA compressor, 19m13s \u2014 when switching the compressor to LZO.<\/p>\n

      <\/p>\n

      The load on the server with the source data was as follows (an example with LZMA is shown; with LZO, the picture was approximately the same, but the rsync share was about a quarter of the time):<\/p>\n

      <\/p>\n

      \"Backup,<\/a><\/noindex><\/p>\n

      <\/p>\n

      It is clearly evident that such a backup process is only suitable for relatively rare and minor changes. It is also highly advisable to limit the operation of zbackup to 1 thread; otherwise, CPU load will be significantly high, as the program is quite adept at working with multiple threads. The disk load was low, which, given the modern SSD-based disk subsystem, will generally go unnoticed. It is also clearly visible that the data synchronization process to the remote server is underway, with speeds comparable to regular rsync, limited by the performance of the backup server's disk subsystem. A downside of this approach is the storage of a local repository, which consequently leads to data duplication.<\/p>\n

      <\/p>\n

      A more interesting and practical alternative is the second option of running zbackup directly on the backup storage server.<\/p>\n

      <\/p>\n

      To start, the operation will be tested without using encryption with the lzma compressor:<\/p>\n

      <\/p>\n

      \"Backup,<\/a><\/noindex><\/p>\n

      <\/p>\n

      Execution time of each test run:<\/p>\n

      <\/p>\n

      Run 1
      \nRun 2
      \nRun 3<\/p>\n

      39m45s
      \n40m20s
      \n40m3s<\/p>\n

      7m36s
      \n8m3s
      \n7m48s<\/p>\n

      15m35s
      \n15m48s
      \n15m38s<\/p>\n

      <\/p>\n

      If encryption using aes is activated, the results are quite close:<\/p>\n

      <\/p>\n

      \"Backup,<\/a><\/noindex><\/p>\n

      <\/p>\n

      Running time on the same data, with encryption:<\/p>\n

      <\/p>\n

      Run 1
      \nRun 2
      \nRun 3<\/p>\n

      43m40s
      \n44m12s
      \n44m3s<\/p>\n

      8m3s
      \n8m15s
      \n8m12s<\/p>\n

      15m0s
      \n15m40s
      \n15m25s<\/p>\n

      <\/p>\n

      If encryption is combined with lzo compression, it appears as follows:<\/p>\n

      <\/p>\n

      \"Backup,<\/a><\/noindex><\/p>\n

      <\/p>\n

      Running time:<\/p>\n

      <\/p>\n

      Run 1
      \nRun 2
      \nRun 3<\/p>\n

      18m2s
      \n18m15s
      \n18m12s<\/p>\n

      5m13s
      \n5m24s
      \n5m20s<\/p>\n

      8m48s
      \n9m3s
      \nThe size of the resulting repository was relatively consistent at 13GB. This means that deduplication works correctly. Additionally, applying lzo to already compressed data yields a noticeable effect, causing the total running time of zbackup to closely approach duplicity\/duplicati, although it lags behind those based on librsync by 2-5 times.<\/p>\n

      <\/p>\n

      The advantages are obvious \u2014 saving disk space on the backup storage server. As for repository verification tools, they are not provided by the zbackup author, and it is recommended to use a fault-tolerant disk array or cloud provider.<\/p>\n

      <\/p>\n

      Overall, a rather pleasant impression, despite the fact that the project has been stagnant for about 3 years (the last feature request was made around a year ago, but without a response).<\/p>\n

      <\/p>\n

      Testing borgbackup<\/p>\n

      <\/p>\n

      Borgbackup is a fork of attic, another similar system to zbackup. Written in Python, it has a similar list of features to zbackup, but additionally can:<\/h3>\n

      <\/p>\n

      BorgBackup is a fork of Attic, another system similar to zbackup. Written in Python, it has a feature set comparable to zbackup, but additionally it can:<\/p>\n

      <\/p>\n