{"id":34954,"date":"2019-10-31T22:01:30","date_gmt":"2019-10-31T19:01:30","guid":{"rendered":"https:\/\/prohoster.info\/blog\/rezervnoe-kopirovanie-chast-3-obzor-i-testirovanie-duplicity-duplicati\/"},"modified":"2019-10-31T22:01:30","modified_gmt":"2019-10-31T19:01:30","slug":"rezervnoe-kopirovanie-chast-3-obzor-i-testirovanie-duplicity-duplicati","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervnoe-kopirovanie-chast-3-obzor-i-testirovanie-duplicity-duplicati","title":{"rendered":"Backup, part 3: Overview and testing of duplicity, duplicati","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Backup,<\/p>\n

<\/p>\n

This note discusses backup tools that perform backups by creating archives on a backup server.<\/p>\n

<\/p>\n

Among those that meet the requirements are duplicity (which has a nice interface in the form of deja dup) and duplicati.<\/p>\n

<\/p>\n

Another noteworthy backup tool is dar, but since it has a very extensive list of options \u2014 the testing method covers hardly 10% of what it is capable of \u2014 it will not be tested within the current cycle.<\/p>\n

<\/a><\/noindex><\/p>\n

Expected Results<\/h3>\n

<\/p>\n

Since both candidates create archives one way or another, a standard tar can be used as a benchmark.<\/p>\n

<\/p>\n

Additionally, we will assess how well data storage on the storage server is optimized by creating backups that contain only the differences between the full copy and the current state of files, or between past and current archives (incremental, differential, etc.).<\/p>\n

<\/p>\n

Behavior during Backup Creation:<\/strong><\/p>\n

<\/p>\n

    \n
  1. A relatively small number of files on the backup storage server (comparable to the number of backups or the size of data in GB), but their size is quite large (dozens to hundreds of megabytes).<\/li>\n
  2. The size of the repository will include only changes \u2014 duplicates will not be stored, thus the size of the repository will be smaller than when using software based on rsync.<\/li>\n
  3. There is expected to be a high CPU load when using compression and\/or encryption, as well as likely significant load on the network and disk subsystem if the archiving and\/or encryption process runs on the backup storage server.<\/li>\n<\/ol>\n

    <\/p>\n

    As a reference value, we will run the following command:<\/p>\n

    <\/p>\n

    cd \/src\/dir; tar -cf - * | ssh backup_server \"cat > \/backup\/dir\/archive.tar\"<\/code><\/pre>\n
    <\/p>\n
    The results of the execution are as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Execution time 3m12s. It is clear that the speed was limited by the disk subsystem of the backup storage server, as in the example with rsync<\/a><\/noindex>. Just a bit faster, as the write is going to a single file.<\/p>\n
    <\/p>\n
    We will also run the same option for compression, but enable compression on the backup server side:<\/p>\n
    <\/p>\n
    cd \/src\/dir; tar -cf - * | ssh backup_server \"gzip > \/backup\/dir\/archive.tgz\"<\/code><\/pre>\n
    <\/p>\n
    The results are as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Execution time 10m11s. Most likely, the bottleneck is the single-threaded compressor on the receiving side.<\/p>\n
    <\/p>\n
    The same command, but with compression offloaded to the server with the original data to test the hypothesis that the bottleneck is the single-threaded compressor.<\/p>\n
    <\/p>\n
    cd \/src\/dir; tar -czf - * | ssh backup_server \"cat > \/backup\/dir\/archive.tgz\"<\/code><\/pre>\n
    <\/p>\n
    It turned out that:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    The execution time was 9m37s. It is clearly visible that one core is being utilized by the compressor, as the network transmission speed and load on the source's disk subsystem are similar.<\/p>\n
    <\/p>\n
    To assess encryption, you can use openssl or gpg, adding an additional command in the pipe. openssl<\/code> or gpg<\/code> For reference, the command would be:<\/p>\n
    <\/p>\n
    cd \/src\/dir; tar -cf - * | ssh backup_server \"gzip | openssl enc -e -aes256 -pass pass:somepassword -out \/backup\/dir\/archive.tgz.enc\"<\/code><\/pre>\n
    <\/p>\n
    The results were as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    The execution time was 10m30s, with 2 processes running on the receiving side\u2014once again, the bottleneck was the single-threaded compressor, plus some overhead for encryption.<\/p>\n
    <\/p>\n
    UPD:<\/strong> At the request of bliznezz, I\u2019m adding tests with pigz. If using the compressor alone, it took 6m30s; if encryption is also included, it\u2019s about 7m. The drop on the lower graph is due to an unflushed disk cache:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Testing duplicity<\/h3>\n
    <\/p>\n
    Duplicity is Python software for backing up by creating encrypted archives in tar format.<\/p>\n
    <\/p>\n
    For incremental archives, librsync is used, so one can expect the behavior described in the previous note of the cycle.<\/a><\/noindex>.<\/p>\n
    <\/p>\n
    Backups can be encrypted and signed with gnupg, which is important when using different providers for backup storage (s3, backblaze, gdrive, etc.).<\/p>\n
    <\/p>\n
    Let's see what the results will be:<\/p>\n
    \nHere are the results obtained when running without encryption:<\/b><\/p>\n
    spoiler<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Execution time of each test run:<\/p>\n
    <\/p>\n
    Run 1
    \nRun 2
    \nRun 3<\/p>\n
    16m33s
    \n17m20s
    \n16m30s<\/p>\n
    8m29s
    \n9m3s
    \n8m45s<\/p>\n
    5m21s
    \n6m04s
    \n5m53s<\/p>\n
    <\/p>\n
    And here are the results when gnupg encryption was enabled, with a key size of 2048 bits:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Running time on the same data, with encryption:<\/p>\n
    <\/p>\n
    Run 1
    \nRun 2
    \nRun 3<\/p>\n
    17m22s
    \n17m32s
    \n17m28s<\/p>\n
    8m52s
    \n9m13s
    \n9m3s<\/p>\n
    5m48s
    \n5m40s
    \n5m30s<\/p>\n
    <\/p>\n
    A block size of 512 megabytes was specified, which is clearly visible on the graphs; CPU usage was consistently around 50%, meaning the program utilizes no more than one CPU core.<\/p>\n
    <\/p>\n
    The principle of how the program works is also quite clear: a chunk of data is taken, compressed, and sent to a backup storage server that may be relatively slow.
    \nAnother characteristic is the predictable runtime of the program, which depends only on the size of the changed data.<\/p>\n
    <\/p>\n
    Enabling encryption did not significantly increase the program's runtime, but it increased CPU load by about 10%, which can be quite a nice bonus.<\/p>\n
    <\/p>\n
    Unfortunately, this program was unable to correctly detect the situation with the directory renaming, resulting in the repository size being equal to the size of changes (i.e., all 18GB), but the ability to use an untrusted server for backups definitely outweighs this behavior.<\/p>\n
    <\/p>\n
    Testing duplicati<\/h3>\n
    <\/p>\n
    This software is written in C# and runs using a set of Mono libraries. It has a GUI, as well as a CLI version.<\/p>\n
    <\/p>\n
    The approximate list of main features is close to duplicity, including various providers for backup storage, however, unlike duplicity, most features are available without third-party tools. Whether this is a plus or a minus depends on the specific case, but for beginners, it is likely easier to have all features listed in front of them rather than having to install packages for Python, as is the case with duplicity.<\/p>\n
    <\/p>\n
    One small nuance is that the program actively writes a local SQLite database under the name of the user who starts the backup, so you need to pay extra attention to correctly specifying the necessary database at each start of the process using the CLI. When working through GUI or WEBGUI, the details will be hidden from the user.<\/p>\n
    \nLet's take a look at the metrics this solution can output:<\/b><\/p>\n
    If you disable encryption (though WEBGUI does not recommend doing this), the results are as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Running time:<\/p>\n
    <\/p>\n
    Run 1
    \nRun 2
    \nRun 3<\/p>\n
    20m43s
    \n20m13s
    \n20m28s<\/p>\n
    5m21s
    \n5m40s
    \n5m35s<\/p>\n
    7m36s
    \n7m54s
    \n7m49s<\/p>\n
    <\/p>\n
    With encryption enabled, using AES, the results are as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Running time:<\/p>\n
    <\/p>\n
    Run 1
    \nRun 2
    \nRun 3<\/p>\n
    29m9s
    \n30m1s
    \n29m54s<\/p>\n
    5m29s
    \n6m2s
    \n5m54s<\/p>\n
    8m44s
    \n9m12s
    \n9m1s<\/p>\n
    <\/p>\n
    And if you use the external program GnuPG, the results are as follows:<\/p>\n
    <\/p>\n
    \"Backup,<\/a><\/noindex><\/p>\n
    <\/p>\n
    Run 1
    \nRun 2
    \nRun 3<\/p>\n
    26m6s
    \n26m35s
    \n26m17s<\/p>\n
    5m20s
    \n5m48s
    \n5m40s<\/p>\n
    8m12s
    \n8m42s
    \n8m15s<\/p>\n
    <\/p>\n
    As can be seen, the program can work in multiple threads, but that does not make it a more efficient solution, and when comparing encryption performance\u2014starting an external program
    \nproved to be faster than using the library from the Mono set. This may be due to the external program being more optimized.<\/p>\n
    <\/p>\n
    A pleasant aspect was also the fact that the size of the repository corresponds exactly to the actual amount of changed data, i.e., Duplicati detected the renaming of the directory and handled this situation correctly. This can be seen during the second test run.<\/p>\n
    <\/p>\n
    Overall, the impressions of the program are quite positive, including its friendliness to beginners.<\/p>\n
    <\/p>\n
    Results<\/h3>\n
    <\/p>\n
    Both candidates worked rather slowly, but overall, compared to the usual tar, there is progress, at least with Duplicati. The cost of such progress is also clear \u2014 noticeable CPU load.
    \nIn general, there are no particular deviations when predicting results.<\/p>\n
    <\/p>\n
    Conclusions<\/h3>\n
    <\/p>\n
    If there is no rush and there is spare CPU capacity, any of the considered solutions will work; in any case, a significant amount of work has been done that should not be repeated by writing wrapper scripts over tar. The presence of encryption is a very useful feature if the server used for backup storage cannot be fully trusted.<\/p>\n
    <\/p>\n
    When compared to solutions based on rsync<\/a><\/noindex> \u2014 performance may be several times worse, despite the fact that plain tar worked faster than rsync by 20-30%.
    \nThere are savings on the size of the repository, but only with Duplicati.<\/p>\n
    <\/p>\n
    Announcement<\/h3>\n
    <\/p>\n
    Backup, part 1: Why backup is necessary, overview of methods and technologies<\/a><\/noindex>
    \n    Backup, part 2: Overview and testing of rsync-based backup solutions<\/a><\/noindex>
    \nBackup, Part 3: Overview and Testing of Duplicity, Duplicati, Deja Dup
    \nBackup, Part 4: Overview and Testing of zbackup, restic, borgbackup
    \nBackup, Part 5: Testing Bacula and Veeam Backup for Linux
    \nBackup, Part 6: Comparison of Backup Solutions
    \nBackup, Part 7: Conclusions<\/p>\n
    <\/p>\n
    Author:<\/u> Pavel Demkovich<\/p>\n
    Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
    \u0412 \u0434\u0430\u043d\u043d\u043e\u0439 \u0437\u0430\u043c\u0435\u0442\u043a\u0435 \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0435\u043d\u044b \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u044e\u0442 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0435 \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u043f\u0443\u0442\u0435\u043c \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u0430\u0440\u0445\u0438\u0432\u043e\u0432 \u043d\u0430 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u043c \u0441\u0435\u0440\u0432\u0435\u0440\u0435. \u0418\u0437 \u0442\u0435\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0443\u0434\u043e\u0432\u043b\u0435\u0442\u0432\u043e\u0440\u044f\u044e\u0442 \u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043d\u0438\u044f\u043c \u2014 duplicity (\u043a \u043a\u043e\u0442\u043e\u0440\u043e\u043c\u0443 \u0435\u0441\u0442\u044c \u043f\u0440\u0438\u044f\u0442\u043d\u044b\u0439 \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u0432 \u0432\u0438\u0434\u0435 deja dup) \u0438 duplicati. \u0415\u0449\u0435 \u043e\u0434\u043d\u043e \u0432\u0435\u0441\u044c\u043c\u0430 \u043f\u0440\u0438\u043c\u0435\u0447\u0430\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u043e \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u2014 dar, \u043d\u043e \u043f\u043e\u0441\u043a\u043e\u043b\u044c\u043a\u0443 \u0443 \u043d\u0435\u0433\u043e \u0438\u043c\u0435\u0435\u0442\u0441\u044f \u0432\u0435\u0441\u044c\u043c\u0430 \u043e\u0431\u0448\u0438\u0440\u043d\u044b\u0439 \u0441\u043f\u0438\u0441\u043e\u043a \u043e\u043f\u0446\u0438\u0439 \u2014 \u043c\u0435\u0442\u043e\u0434\u0438\u043a\u0430 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26307,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-34954","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Backup, Part 3: Overview and Testing of Duplicity, Duplicati | ProHoster","description":"In this.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervnoe-kopirovanie-chast-3-obzor-i-testirovanie-duplicity-duplicati","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0420\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0435 \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435, \u0447\u0430\u0441\u0442\u044c 3: \u041e\u0431\u0437\u043e\u0440 \u0438 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 duplicity, duplicati | ProHoster","og:description":"\u0412 \u0434\u0430\u043d\u043d\u043e\u0439.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervnoe-kopirovanie-chast-3-obzor-i-testirovanie-duplicity-duplicati","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:01:30+00:00","article:modified_time":"2019-10-31T19:01:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"34954","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 21:15:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:03:10","updated":"2026-01-21 21:15:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34954","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=34954"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34954\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/26307"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=34954"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=34954"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=34954"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}