{"id":86904,"date":"2020-07-01T07:42:30","date_gmt":"2020-07-01T05:42:30","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud"},"modified":"2020-07-01T07:42:30","modified_gmt":"2020-07-01T05:42:30","slug":"kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","title":{"rendered":"How GitLab Helps to Backup Large NextCloud Storage","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Hello, Habr!<\/p>\n<p><\/p>\n<p>Today I want to share our experience in automating backups of large Nextcloud data storage in different configurations. I work as the CTO at 'Lightning AC', where we handle configuration management of IT systems, using Nextcloud for data storage. This includes distributed structures with redundancy.<\/p>\n<p><\/p>\n<p>The challenges stem from the specifics of the installations, as there is a large amount of data. The versioning provided by Nextcloud, backups, subjective reasons, and others create many duplicates.<\/p>\n<p><\/p>\n<h2 id=\"predystoriya\">Background<\/h2>\n<p><\/p>\n<p>When administering Nextcloud, the need to organize an effective backup solution arises, which must be encrypted since the data is valuable.<\/p>\n<p><\/p>\n<p>We offer backup storage options either with us or with the client on their separate machines from Nextcloud, which requires a flexible automated approach to administration.<\/p>\n<p><\/p>\n<p>There are many clients, all with different configurations, each on their own premises with their own peculiarities. The standard method, when the entire site belongs to you and backups are taken from cron, is poorly applicable here.<\/p>\n<p><\/p>\n<p>First, let\u2019s look at the initial data. We need:<\/p>\n<p><\/p>\n<ul>\n<li>Scalability regarding one node or several. For large installations, we use minio as the storage solution.<\/li>\n<li>To be informed about issues with backup execution.<\/li>\n<li>Backups need to be stored with the clients and\/or with us.<\/li>\n<li>Quickly and easily resolve issues.<\/li>\n<li>Clients and installations differ significantly from one another\u2014achieving uniformity is not possible.<\/li>\n<li>Recovery speed must be minimal in two scenarios: full restoration (disaster), and one folder\u2014accidentally deleted.<\/li>\n<li>Deduplication functionality is mandatory.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"How GitLab Helps to Backup Large NextCloud Storage\" src=\"\/wp-content\/uploads\/2020\/07\/1a0048e3295fef94ddca28e50612e3cb.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>To address the task of backup management, we integrated GitLab. More details below. <\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Of course, we are not the first to tackle such a challenge, but we believe that our hard-earned practical experience could be interesting, and we are ready to share it.<\/p>\n<p><\/p>\n<p>Since our company follows an open-source policy, we sought a solution specifically with open source. In turn, we share our developments and publish them. For example, on GitHub there is <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/molnia-com\/nextcloud-safe-trash\">our plugin for Nextcloud<\/a><\/noindex>, which we install for clients, enhancing data retention in case of accidental or intentional deletion.<\/p>\n<p><\/p>\n<h2 id=\"sredstva-bekapirovaniya\">Backup tools<\/h2>\n<p><\/p>\n<p>We began searching for solutions by selecting a backup creation tool.<\/p>\n<p><\/p>\n<p>Regular tar + gzip works poorly \u2014 data gets duplicated. The increment often contains very few actual changes, and most of the data within a single file is repeated.<br \/>\nThere is another issue \u2014 redundancy in the distributed data storage. We use minio and its data is inherently redundant. Backup needed to be done through minio itself \u2014 putting a load on it and utilizing all interspaces between the file system, and importantly, there's a risk of forgetting about some buckets and meta-information. Alternatively, deduplication could be used.<\/p>\n<p><\/p>\n<p>Deduplicating backup tools are available in open source (there was an article on Habr <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/449282\/\">article<\/a><\/noindex> <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/420055\/\">on this topic<\/a><\/noindex>) and our finalists became <noindex><a rel=\"nofollow\" href=\"https:\/\/borgbackup.readthedocs.io\/en\/stable\/\">Borg<\/a><\/noindex> and <noindex><a rel=\"nofollow\" href=\"https:\/\/restic.net\/\">Restic<\/a><\/noindex>. Below is our comparison of the two applications, but first, let's discuss how we organized the entire scheme.<\/p>\n<p><\/p>\n<h2 id=\"upravlenie-sozdaniem-rezervnyh-kopiy\">Backup management<\/h2>\n<p><\/p>\n<p>Borg and Restic are good, but neither product has a centralized management mechanism. For management and control purposes, we chose a tool that is already implemented with us, without which we cannot envision our work, including automation \u2014 this is the well-known CI\/CD \u2013 GitLab.<\/p>\n<p><\/p>\n<p>The idea is as follows: a gitlab-runner is installed on each node storing Nextcloud data. The runner schedules a script that monitors the backup process, which then initiates Borg or Restic.<\/p>\n<p><\/p>\n<p>What do we gain? Feedback from executions, convenient change control, and details in case of errors.<\/p>\n<p><\/p>\n<p>Here <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/molnia-com\/backup-by-runner\">here on GitHub<\/a><\/noindex> we have provided script examples for different tasks, and we ultimately integrated it not only for Nextcloud backup but for many other services as well. There is also a scheduler included for those who prefer not to set it up manually (and we don\u2019t want to) along with .gitlab-ci.yml.<\/p>\n<p><\/p>\n<p>In the GitLab API, there is currently no option to change the CI\/CD timeout, and it is quite short. It needs to be increased, say to <code>1d<\/code>.<\/p>\n<p><\/p>\n<p>Fortunately, GitLab can be triggered not only by commits but also by schedules, which is exactly what we need.<\/p>\n<p><\/p>\n<p>Now about the wrapper script.<\/p>\n<p><\/p>\n<p>We set the following conditions for this script:<\/p>\n<p><\/p>\n<ul>\n<li>It should run both as a runner and manually from the console with the same functionality.<\/li>\n<li>Error handlers must be included:<\/li>\n<li>return code.<\/li>\n<li>search for a string in the log. For example, an error for us could be a message that the program does not consider fatal.<\/li>\n<li>Timeout processing. Execution time should be reasonable.<\/li>\n<li>We need a detailed log. But only in case of an error.<\/li>\n<li>A series of tests are also conducted before starting.<\/li>\n<li>Some handy features that we found useful during support:<\/li>\n<li>Start and finish are recorded in the local machine's syslog. This helps link system errors and backup operations.<\/li>\n<li>A portion of the error log, when present, is output to stdout, while the entire log is written to a separate file. It's convenient to check directly in CI and assess the error if it's trivial.<\/li>\n<li>Debugging modes.<\/li>\n<\/ul>\n<p><\/p>\n<p>The full log is saved as an artifact in GitLab; if there are no errors, the log is deleted. We write the script in bash.<\/p>\n<p><\/p>\n<p>We welcome any suggestions and feedback on open source.<\/p>\n<p><\/p>\n<h2 id=\"kak-eto-rabotaet\">How it works<\/h2>\n<p><\/p>\n<p>A runner with a bash executor runs on the backed-up node. In the scheduler, a job CI\/CD is launched in a special repo. The runner executes a universal wrapper script for such tasks, which checks the validity of the backup repository, mount points, and everything else we want, then backup and old data cleanup is performed. The completed backup is sent to S3.<\/p>\n<p><\/p>\n<p>We operate on this scheme \u2014 it\u2019s an external provider like AWS or a Russian equivalent (this is faster and data does not leave Russia). Alternatively, we set up a separate MinIO cluster at the client's site for these purposes. Usually, we do this for security reasons when the client absolutely does not want their data to leave their perimeter.<\/p>\n<p><\/p>\n<p>We did not use the feature to send backups over SSH. It does not add security, and the network capabilities of the S3 provider are significantly higher than our single SSH machine.<\/p>\n<p><\/p>\n<p>To protect against hackers on the local machine \u2014 as they could erase data on S3, it is essential to enable versioning.<br \/>\nThe backup tool always encrypts the backup.<\/p>\n<p><\/p>\n<p>Borg has a mode without encryption. <code>none<\/code>, but we strongly advise against enabling it. In this mode, not only is there no encryption, but checksums are not calculated for what is being written, meaning integrity can only be verified indirectly, through indexes.<\/p>\n<p><\/p>\n<p>A separate scheduler checks the integrity of backups for indexes and content. The check runs slowly and takes a long time, so we run it separately once a month. It can take several days.<\/p>\n<p><\/p>\n<h2 id=\"ridmi-na-russkom\">Readme in Russian<\/h2>\n<p><\/p>\n<h3 id=\"osnovnye-funkcii\">Main features<\/h3>\n<p><\/p>\n<ul>\n<li><code>prepare<\/code> Preparation<\/li>\n<li><code>testcheck<\/code> Readiness check<\/li>\n<li><code>maincommand<\/code> Main command<\/li>\n<li><code>forcepostscript<\/code> A function that is executed at the end or on error. Used to unmount the partition.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"service-functions\">Service functions<\/h3>\n<p><\/p>\n<ul>\n<li><code>Cleanup<\/code> We record errors or erase the log file.<\/li>\n<li><code>Check log<\/code> We parse the log for occurrences of strings with errors.<\/li>\n<li><code>ret<\/code> Exit handler.<\/li>\n<li><code>Check timeout<\/code> Timeout check.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"environment\">Environment<\/h3>\n<p><\/p>\n<ul>\n<li><code>VERBOSE=1<\/code> Output errors to the screen immediately (stdout).<\/li>\n<li><code>SAVELOGSONSUCCES=1<\/code> Save the log on success.<\/li>\n<li><code>INIT_REPO_IF_NOT_EXIST=1<\/code> Create the repository if it does not exist. Off by default.<\/li>\n<li><code>TIMEOUT<\/code> maximum time for the main operation. You can set it as 'm', 'h' or 'd' at the end.<\/li>\n<\/ul>\n<p><\/p>\n<p>Old copy storage mode. By default:<\/p>\n<p><\/p>\n<ul>\n<li><code>KEEP_DAILY=7<\/code><\/li>\n<li><code>KEEP_WEEKLY=4<\/code><\/li>\n<li><code>KEEP_MONTHLY=6<\/code><\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"peremennye-vnutri-skripta\">Variables inside the script<\/h2>\n<p><\/p>\n<ul>\n<li><code>ERROR_STRING<\/code> \u2014 String for the check in the log for errors.<\/li>\n<li><code>EXTRACT_ERROR_STRING<\/code> \u2014 Expression to show the string if there's an error.<\/li>\n<li><code>KILL_TIMEOUT_SIGNAL<\/code> \u2014 Signal for killing if timeout.<\/li>\n<li><code>TAIL<\/code> \u2014 How many strings with errors on the screen.<\/li>\n<li><code>COLORMSG<\/code> \u2014 Color of the message (default yellow).<\/li>\n<\/ul>\n<p><\/p>\n<p>The script referred to as Wordpress is so called conditionally, its feature being that it also backs up the MySQL database. Hence, it can be used for one-time installations of Nextcloud, where you can also back up the database. The convenience lies not only in having everything in one place but also in the fact that the database content is close to the file content, since the time difference is minimal.<\/p>\n<p><\/p>\n<h2 id=\"restic-vs-borg\">Restic vs Borg<\/h2>\n<p><\/p>\n<p>Comparisons between Borg and Restic include <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/454734\/\">here on Habr<\/a><\/noindex>, and we did not have the goal of making just another one, but our own. It was important for us how it would look on our data, with our specifics. We present them.<\/p>\n<p><\/p>\n<p>Our selection criteria, apart from the already mentioned ones (deduplication, quick recovery, etc.):<\/p>\n<p><\/p>\n<ul>\n<li>Resistance to unfinished jobs. Check for kill -9.<\/li>\n<li>Size on disk.<\/li>\n<li>Resource requirements (CPU, memory).<\/li>\n<li>Size of stored blobs.<\/li>\n<li>Working with S3.<\/li>\n<li>Integrity checks.<\/li>\n<\/ul>\n<p><\/p>\n<p>For testing, we took one client with real data with a total size of 1.6TB.<br \/>\nConditions.<\/p>\n<p><\/p>\n<p>Borg can't work directly with S3, so we mounted it as a fuse disk, via <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kahing\/goofys\">goofys<\/a><\/noindex>. Restic sent data directly to S3.<\/p>\n<p><\/p>\n<p>Goofys works very quickly and well, and it has a <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kahing\/catfs\">disk cache module<\/a><\/noindex>, which further speeds up operations. It is in beta stage, and frankly, we did experience crashes with data loss during tests (others). But the convenience is that the backup procedure does not require much reading, mostly writing, so we use the cache only during integrity checks.<\/p>\n<p><\/p>\n<p>To reduce network impact, a local provider was used \u2014 Yandex Cloud.<\/p>\n<p><\/p>\n<p>Comparison test results.<\/p>\n<p><\/p>\n<ul>\n<li>Kill -9 followed by a restart both passed successfully.<\/li>\n<li>Disk size. Borg can compress, so the results are expected.<\/li>\n<\/ul>\n<p><\/p>\n<p>Backuper<br \/>\nSize<\/p>\n<p>Borg<br \/>\n562Gb<\/p>\n<p>Restic<br \/>\n628Gb<\/p>\n<p><\/p>\n<ul>\n<li>By CPU<br \/>\nBorg itself consumes little, with default compression, but it should be evaluated together with the goofys process. In total, they are comparable and utilize about 1.2 cores on the same test virtual machine.<\/li>\n<li>Memory. Restic roughly 0.5Gb, Borg about 200Mb. But this is all insignificant compared to the system's file cache. So it's advisable to allocate more memory.<\/li>\n<li>The difference in blob sizes was striking.<\/li>\n<\/ul>\n<p><\/p>\n<p>Backuper<br \/>\nSize<\/p>\n<p>Borg<br \/>\nabout 500Mb<\/p>\n<p>Restic<br \/>\nabout 5Mb<\/p>\n<p><\/p>\n<ul>\n<li>S3 interaction from Restic is excellent. Borg via goofys raises no issues, but it has been noted that it is advisable to do an umount at the end of backup to fully reset the cache. The peculiarity of S3 is that incomplete chunks will never be sent to the bucket, and thus partially uploaded data leads to significant corruption.<\/li>\n<li>Integrity checks work well in both cases, but the speed differs significantly.<br \/>\nRestic \u2013 <strong>3.5 hours<\/strong>.<br \/>\nBorg, with a 100Gb SSD file cache \u2013 <strong>5 hours<\/strong>. Approximately the same speed result if the data is on a local disk.<br \/>\nBorg reads directly from S3 without cache <strong>33 hours<\/strong>. Horribly long.<\/li>\n<\/ul>\n<p><\/p>\n<p>In summary, Borg can compress and has larger blobs \u2014 making storage and GET\/PUT operations in S3 cheaper. But this comes at the cost of a more complex and slower verification process. As for recovery speed \u2014 we noticed no difference. Subsequent backups (after the first) take slightly longer with restic, but not significantly.<\/p>\n<p><\/p>\n<p>The size of the community was also a key factor in the choice.<\/p>\n<p><\/p>\n<p>And we chose Borg.<\/p>\n<p><\/p>\n<h2 id=\"paru-slov-o-szhatii\">A few words about compression<\/h2>\n<p><\/p>\n<p>Borg has a wonderful new compression algorithm in its arsenal \u2014 zstd. Its compression quality is no worse than gzip, but significantly faster. And comparable in speed to the default lz4.<\/p>\n<p><\/p>\n<p>For example, a MySQL database dump compresses about twice as well as lz4 at the same speed. However, experience with real data shows only a minimal difference in the compression degree of Nextcloud nodes.<\/p>\n<p><\/p>\n<p>Borg has a quite bonus compression mode \u2014 if a file has high entropy, no compression is applied at all, which increases work speed. It is enabled with an option during creation<br \/>\n<code>-C auto,zstd<\/code><br \/>\nfor the zstd algorithm.<br \/>\nSo with this option compared to the default compression, we got<br \/>\n560Gb and 562Gb respectively. The data from the example above, reminds me, without compression the result is 628Gb. The 2Gb difference surprised us somewhat, but we decided that we would still choose <code>auto,zstd<\/code>.<\/p>\n<p><\/p>\n<h2 id=\"metodika-proverki-bekapa\">Backup verification methodology<\/h2>\n<p><\/p>\n<p>The virtual machine is launched either at the provider's site or at the client's, which significantly reduces network load. At the very least, this is cheaper than hosting it on our end and pushing traffic.<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">goofys --cache \"--free:5%:\/mnt\/cache\" -o allow_other --endpoint https:\/\/storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com \/mnt\/goofys\nexport BORG_PASSCOMMAND=\"cat \/home\/borg\/.borg-passphrase\"\nborg list \/mnt\/goofys\/borg1\/\nborg check --debug -p --verify-data \/mnt\/goofys\/borg1\/<\/code><\/pre>\n<p><\/p>\n<p>We verify files with antivirus in the same manner (post factum). After all, users upload different files to Nextcloud, and not all of them have antivirus. Checking at the moment of upload takes too much time and interferes with business.<\/p>\n<p><\/p>\n<p>Scalability is achieved by launching runners on different nodes with different tags.<br \/>\nIn our monitoring system, backup statuses are collected through the GitLab API in one window; if necessary, problems are easily identified and localized.<\/p>\n<p><\/p>\n<h2 id=\"zaklyuchenie\">Conclusion<\/h2>\n<p><\/p>\n<p>As a result, we know for sure that we are making backups, that our backups are valid, and the problems that arise with them take little time and are resolved at the duty administrator level. Backups really take up little space compared to tar.gz or Bacula.<\/p>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/508758\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0421\u0422\u041e \u0432 \u00ab\u041c\u043e\u043b\u043d\u0438\u044f \u0410\u041a\u00bb, \u0433\u0434\u0435 \u043c\u044b \u0437\u0430\u043d\u0438\u043c\u0430\u0435\u043c\u0441\u044f \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u043c \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c IT \u0441\u0438\u0441\u0442\u0435\u043c, \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f Nextcloud. \u0412 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435, \u0441 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u043e\u0439 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439, \u0441 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u0432\u044b\u0442\u0435\u043a\u0430\u044e\u0449\u0438\u0435 \u0438\u0437 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0439 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u043d\u043e\u0433\u043e. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":86905,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-86904","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a GitLab \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0431\u044d\u043a\u0430\u043f\u044b \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 NextCloud | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-01T05:42:30+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-01T05:42:30+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47How GitLab Helps Create Backups for Large NextCloud Storages | ProHoster","description":"Hello, Habr! Today I want to share our experience of automating backup for large data storages in Nextcloud across different configurations.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a GitLab \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0431\u044d\u043a\u0430\u043f\u044b \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 NextCloud | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-01T05:42:30+00:00","article:modified_time":"2020-07-01T05:42:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"86904","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 14:03:54","updated":"2022-10-06 09:05:46","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/86904","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=86904"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/86904\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/86905"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=86904"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=86904"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=86904"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}