Anamnesis, so to speak:
Fujitsu rx300 s6 server, RAID6 of 6 1TB disks, XenServer 6.2 is up, several servers are spinning, among them Ubuntu with several balls, 3,5 million files, 1,5 TB of data, it all gradually grows and swells.
Task: set up data backup from the file server, partly daily, partly weekly.
We have a Windows machine for backup with RAID5 (for poverty, a regular system unit with a RAID controller built into the mother) plus a separate 2TB disk for intermediate copying of the current state of the files. It was possible to use any Linux distribution, but this machine was already available with a raid array and a Windows license.
Installing on a backup server , we set up a βmirrorβ of everything in a row from all the file server shares once a day in the evening after 18:XNUMX by launching through the scheduler.
An important point: when saving a batch task, be sure to check "Close the task window on completion", otherwise the processes will multiply and multiply.
We throw temporary files into exceptions by mask: *.dwl, *.dwl2, *.tmp.
FreeFileSync uses the network extremely well, copying goes in several streams, the speed reaches 80 Mbps when copying large files, no plugging was found on small files.
Archiving will be carried out already on a local backup server, instead of the previously used with network archiving. By the way, TheCopier is great! But with such volumes, it simply does not have time to transfer everything, despite the 1Gbps interface on the backup and 2Gbps on the file one (bond of two network cards).
Also previously used , but when the number of files was more than 1,5-2 million, it stopped working normally, it just couldn't cope.
To archive the necessary folders, we write a batch file for :
set now=%TIME:~0,-3%
set now=%now::=.%
set now=%now: =0%
set now=%DATE:~-4%.%DATE:~3,2%.%DATE:~0,2%_%now%
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_10-04.zip E:10-04
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_35-110.zip E:35-110
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_asu.zip E:asu
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_director.zip E:director
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_gpr.zip E:gpr
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_otiz.zip E:otiz
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_ps.zip E:ps
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_pto.zip E:pto
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_rza.zip E:rza
C:"Program Files"7-Zip7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backupsAll%now%_smeta.zip E:smeta
:: a - creating an archive
:: -tzip or -t7z - type of archive (zip is 1.5-2 times faster)
:: -mx=1 β compression ratio (1 minimum, 9 maximum x=[0 | 1 | 3 | 5 | 7 | 9 ])
:: -mmt=on - turns on multithreading where suddenly it is not turned on
:: -mtc=off - disables filesystem timestamps (when saved, changed, etc.)
:: -ssw - also compresses files opened for writing
:: -xr!.Sync* - excludes temporary BtSync files from archiving, leaving permanent ones
The construction of set now=% and so on allows you to save the time format in the file name without the problems that arose when the number of the day or month was less than 10, that is, we substitute a zero.
The -xr!.Sync* comment is a vestige from the original .
Up to 500 GB and 700-800 thousand files, BTSync still worked, synchronized on the fly, but at the current volumes it ate a lot of memory and processor resources both on the Ubuntu file server and on the Windows backup, where it was launched by the service, and also simply raped disk system with constant reads and writes.
Although the archiver is 7-zip, we archive it in the zip format instead of the native 7z, because it is much faster, and there is practically no difference in compression with mx=1, it has been verified by many experiments.
Archives are executed in turn.
The folder with archives is also cleaned through a scheduled task using the fpurge utility, leaving archives no older than a week.
As a result, we have a copy of the files for the previous day, as well as archives for the last week, FreeFileSync puts the deleted files in the trash.
Source: habr.com