Background information, so to speak:
Fujitsu rx300 s6 server, RAID6 made of 6 1TB disks, running XenServer 6.2, several servers are operational, including Ubuntu with multiple shares, 3.5 million files, 1.5TB of data, all of which is gradually growing and swelling.
Task: configure data backup from the file server, partially daily, partially weekly.
We have a Windows machine for backup with RAID5 (a basic system unit with an integrated RAID controller) plus a separate 2TB disk for intermediate copying of the current file state. Any Linux distribution could be used, but this machine was already available with a RAID array and Windows license.
We install on the backup server , configure a "mirror" of everything from all shares of the file server once a day in the evening after 6 PM using a scheduler.
An important note: when saving the batch job, be sure to check "Close the task window on completion"; otherwise, processes will proliferate and multiply.
We add temporary files to the exclusions with the mask: *.dwl, *.dwl2, *.tmp.
FreeFileSync utilizes the network exceptionally well; copying occurs in multiple streams, reaching speeds of up to 80 Mbps when copying large files, no bottlenecks have been found with small files.
Archiving will be performed on the local backup server, instead of the previously used with network archiving. By the way, TheCopier is excellent! But with such volumes, it simply can't transfer everything in time, despite the 1Gbps interface on the backup machine and 2Gbps on the file server (bonding two network cards).
SyncToy was also previously used To archive the necessary folders, we write a batch file for
7-zip :
set now=%now::=.
set now=%now: =0%
set now=TE:~-4%.TE:~3,2%.TE:~0,2%_%now%
set now=TE:~-4%.TE:~3,2%.TE:~0,2%_%now%
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_35-110.zip E:\35-110
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_asu.zip E:\asu
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_director.zip E:\director
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_gpr.zip E:\gpr
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_otiz.zip E:\otiz
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_ps.zip E:\ps
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All%now%_copy.zip E:\copy
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backups\All%now%_pto.zip E:pto
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backups\All%now%_rza.zip E:rza
C:"Program Files"\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:backups\All%now%_smeta.zip E:smeta
:: a — creating the archive
:: -tzip or -t7z — archive type (zip is 1.5-2 times faster)
:: -mx=1 — compression level (1 minimum, 9 maximum, values x=[0 | 1 | 3 | 5 | 7 | 9])
:: -mmt=on — enables multithreading where it might not be enabled
:: -mtc=off — disables file system timestamps (when created, modified, etc.)
:: -ssw — also compresses files that are open for writing
:: -xr!.Sync* — excludes temporary BtSync files from archiving, leaving permanent files
The construction set now=% and so on allows you to preserve the timestamp format in the file name without issues that arose when the day or month number was less than 10, meaning we add a zero.
The comment -xr!.Sync* is a remnant left from the originally used setup .
Up to 500 GB and 700-800 thousand files BTSync worked fine, synchronizing on the fly, but with the current volumes, it consumed a lot of memory and CPU resources both on the Ubuntu file server and on the Windows backup server, where it was running as a service, as well as just heavily stressing the disk system with constant reads and writes.
Although we use an archiver and 7-zip, we archive in zip format instead of the native 7z, because it is much faster, and there is practically no difference in compression at mx=1, which has been tested many times.
Archives are processed sequentially.
Cleaning the archive folder also happens through a scheduled task using the fpurge utility, retaining archives no older than a week.
As a result, we have a copy of files from the previous day, along with archives from the last week, and deleted files FreeFileSync moves to the recycle bin.
Source: habr.com
