This note concludes the series on backups. It will discuss the logical organization of a dedicated server (or VPS) that is convenient for backups, and will also offer a method for quickly restoring the server from a backup with minimal downtime in case of a failure.
Source Data
A dedicated server usually has at least two hard drives used to organize a RAID level 1 array (mirroring). This is necessary to ensure the server can continue operating if one disk fails. If it's a standard dedicated server, there may be a separate hardware RAID controller with active caching technology on SSDs, allowing one or more SSDs to be connected in addition to the standard hard drives. Sometimes dedicated servers are offered where the local drives consist only of SATADOMs (small drives, essentially USB sticks connected to SATA ports), or even standard small (8-16GB) USB drives connected to a special internal port, while data is retrieved from a SAN connected via a dedicated storage network (Ethernet 10G, FC, etc.). There are also dedicated servers that boot directly from the SAN. I will not consider such options here, as in these cases, the task of backing up the server smoothly transitions to the specialist managing the SAN, where various proprietary technologies for creating state snapshots, built-in deduplication, and other sysadmin delights discussed in previous parts of this series are usually available. The disk array of a dedicated server can reach several tens of terabytes, depending on the number and size of the disks connected to the server. In the case of VPS, the capacities are more modest: usually no more than 100GB (though there are larger ones), and the rates for such VPS can easily be higher than those for the cheapest dedicated servers from the same hoster. A VPS often has a single disk, as it will typically be managed by a SAN (or something hyper-converged). Sometimes, a VPS may have several disks with different characteristics for various purposes:
- small system — for installing the operating system;
- large — for storing user data.
When reinstalling the system through the control panel, the disk with user data is not erased, while the system disk is completely overwritten. Additionally, in the case of VPS, the host may offer a button to take a snapshot of the VPS (or disk) state, but if you install your own operating system or forget to activate the necessary service within the VPS, part of the data may still be lost. Along with the button, a data storage service is usually offered, which is often severely limited. Typically, this is an account with access via FTP or SFTP protocol, sometimes along with SSH, with a limited shell (for example, rbash), or restrictions on running commands through authorized_keys (via ForcedCommand).
A dedicated server is connected to the network via two ports at a speed of 1 Gbps; sometimes, this may include cards at 10 Gbps. A VPS usually has a single network interface. Data centers typically do not limit internal data center network speed, but they do restrict internet access speed.
The typical load on such a dedicated server or VPS consists of a web server, a database, and an application server. Different auxiliary services may be installed, including for the web server or database: search engine, email system, and so on.
The storage space for backups is provided by a specially prepared server, which will be discussed in more detail later.
Logical organization of the disk system.
If there is a RAID controller, or if this is a VPS with a single disk, and there are no special preferences for the disk subsystem (for example, a separate fast disk for the database) — all free space is divided as follows: one partition is created, on top of which an LVM volume group is created, within which several volumes are created: 2 small volumes of the same size, used as the root filesystem (they alternate during updates for quick rollback, an idea borrowed from the Calculate Linux distribution), another one for the swap partition, and the remaining free space is divided into small volumes used as root filesystems for full-fledged containers, disks for virtual machines, and filesystems for accounts in /home (each account has its own filesystem), as well as filesystems for application containers.
Important note: volumes must be completely self-sufficient, i.e., they should not depend on each other or the root filesystem. In the case of virtual machines or containers, this requirement is automatically met. However, if these are application containers or home directories, it is worth considering separating the configuration files of the web server and other services in such a way as to minimize inter-volume dependencies. For example, each site runs under its own user, and the site's configuration files are located in the user's home directory, while in the web server settings, the site configuration files are included not through /etc/nginx/conf.d/.conf, but, for instance, /home//configs/nginx/*.conf
If there are multiple disks - a software RAID array can be created (and its caching can be configured on SSDs, if needed and possible), on top of which LVM can be assembled according to the rules suggested above. In this case, ZFS or BtrFS can also be used, but here you should think carefully: both require a much more serious approach to resources, and moreover, ZFS does not come bundled with the Linux kernel.
Regardless of the scheme used, it's always advisable to estimate the approximate write speed of changes to the disks in advance, and then calculate the amount of free space that will be reserved for creating snapshots. For example, if our server writes data at a speed of 10 megabytes per second and the total size of the data array is 10 terabytes, the synchronization time could reach up to a day (22 hours — that’s how long it would take to transfer that amount over a 1 Gbps network) — it’s wise to reserve about 800 GB. In reality, the figure will be less; it can be safely divided by the number of logical volumes.
Backup Storage Server Device
The main difference of a backup storage server is large, inexpensive, and relatively slow disks. Since modern HDDs have already surpassed the 10 TB mark in a single disk, the use of file systems or RAID with checksums is essential, because during the time of array reconstruction or file system recovery (which can take several days!), a second disk may fail due to increased load. This was not as critical with disks of up to 1 TB. For simplicity, I will assume that the disk space is divided into two approximately equal parts (for example, using LVM):
- volumes corresponding to the servers used for storing user data (where the latest backup will be deployed for verification);
- volumes used as BorgBackup repositories (this is where the backup data will directly go).
The working principle consists of creating separate volumes for each server for the BorgBackup repositories, where data from production servers will be sent. The repositories operate in append-only mode, which eliminates the possibility of intentional data deletion, and due to deduplication and periodic cleaning of repositories from old backups (annual copies remain, monthly for the last year, weekly for the last month, daily for the last week, and possibly — in special cases — hourly for the last day: a total of about 50 copies for each server).
In BorgBackup repositories, the append-only mode is not activated; instead, ForcedCommand in .ssh/authorized_keys is used in a form like this:
from="server address",command="/usr/local/bin/borg serve --append-only --restrict-to-path /home/servername/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc AAAAA.......A wrapper script is located at the specified path, which not only launches the binary with parameters but also starts the process of restoring the backup after the data extraction is complete. To do this, the wrapper script creates a marker file next to the corresponding repository. The last taken backup is automatically restored to the corresponding logical volume after the data upload process is finished.
This design allows for periodic cleaning of unnecessary backups and also prevents production servers from deleting anything on the backup storage server.
Backup process
The backup initiator is the dedicated server or VPS itself, as this scheme provides more control over the backup process from this server. First, a snapshot of the active root filesystem state is created, which is mounted and uploaded using BorgBackup to the backup storage server. After the data extraction is complete, the snapshot is unmounted and deleted.
In the case of a small database (up to 1 GB for each site), a database dump is created, which is saved in the corresponding logical volume, where other data of the same site is located, but in a way that the dump is not accessible through the web server. If the databases are large, it is necessary to set up 'hot' data extraction, for example, using xtrabackup for MySQL, or to work with WAL with archive_command in PostgreSQL. In this case, the database will be restored separately from the site data.
If containers or virtual machines are used, it is necessary to set up qemu-guest-agent, CRIU, or other necessary technologies. In other cases, no additional configurations are usually required — just create snapshots of the logical volumes, which are then processed similarly to the snapshot of the root filesystem state. After data extraction, the snapshots are deleted.
Subsequent work is carried out on the backup storage server:
- the latest backup in each repository is being checked,
- checking for the presence of a marker file indicating that the data extraction process is complete,
- data is being deployed to the corresponding local volume,
- the marker file is deleted,
Server recovery process,
If the primary server fails, a similar dedicated server is started, which boots from a standard image. Most likely, the booting will occur over the network, but the data center technician setting up the server can also copy this standard image directly to one of the disks. The booting occurs in RAM, after which the recovery process is initiated:
- a request is made to connect the block device via iscsinbd or another similar logical volume protocol containing the root file system of the failed server; since the root file system should be small — this stage should be completed within minutes. The bootloader is also restored;
- the structure of local logical volumes is recreated, logical volumes from the backup server are attached using the dm_clone kernel module: data restoration begins, and changes are immediately written to the local disks,
- a container with all available physical disks is launched — the server is fully restored, but with reduced performance;
- after data synchronization is complete, the logical volumes from the backup server are detached, the container is shut down, and the server is restarted;
After rebooting, the server will have all the data that existed at the time the backup was created, as well as include all changes made during the recovery process.
Other cycle articles,
Backup, Part 7: Conclusions
I invite you to discuss the proposed option in the comments, thank you for your attention!
Source: habr.com
