There are many backup systems, but what if the serviced servers are scattered across different regions and clients and you need to manage using the operating system tools?
Good afternoon, Habr!
My name is Natalya. I am a team leader of the application administrators group at NPO Krista. We ops for our company's project team. We have a rather peculiar situation: we install and maintain our software both on our company's servers and on servers located at clients' sites. In this case, there is no need to backup the entire server. Only the βessential dataβ is important: the DBMS and individual directories of the file system. Of course, clients have (or don't have) their own backup policies and often provide some kind of external storage to store backups there. In this case, after creating a backup, we ensure that it is sent to external storage.
For some time, for backup purposes, we managed with a bash script, but as the options for settings grew, the complexity of this script grew proportionately, and at one fine moment we came to the need to βdestroy it to the ground, and then ....β.
Ready-made solutions did not fit for various reasons: due to the need to decentralize backups, the obligation to store backups locally at the client, the complexity of settings, import substitution, and access restrictions.
It seemed to us that it is easier to write something of our own. At the same time, we wanted to get something that would be enough for our situation for the next N years, but with the possibility of potentially expanding the scope.
The task conditions were as follows:
- the base backup instance is offline, running locally
- storage of backups and logs always within the client's network
- the instance consists of modules - such a kind of "constructor"
- compatibility with Linux distributions used, including obsolete ones, potential cross-platform is desirable
- ssh access is sufficient to work with the instance, opening additional ports is optional
- maximum ease of setup and operation
- it is possible (but not necessary) to have a separate instance that allows you to centrally view the status of backups from different servers
What we got can be seen here:
The software is written in python3; works on Debian, Ubuntu, CentOS, AstraLinux 1.6.
The documentation is posted in the docs directory of the repository.
The main concepts that the system operates on:
action - an action that implements one atomic operation (database backup, directory backup, transfer from directory A to directory B, etc.). Existing actions are in the core/actions directory
task - a task, a set of actions that describes one logical "backup task"
schedule β schedule, task set with optional indication of task execution time
The backup configuration is stored in a yaml file; general config structure:
- General settings
- actions section: description of the actions used on this server
- schedule section: description of all tasks (sets of actions) and the schedule for their launch by cron, if such launch is required
What the application can do at the moment:
- the main operations for us are supported: PostgreSQL backup via pg_dump, file system directory backup via tar; operations with external storage; rsync between directories; backup rotation (deleting old copies)
- calling an external script
- manual execution of a single task
/opt/KristaBackup/KristaBackup.py run make_full_dump - you can add (or remove) an individual task or the entire schedule in the crontab
/opt/KristaBackup/KristaBackup.py enable all - trigger file generation based on backup results. This feature is useful in conjunction with Zabbix for monitoring backups.
- can work in background in webapi or web mode
/opt/KristaBackup/KristaBackup.py web start [--api]
The difference between modes is that webapi does not have a proper web interface, but the application responds to requests from another instance. For web mode, you need to install flask and a few additional packages, and this is not acceptable everywhere, for example, in certified AstraLinux SE.
Through the web interface, you can view the status and logs of backups of connected servers: the βweb instanceβ requests data from the βbackup instancesβ via the API. Web access requires authorization, webapi access does not.
Logs of incorrectly past backups are marked with a color: warning - yellow, error - red.
If the administrator does not need a cheat sheet on the parameters and the server operating systems are homogeneous, you can compile the file and distribute the finished package.
We distribute this utility mainly through Ansible, rolling it out first to some of the least important servers, and after testing to all the rest.
The end result is a compact stand-alone copy utility that can be automated and usable even by inexperienced administrators. It's convenient for us - maybe it will be useful for you too?
Source: habr.com