is a free, open-source deduplicator — a utility similar to an archiver that significantly reduces the size of backups and organizes incremental and differential backup schemes, among other things.
This is an overview article describing the capabilities. The actual use of hashget (which is quite simple) is detailed in the project and .
Comparison
As per the genre conventions, let me start with the intrigue — a comparison of the results:
Data sample
unpacked size
.tar.gz
hashget .tar.gz
WordPress-5.1.1
43 Mb
11 Mb ( 26% )
155 Kb ( 0.3% )
Linux kernel 5.0.4
934 Mb
161 Mb ( 20% )
4.7 Mb ( 0.5% )
Debian 9 (LAMP) LXC VM
724 Mb
165 Mb ( 23% )
4.1 Mb ( 0.5% )
Background, what an ideal and efficient backup should be
Every time I backed up a freshly created virtual machine, I was bothered by the feeling that I was doing something wrong. Why was I getting a hefty backup from a system where my priceless immaterial creation was just a single line index.html with the text 'Hello world'?
Why is there a 16-megabyte /usr/sbin/mysqld in my backup? Is it really my honor in this world to store this important file, and if I don't handle it right, it will be lost to humanity? Most likely not. It is stored on highly reliable Debian servers (whose reliability and uptime are incomparable to what I can provide), as well as in backups (millions of them) from other admins. Do we really need to create 10,000,000 + 1 extra copy of this important file for increased reliability?
In general solves this problem. When packing, it creates a very small backup. When unpacking, it restores a fully unpacked system, similar to what would result from tar -c / tar -x. (In other words, this is lossless packing)
How hashget works
In hashget, there are concepts of Package and HashPackage, which help in performing deduplication.
Package (package). A file (usually a .deb or .tar.gz archive) that can be reliably downloaded from the internet, and from which one or more files can be extracted.
HashPackage is a small JSON file representing a Package, containing the URL of the package and the hash sums (sha256) of its files. For example, for the mariadb-server-core package, which is 5 megabytes in size, the size of the hashpackage is only 6 kilobytes. Roughly a thousand times smaller.
Deduplication — creating an archive without duplicated files (if the deduplicator knows where the original package can be downloaded, it omits duplicates from the archive).
Packing
When packing, all files from the directory being packed are reviewed, their hash sums are calculated, and if a sum is found in one of the known HashPackages, metadata about the file (name, hash, permissions, etc.) is saved in a special file .hashget-restore.json, which will also be included in the archive.
The packing itself in the simplest case looks no more complicated than tar:
hashget -zf /tmp/mybackup.tar.gz --pack /path/to/dataUnboxing
Unpacking is done in two stages. First, the regular tar unpacking:
tar -xf mybackup.tar.gz -C /path/to/datathen recovery from the network:
hashget -u /path/to/dataDuring recovery, hashget reads the .hashget-restore.json file, downloads the necessary packages, unpacks them, and extracts the required files, placing them in the appropriate paths with the correct owner/group/permissions.
More complex things
What was described above is already sufficient for those who want 'something like tar, but packing my Debian into 4 megabytes'. Let's look at more complex things.
Indexing
If hashget didn't have any HashPackage at all, it wouldn't be able to deduplicate anything.
You can also create a HashPackage manually (just: hashget --submit https://wordpress.org/wordpress-5.1.1.zip -p my), but there is a more convenient way.
To get the necessary hashpackages, there is a stage of indexing (it is performed automatically with the command --pack) and heuristics. During indexing, hashget 'feeds' each found file to all existing heuristics that it is interested in. Heuristics can then index some Package to create a HashPackage.
For example, the Debian heuristic likes the file /var/lib/dpkg/status and detects installed Debian packages, and if they are not indexed (no HashPackage has been created for them), it downloads and indexes them. This results in a very nice effect - hashget will always efficiently deduplicate Debian OS, even if they have the latest packages.
Hint files
If your network uses some proprietary package or a public package not included in hashget's heuristics, you can add a simple hint file hashget-hint.json as follows:
{
"project": "wordpress.org",
"url": "https://ru.wordpress.org/wordpress-5.1.1-ru_RU.zip"
}Then, each time an archive is created, the package will be indexed (if it hasn't been before), and the package files will be deduplicated from the archive. No programming is needed; everything can be done from vim, saving time on every backup. Note that due to the checksum approach, if some files from the package are changed locally (for example, a configuration file is modified), the changed files will be kept in the archive 'as is' and will not be reduced.
If one of your own packages is updated periodically but the changes are not significant, you can only do a hint for the major versions. For example, in version 1.0, a hint was made pointing to mypackage-1.0.tar.gz, and it will be fully deduplicated; then version 1.1 was released, which differs slightly, but the hint was not updated. No problem. Only files that match (which can be restored) with version 1.0 will be deduplicated.
The heuristic that processes the hint file is a good example for understanding the internal mechanism of how heuristics work. It only processes hashget-hint.json files (or .hashget-hint.json with the dot) and ignores all others. Based on this file, it determines which package URL should be indexed, and hashget indexes it (if it hasn't been done before).
HashServer
It would be quite labor-intensive to perform full indexing when creating backups. For this, each package would need to be downloaded, unpacked, and indexed. Therefore, hashget uses a scheme with . Upon detecting an installed Debian package, if it is not found in local HashPackage, a simple attempt is made to download HashPackage from the hash server first. Only if that fails does hashget download and hash the package itself (and uploads it to the hash server for future provision).
HashServer is not a mandatory element of the scheme; it is not critical and is solely used to speed up and reduce the load on repositories. It can be easily disabled (with the option --hashserver without parameters). Additionally, it is straightforward to .
Incremental and differential backups, scheduled obsolescence
allow for a very simple scheme of Why not index our own backup (with all our unique files)? One command --submit And everything is ready! The next backup that hashget creates will not include files from this archive.
However, this is not a very good approach, because it may turn out that during restoration we will have to pull all hashget backups from the entire history (if each contains at least one unique file). For this, there is a mechanism . When indexing, you can specify the expiration date for HashPackage --expires 2019-06-01, and upon reaching this date (at 00:00), it will not be used. The archive itself does not need to be deleted after this date (although hashget can conveniently show the URLs of all backups that are currently expired or will expire on any date).
For example, if a full backup is made on the 1st, and indexed with a lifespan until the end of the month — we will get a differential backup scheme.
If we also index the new backups — it will be an incremental backup scheme.
Unlike traditional schemes, hashget allows the use of multiple base sources. The backup will be reduced by cutting files from previous backups (if any), as well as from public files (the ones that can be downloaded).
If for some reason we do not trust the reliability of Debian resources () or use another distribution, we can simply do a full backup with all packages once, and then rely on it (). Now, if all the servers of our distributions become inaccessible (in a souvenir Internet or during a zombie apocalypse), but our backups are fine — we can restore from any short differential backup that relies only on our earlier backups.
Hashget relies only on reliable recovery sources at YOUR discretion. The ones you consider reliable will be used.
FilePool and Glacier
The mechanism allows you not to constantly contact external servers to download packages, but to use packages from a local directory or a corporate server, for example:
$ hashget -u . --pool /tmp/poolor
$ hashget -u . --pool http://myhashdb.example.com/To create a pool in a local directory, simply create a directory and place files in it; hashget will find what it needs based on the hashes. To make the pool accessible via HTTP, you need to create symbolic links in a specific manner, which can be done with one command (hashget-admin --build /var/www/html/hashdb/ --pool /tmp/pool). The HTTP FilePool consists of static files, so it can be served by any simple web server, with virtually no load on the server.
Thanks to FilePool, you can use not only resources on http(s) as basic resources, but also, , Amazon Glacier.
After uploading a backup to Glacier, you receive its Upload ID and use that as the URL. For example:
hashget --submit Glacier_Upload_ID --file /tmp/my-glacier-backup.tar.gz --project glacier --hashserver --expires 2019-09-01Now new (differential) backups will rely on this backup and be shorter. After unpacking the differential backup with tar, we can see which resources it depends on:
hashget --info /tmp/unpacked/ listand just use a shell script to download all these files from Glacier into the pool and run a normal recovery: hashget -u /tmp/unpacked --pool /tmp/pool
Is it worth it?
In the simplest case, you will simply pay less for backups (if you store them somewhere in the cloud for a fee). It could be significantly less.
But that's not the only benefit. Quantity turns into quality. You can use this to achieve a qualitative upgrade in your backup scheme. For example, since our backups are now shorter, you can perform daily backups instead of monthly ones. Store them for 5 years instead of the previous 6 months. Previously, backups were stored in slow but cheap 'cold' storage (Glacier), now you can store them in hot storage from where you can quickly download a backup and recover in minutes rather than a day.
You can increase the reliability of storing backups. If we currently store them in one repository, by reducing the backup volume, we can store them in 2-3 repositories and easily withstand it if one of them is damaged.
How to try it and start using it?
Go to the GitLab page , install with one command (pip3 install hashget[plugins]) and just read-follow the quick-start. I think doing all the simple tasks will take about 10-15 minutes. Then you can try compressing your virtual machines, create hint files if needed for further compression, play around with pools, the local hash database, and the hash server if you're interested, and the next day check the size of the incremental backup on top of yesterday's.
Source: habr.com
