As you know, if trouble can happen, then it will definitely happen. Probably, everyone had cases when a fresh important file was accidentally deleted, or text was accidentally selected and destroyed in a text editor.
If you are a hoster or a site owner, then you have probably experienced hacking of user accounts or your site. In such cases, it is important to restore the chronology, find the penetration method and the vulnerability that the attacker used.
To solve such problems, the NILFS2 file system is perfect.
It has been present in the Linux kernel since version 2.6.30.
The peculiarity of this file system is that it is similar to a version control system: you can always roll back the state of the system and look at what it was some time ago.
To provide this functionality, you do not need to set up Cron scripts, take snapshots, etc. The NILFS2 file system does this all by itself. It never overwrites old data and always writes to new disk areas if there is enough free disk space. In full compliance with the principle of Copy-on-Write.
In fact, any change to a file will automatically create a new snapshot of the file system, so you can use this FS as a time machine and rewind the state of the files.
History
NILFS2 was developed from the ground , in fact, the state (it has a controlling stake) and the largest telecommunications company in Japan. More specifically, in the CyberSpace Laboratories under the direction of .
What exactly it was developed for is unknown, however, it can be assumed that such a file system, with its βtime machineβ functionality, is ideal for storing data in which there may be a desire to dig deeper into the special services in order to re-play the whole picture of SMS, emails, etc ...
NILFS2 is also potentially a very valuable tool for internal security services, as it allows you to recover all deleted messages in the mail database, open jambs of employees who may later try to disguise them by deleting or changing their files.
How can I track the entire history of correspondenceOn Linux servers (namely, where NILFS2 should be installed for internal security purposes), the file method of storing e-mails is very often used to store mail messages. The so-called format . Enough to put and configure the storage of letters in Maildir. Other format is a large text file that can be easily parsed into individual messages.
If the mail server uses a database, then NILFS2 will make it possible to restore the exact timing of database changes and the ability to restore the database to any of these moments. And then you need to use the database tools to see what was in it at that time ...
However, something went wrong. Either the Japanese government changed its mind about keeping track of everyone (a la the Yarovaya principle), or the performance of NILFS2 on traditional HDDs turned out to be below the plinth, and NILFS2 was released under the GPL license and very quickly entered the Linux kernel, since there were no special claims to the code written highly skilled Japanese, the developers of the Linux kernel did not have.
What is NILFS2 like?
From the point of view of use: on the version control system . Each FS checkpoint is a commit that is made automatically without the knowledge of the user for any change: be it deletion, changing the contents of a file or changing access rights. Each commit has a number that increases linearly.
From a programmer's point of view: on a circular buffer. The file system accumulates changes and writes them into a piece equal to approximately 8 MB (2000 * 4096, where 2000 is the number of elements in the block, and 4096 is the size of the memory page). The entire disk is divided into such chunks. The recording is sequential. When free space runs out, the oldest snapshots are deleted and the chunks are overwritten.
Main perks of NILFS2
- Versioning!!!
- The procedure for restoring a file system after a failure is elementary: when loading, the last chunk is searched for with the correct checksum, and a superblock is installed on it. It's almost an instant operation.
- Due to the fact that the recording always goes linearly, then:
- may perform well when running on an SSD, with slow random writes.
- NILFS2 saves SSD resource as there is almost no write multiplier factor.
More precisely, it is not more than 2.The fact is that when cyclically overwriting the entire disk, NILFS2 will transfer immutable data to new pieces (chunks).If we have 10% unchanged data on the disk, then we will get a 10% write increase with 1 complete overwrite. Well, a 50% increase at 50% fullness of the device for 1 complete overwriting of the disk.
The maximum recording amplification factor is 2. This is very small, given that everything is written sequentially. In general, the write multiplication will be less than that of a conventional fragmented FS with a 4096 byte sector. (Indicated ).
- Potential ease of implementation of replication to a remote NILFS2 FS
NILFS2 for /home
In Unix-like operating systems, as a rule, there is a /home folder in which user data is stored. Various programs save their settings related to a specific user in this folder.
And who, if not users, most often mows? Therefore, as they say, God himself ordered to use NILFS2 on /home.
Moreover, with the ubiquity of SSDs, we no longer have to worry about a strong drawdown when using CoW file systems.
Yes, we can create FS snapshots (snapshots) as often as we like in ZFS and BTRFS, but there is always a risk that a lost file change will end up between snapshots. And the pictures still need to be administered: delete the old ones. In NILFS2, all this happens automatically, literally every few seconds.
I created a logical volume with lvcreate (on nvme volume group, thin pool). I recommend creating it on the lvm volume, as it can be easily expanded later. I recommend having 50% free disk space with NILFS2 for decent versioning depth.
lvcreate -V10G -T nvme/thin -n homeand formatted it to NILFS2:
mkfs.nilfs2 -L nvme_home /dev/nvme/home
mkfs.nilfs2 (nilfs-utils 2.1.5)
Start writing file system initial data to the device
Blocksize:4096 Device:/dev/nvme/home1 Device Size:10737418240
File system initialization succeeded !!
After that, you need to copy all the data from the current / home.
I did this right after booting up the computer, before logging into my account as root. If I were to log in as my user, then some programs would open sockets and files in my user's /home/user folder, which would make a clean copy difficult. As you know, the home folder for the root user is usually located at the /root path, so no files will open on the /home partition.
mkdir /mnt/newhome
mount -t nilfs2 /dev/nvme/home /mnt/newhome
cp -a /home/. /mnt/newhome
For the last line, see .
Next, we edit /etc/fstab, in which the file system for /home is mounted, on
/dev/disk/by-label/nvme_home /home nilfs2 noatime 0 0
Option noatime is needed to improve performance so that atime does not change with each file access. Next, we reboot.
Types of snapshots in NILFS2.
A regular snapshot without immunity to deletion is called a checkpoint (checkpoint or restore point).
A snapshot with auto-delete protection is called a snapshot, hereinafter simply a snapshot.
Viewing checkpoints is done using the lscp command
View snapshots lscp -s
We can create snapshots and checkpoints ourselves at any time using:
mkcp [-s] ΡΡΡΡΠΎΠΉΡΡΠ²ΠΎRestoring data.
NILFS allows us to mount as many old snapshots as we want while working with the main branch of the file system. But only in read mode.
It's all set up like that. The usual checkpoints that NILFS2 makes can be automatically deleted by it at any time (when disk space runs out or according to the rules of nilfs_cleanerd), so before editing, we must convert the checkpoint to a snapshot or, in Russian speaking, fix the snapshot.
chcp ss Π½ΠΎΠΌΠ΅Ρ_ΡΠ΅ΠΊΠΏΠΎΠΈΠ½ΡΠ°After that, we can mount the snapshot, for example, like this:
mount -t nilfs2 -r -o cp=Π½ΠΎΠΌΠ΅Ρ_ΡΠ΅ΠΊΠΏΠΎΠΈΠ½ΡΠ° /dev/nvme/home /mnt/nilfs/Π½ΠΎΠΌΠ΅Ρ_ΡΠ΅ΠΊΠΏΠΎΠΈΠ½ΡΠ°
Then we copy the recoverable files from the snapshot to /home.
And then we remove the non-removable flag from the snapshot so that in the future the automatic garbage collector can remove obsolete data:
chcp cp Π½ΠΎΠΌΠ΅Ρ_ΡΠ΅ΠΊΠΏΠΎΠΈΠ½ΡΠ°Utilities for NILFS2
But this is the trouble. Yes, of course, we can create a file system, resize it online, view the list of checkpoints, make and delete them. The nilfs2-utils package provides a minimal gentleman's kit.
Since NTT has cut funding, there are no fast low-level utilities that allow you to display the history of file changes, do diff between snapshots.
My n2u utility
To fill this vacuum I wrote , which can display the history of changes to a specific file/directory:
n2u log filenameThe output is something like this:
CHECKPOINT DATE TIME TYPE SIZE MODE
1787552 2019-11-24 22:08:00 first 7079 cp
1792659 2019-11-25 23:09:05 changed 7081 cpIt works pretty fast for the chosen way of implementation: it looks for differences between files by bisecting, quickly mounting and comparing a file/directory in different snapshots.
You can set a range of checkpoints using the key -cp CP1:CP2 or -cp {YEAR-MM-DD}:{YEAR-MM-DD}.
You can also see the difference between checkpoints for a particular file or directory:
n2u diff -r cp1:cp2 filenameYou can display the entire history of changes: all the differences between the checkpoints of a certain file/directory:
n2u blame [-r cp1:cp2] filenameDate interval in this command is also supported.
Call to developers
There are many specialists on HabrΓ©. Please finish NILFS2. Make replication, low-level quick diff between revisions, reflink and other goodies!
references
.
Repositories:
.
.
Newsletters:
. Identifier for the linux-nilfs subscription.
.
.
.
Acknowledgments:
- NILFS2 Developers: Ryusuke Konishi, Koji Sato, Naruhiko Kamimura, Seiji Kihara, Yoshiji Amagai, Hisashi Hifumi and Satoshi Moriai. Other major contributors are: Andreas Rohner, Dan McGee, David Arendt, David Smid, dexen deVries, Dmitry Smirnov, Eric Sandeen, Jiro SEKIBA, Matteo Frigo, Hitoshi Mitake, Takashi Iwai, Vyacheslav Dubeyko.
- To Amblin Entertainment and Universal Pictures for a wonderful series of films . The first picture of the post is taken from the movie "Back to the Future - 3".
- Company for support and the opportunity to publish on his blog on HabrΓ©.
PS Please report any errors you find. I raise my karma for this.
You can experiment with NILFS2 by ordering a virtual machine from coupon below. For all new customers, a free trial period of 3 days.
Source: habr.com