Data classification is an interesting research topic in itself. I love collecting information that seems necessary, and I've always tried to create logical hierarchies of directories for my files. One night, I had a dream about a beautiful and convenient program for assigning tags to files, and I decided that I could no longer live this way.

The Problem of Hierarchical File Systems

Users often face the problem of choosing where to save a new file and the challenge of finding their own files (sometimes file names are not meant to be easily remembered by people).

The solution could be semantic file systems, which are typically an extension of traditional file systems. In these systems, directories are replaced with semantic attributes, also known as tags, categories, or metadata. I will often use the term 'category' since, in the context of file systems, the word 'tag' can sometimes feel odd, especially when there are 'sub-tags' and 'tag aliases.'

Assigning categories to files largely alleviates the issues of storage location and file retrieval: if you remember (or can guess) at least one of the categories assigned to a file, the file will never go missing.

This topic has been raised multiple times on Habr (one, two, three, four and others), here I describe my solution.

Path to Implementation

Right after the mentioned dream, I sketched out a command interface in a notebook to handle categories. I thought that within a week or two, I could write a prototype using Python or Bash, and later work on creating a graphical interface with Qt or GTK. Reality, as always, turned out to be much harsher, and the development took longer than expected.

The initial idea was to create a program with a user-friendly and concise command-line interface that would allow users to create and delete categories, assign categories to files, and remove categories from files. I named the program vitis.

The First Attempt to Create vitis ended up going nowhere, as a lot of time was spent on work and school. The second attempt was something: I managed to complete the intended project for my master's thesis and even create a GTK shell prototype. However, that version turned out to be so unreliable and inconvenient that I had to rethink many things.

I actually used the third version for a long time, categorizing several thousand of my files. This was greatly aided by the implemented bash autocompletion. However, some issues remained, such as the absence of automatic categories and the ability to store files with the same name, and the program was already struggling under its own complexity. Thus, I felt the need to address the challenges of developing complex software: writing detailed requirements, developing a functional testing system, studying packaging instructions, and much more. Now I have reached my goal, and this modest creation can be presented to the free community. Such a specific file management method, like managing through the concept of categories, raises unexpected questions and problems, and in addressing them vitis it spawned five more projects, some of which will be mentioned in the article. To this day vitis I haven't acquired a graphical interface, but the convenience of using file categories from the command line already outweighs any advantages of a typical graphical file manager.

Examples of usage

Let's start simple — we'll create a category:

vitis create Music

We'll add a piece of music for example:

vitis assign Music -f "The Ink Spots - I Don't Want To Set The World On Fire.mp3"

You can view the contents of the 'Music' category with the 'show' subcommand:

vitis show Music

You can play it using the 'open' subcommand

vitis open Music

Since we have only one file in the 'Music' category, only that one will launch. For the purpose of opening files with default programs, I created a separate utility vts-fs-open (Standard tools like xdg-open or mimeopen didn't suit me for various reasons; however, if needed, you can specify another utility for universal file opening in the settings). This utility works well on various distributions with different desktop environments, so I recommend installing it along with vitis.

You can also directly specify a program to open files:

vitis open Music --app qmmp

Let's create additional categories and add files using 'assign'. If files are assigned to non-existing categories, a request to create them will be issued. You can avoid an unnecessary request by using the --yes flag.

vitis assign Programming R -f "Introduction to R.pdf" "Statistical Package R: Probability Theory and Mathematical Statistics.pdf" --yes

Now we want to add the category 'Mathematics' to the file 'Statistical Package R: Probability Theory and Mathematical Statistics.pdf'. We know that this file already has the category 'R', so we can use the category path from the Vitis system:

vitis assign Mathematics -v "R/Statistical Package R: Probability Theory and Mathematical Statistics.pdf"

Fortunately, bash autocompletion will make this easy.

Let's see what we've got by using the --categories flag to display the list of categories for each file:

vitis show R --categories

Note that files also have been automatically assigned categories based on format, type (which combines formats), and file extensions. These categories can be disabled if desired. I will definitely localize their names later.

Let's add something else to 'Mathematics' for variety:

vitis assign Mathematics -f "Mathematical Analysis - 1984.pdf" Perelman_Entertaining_Mathematics_1927.djvu 

Now things get interesting. Instead of categories, you can write expressions with union, intersection, and subtraction operations, that is, use set operations. For example, the intersection of 'Mathematics' with 'R' will result in one file.

vitis show R i: Mathematics

Let's subtract mentions of the language 'R' from 'Mathematics':

vitis show Mathematics R  #or vitis show Mathematics c: R

We can randomly unite music and language R:

vitis show Music u: R

The -n flag allows 'pulling out' the required files from the query results by numbers and/or ranges, for example, -n 3-7, or something more complex: -n 1,5,8-10,13. It is often useful with the open sub-command, which allows opening the necessary files from the list.

Although we are moving away from using conventional directory hierarchies, it's often useful to have nested categories. Let's create a subcategory "Statistics" under the category "Mathematics" and add this category to the appropriate file:

vitis create Mathematics/Statistics

vitis assign Mathematics/Statistics -v "R/Introduction to R.pdf"

vitis show Mathematics --categories

We can see that this file now has the category "Mathematics/Statistics" instead of "Mathematics" (unnecessary links are tracked).

Referring to the full path can be inconvenient, so let's create a "global" alias:

vitis assign Mathematics/Statistics -a Statistics

vitis show Statistics

Not just regular files

Internet links

To unify the storage of any information, it would be useful to at least categorize links to Internet resources. And this is possible:

vitis assign Habr ColorAnomaly -i https://habr.com/en/company/sfe_ru/blog/437304/ --yes

A file with the HTML page title and a .desktop extension will be created in a special location. This is the traditional shortcut format in GNU/Linux. Such shortcuts receive an automatic category of NetworkBookmarks.

Naturally, shortcuts are created to be used:

vitis open ColorAnomaly

Executing the command opens the just-saved link in the browser. Categorized shortcuts to Internet sources can serve as a replacement for browser bookmarks.

File fragments

It is also useful to have categories for individual file fragments. Not a bad proposition, right? But the current implementation only covers regular text files, audio, and video files for now. Let's say you need to mark a specific piece of a concert or a funny moment in a movie; when using assign, you can take advantage of the flags —fragname, —start, —finish. Let’s save the title sequence from "DuckTales":

vitis assign vitis assign -c Titles -f Duck_Tales/s01s01.avi --finish 00:00:59 --fragname "Duck Tales intro"

vitis open Titles

In reality, no actual file truncation occurs; instead, a pointer file is created to a fragment, which describes the file type, file path, start, and end of the fragment. The creation and opening of pointers to fragments are delegated to utilities that I specifically made for these purposes — mediafragmenter and fragplayer. The first creates, and the second opens. For audio and video recordings, playback of the media file from a specific position to a certain point is done using the VLC player, so it must also be present on the system. I initially wanted to do this based on mplayer, but for some reason, it was very awkward with positioning at the required moment.
In our example, a file named "Duck Tales intro.fragpointer" is created (it is placed in a special location), and then the fragment is played from the beginning of the file (since —start was not specified during creation) to the mark at 59 seconds, after which VLC closes.

Another example — we decided to categorize a specific performance at a concert of some famous artist:

vitis assign Lepс "Save Our Souls" -f Grigory Leps - Concert Sail - songs of Vladimir Vysotsky.mp4 --fragname "Save Our Souls" --start 00:32:18 --finish 00:36:51

vitis open "Save Our Souls"

When opened, the file will be included at the required position and will close after four and a half minutes.

How it all works + additional features

Storage of categories

At the very beginning of planning the organization of the semantic file system, three methods came to mind: through storing symbolic links, via a database, and through XML description. The first method won, as it is simple to implement and allows the user to view categories directly from the file system (which is convenient and important). At the beginning of usage vitis a directory named "Vitis" is created in the user's home directory along with a configuration file ".config/vitis/vitis.conf". In ~/Vitis, directories corresponding to categories are created, and in these category directories, symbolic links to the original files are placed. Category aliases are also just links to them. Of course, having a "Vitis" directory in the home directory may not suit someone. We can switch to any other location:

vitis service set path /mnt/MyFavoriteDisk/Vitis/

At some point, it becomes clear that categorizing files scattered across different locations is somewhat pointless, as their placement may change. So, initially, I created a directory where I simply dumped everything and assigned categories to them. Then, I thought it would be nice to formalize this at the program level. This is how the concept of 'filespace' came into being. At the beginning of its use, vitis it wouldn't hurt to set up such a place (where all the files we need will be stored) and enable autosave:

vitis service add filespace /mnt/MyFavoriteDisk/Filespace/
vitis service set autosave yes

Without autosave, using the 'assign' subcommand will require the flag --save if you want to save the added file to the filespace.

Moreover, you can add multiple filespaces and change their priorities, which can be useful when there are many files stored on different media. I won't be discussing this option here; you can read more about it in the program's help.

Migrating the semantic filesystem

One way or another, the Vitis directory and filespaces can theoretically move from place to place. To restore functionality, I created a separate utility link-editor, which can batch edit links, replacing parts of the path with others:

cp -r /mnt/MyFavoriteDisk/Vitis/ ~/Vitis
link-editor -d ~/Vitis/ -f /mnt/MyFavoriteDisk/Vitis/ -r ~/Vitis/ -R
cp -r /mnt/MyFavoriteDisk/Filespace/ ~/MyFiles
link-editor -d ~/Vitis/ -f /mnt/FlashDrive-256/Filespace/ -r ~/MyFiles -R

In the first case, after we moved from /mnt/MyFavoriteDisk/Vitis/ to the home directory, symbolic links related to aliases are edited. In the second case, after changing the location of the filespace, all links in Vitis are updated to the new ones according to the request to replace part of their path.

Automatic categories

If you run the command vitis service get autocategorization, you'll see that, by default, automatic category assignment is set up according to the format (Format and Type) and file extension (Extension).

This is useful when, for example, you need to find something among PDFs or check what you have stored as EPUB and FB2; you can simply execute the request

vitis show Format/MOBI u: Format/FB2

It so happened that standard GNU/Linux tools like file or mimetype did not satisfy me precisely because they do not always correctly determine the format, which led me to implement my own solution based on file signatures and extensions. In general, the topic of file format detection is an interesting area for research and deserves a separate article. For now, I can say that while I might not have accounted for accurate recognition for all formats in the world, it is already working reasonably well. However, EPUB currently defines the format as ZIP (which is generally justified, but in practice, this should not be considered normal behavior). For a while, consider this feature experimental; please report any bugs. In unusual situations, you can always use categories based on file extensions, for example, Extension/epub.

If automatic categories by format are enabled, automatic categories combining certain formats like: "Archives", "Pictures", "Video", "Audio", and "Documents" are also included. Localized names will also be created for these subcategories.

What is not mentioned

vitis turned out to be a very versatile tool, and it’s hard to cover everything at once. I will briefly mention what else can be done:

• categories can be deleted and removed from files;
• search results based on expressions can be copied to a specified directory;
• files can be run as programs;
• the show command has many options, such as sorting by name/date modified or by size/extension, displaying properties of files and paths to originals, including hidden files display, etc.;
• when saving links to online sources, local copies of HTML pages can also be saved.

All details can be found in the user guide.

Prospects

Skeptics often say that "nobody will set these tags themselves." From my own experience, I can prove the opposite: I have already categorized over six thousand files, created over a thousand categories and aliases, and it was worth it. When using a single command vitis open Plan you open your to-do list or when using a single command vitis open LaTeX you open Stolyarov's book about the LaTeX markup system, it becomes morally challenging to use the file system "the old-fashioned way."

This gives rise to several ideas. For instance, one could create an automatic radio that plays thematic music according to the current weather, holiday, day of the week, time of day, or season. Another related idea is a music player that understands categories and can play music based on expressions involving operations on those categories as if they were sets. It would also be useful to create a daemon that monitors the Downloads folder and suggests categorizing new files. And of course, a proper graphical semantic file manager should be developed. At one point, I even created a web service for an organization for collective file sharing, but it was not a priority and became outdated, although it achieved a high level of functionality. (Due to major changes in itself, vitis, it is no longer usable.)

here is a small demonstration

Conclusion

Vitis — not the first attempt to radically change the way we work with data, but I found it important to implement my ideas and release the implementation as open source under the GNU GPL license. For convenience, a deb package has been created for x86-64, which should work on all modern Debian distributions. There were some minor issues on ARM (although all other programs related to vitis, work fine), but a working package for this platform (armhf) will also be prepared in the future. I have stopped creating RPM packages for now due to issues with Fedora 30 and the difficulty of spreading across multiple RPM distributions, but packages will still be made for at least a couple of them later. In the meantime, you can use make && make install or checkinstall.

Thank you all for your attention! I hope this article and project can prove useful.

Link to the project repository

Source: habr.com