{"id":35687,"date":"2019-10-31T22:05:46","date_gmt":"2019-10-31T19:05:46","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kategorii-vmesto-direktorij-ili-semanticheskaya-fajlovaya-sistema-dlya-linux\/"},"modified":"2019-10-31T22:05:46","modified_gmt":"2019-10-31T19:05:46","slug":"kategorii-vmesto-direktorij-ili-semanticheskaya-fajlovaya-sistema-dlya-linux","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kategorii-vmesto-direktorij-ili-semanticheskaya-fajlovaya-sistema-dlya-linux","title":{"rendered":"Categories Instead of Directories, or Semantic File System for Linux","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Data classification is an interesting research topic in itself. I love collecting information that seems necessary, and I've always tried to create logical hierarchies of directories for my files. One night, I had a dream about a beautiful and convenient program for assigning tags to files, and I decided that I could no longer live this way.<\/p>\n

<\/a><\/noindex><\/p>\n

The Problem of Hierarchical File Systems<\/h1>\n

<\/p>\n

Users often face the problem of choosing where to save a new file and the challenge of finding their own files (sometimes file names are not meant to be easily remembered by people).<\/p>\n

<\/p>\n

A solution might be semantic file systems, which are typically an extension of traditional file systems. In them, directories are replaced with semantic attributes, also known as tags, categories, and metadata. I will more frequently use the term 'category' as the word 'tag' can sometimes sound odd in the context of file systems, especially when 'subtags' and 'tag aliases' come into play.<\/p>\n

<\/p>\n

Assigning categories to files largely alleviates the issues of storage location and file retrieval: if you remember (or can guess) at least one of the categories assigned to a file, the file will never go missing.<\/p>\n

<\/p>\n

This topic has been raised multiple times on Habr (one<\/a><\/noindex>, two<\/a><\/noindex>, three<\/a><\/noindex>, four<\/a><\/noindex> and others), here I describe my solution.<\/p>\n

<\/p>\n

Path to Implementation<\/h1>\n

<\/p>\n

Right after the mentioned dream, I sketched out a command interface in a notebook to handle categories. I thought that within a week or two, I could write a prototype using Python or Bash, and later work on creating a graphical interface with Qt or GTK. Reality, as always, turned out to be much harsher, and the development took longer than expected.<\/p>\n

<\/p>\n

The initial idea was to create a program with a user-friendly and concise command-line interface that would allow users to create and delete categories, assign categories to files, and remove categories from files. I named the program vitis<\/em>.<\/p>\n

<\/p>\n

The First Attempt to Create vitis<\/em> ended up going nowhere, as a lot of time was spent on work and school. The second attempt was something: I managed to complete the intended project for my master's thesis and even create a GTK shell prototype. However, that version turned out to be so unreliable and inconvenient that I had to rethink many things.<\/p>\n

<\/p>\n

I actually used the third version for a long time, categorizing several thousand of my files. This was greatly aided by the implemented bash autocompletion. However, some issues remained, such as the absence of automatic categories and the ability to store files with the same name, and the program was already struggling under its own complexity. Thus, I felt the need to address the challenges of developing complex software: writing detailed requirements, developing a functional testing system, studying packaging instructions, and much more. Now I have reached my goal, and this modest creation can be presented to the free community. Such a specific file management method, like managing through the concept of categories, raises unexpected questions and problems, and in addressing them vitis<\/em> it spawned five more projects, some of which will be mentioned in the article. To this day vitis<\/em> I haven't acquired a graphical interface, but the convenience of using file categories from the command line already outweighs any advantages of a typical graphical file manager.<\/p>\n

<\/p>\n

Examples of usage<\/h1>\n

<\/p>\n

Let's start simple \u2014 we'll create a category:<\/p>\n

<\/p>\n

vitis create Music<\/code><\/pre>\n
<\/p>\n
We'll add a piece of music for example:<\/p>\n
<\/p>\n
vitis assign Music -f \"The Ink Spots - I Don't Want To Set The World On Fire.mp3\"<\/code><\/pre>\n
<\/p>\n
You can view the contents of the category 'Music' using the command 'show':<\/p>\n
<\/p>\n
vitis show Music<\/code><\/pre>\n
<\/p>\n
You can play it using the command 'open'.<\/p>\n
<\/p>\n
vitis open Music<\/code><\/pre>\n
<\/p>\n
Since we have only one file in the 'Music' category, only it will be launched. For the purpose of opening files with default programs, I created a separate utility. vts-fs-open<\/em> (Standard tools like xdg-open or mimeopen didn't suit me for various reasons; however, if needed, you can specify another utility for universal file opening in the settings). This utility works well on various distributions with different desktop environments, so I recommend installing it along with vitis.<\/p>\n
<\/p>\n
You can also directly specify a program to open files:<\/p>\n
<\/p>\n
vitis open Music --app qmmp<\/code><\/pre>\n
<\/p>\n
\"Categories<\/p>\n
<\/p>\n
Let's create more categories and add files using 'assign'. If the files are assigned to non-existent categories, a request for their creation will be issued. You can avoid the extra request by using the flag \u2014yes.<\/p>\n
<\/p>\n
vitis assign Programming R -f \"Introduction to R.pdf\" \"Statistical Package R: Probability Theory and Mathematical Statistics.pdf\" --yes<\/code><\/pre>\n
<\/p>\n
Now we want to add the file 'Statistical Package R: Theory of Probability and Mathematical Statistics.pdf' to the category 'Mathematics'. We know that this file already has the category 'R', so we can use the category path from the Vitis system:<\/p>\n
<\/p>\n
vitis assign Mathematics -v \"R\/Statistical Package R: Probability Theory and Mathematical Statistics.pdf\"<\/code><\/pre>\n
<\/p>\n
Fortunately, bash autocompletion will make this easy.<\/p>\n
<\/p>\n
Let's see what we have, using the flag \u2014categories to see the list of categories for each file:<\/p>\n
<\/p>\n
vitis show R --categories<\/code><\/pre>\n
<\/p>\n
\"Categories<\/p>\n
<\/p>\n
Note that files also have been automatically assigned categories based on format, type (which combines formats), and file extensions. These categories can be disabled if desired. I will definitely localize their names later.<\/p>\n
<\/p>\n
Let's add something else to 'Mathematics' for variety:<\/p>\n
<\/p>\n
vitis assign Mathematics -f \"Mathematical Analysis - 1984.pdf\" Perelman_Entertaining_Mathematics_1927.djvu <\/code><\/pre>\n
<\/p>\n
Now it gets interesting. Instead of categories, you can write expressions with union, intersection, and subtraction operations, that is, use set operations. For example, the intersection of 'Mathematics' and 'R' will yield one file.<\/p>\n
<\/p>\n
vitis show R i: Mathematics<\/code><\/pre>\n
<\/p>\n
Let's subtract mentions of the language 'R' from 'Mathematics':<\/p>\n
<\/p>\n
vitis show Mathematics R  #or vitis show Mathematics c: R<\/code><\/pre>\n
<\/p>\n
We can randomly unite music and language R:<\/p>\n
<\/p>\n
vitis show Music u: R<\/code><\/pre>\n
<\/p>\n
The flag -n allows you to 'extract' the necessary files by numbers and\/or ranges from the query result, for example, -n 3-7<\/code>, or something more complex: -n 1,5,8-10,13<\/code>. It is often useful with the open sub-command, which allows opening the necessary files from the list.<\/p>\n
<\/p>\n
\"Categories<\/p>\n
<\/p>\n
Even though we are stepping away from using the conventional directory hierarchy, it is often useful to have nested categories. Let's create a subcategory 'Statistics' under the category 'Mathematics' and add this category to the appropriate file:<\/p>\n
<\/p>\n
vitis create Mathematics\/Statistics\n\nvitis assign Mathematics\/Statistics -v \"R\/Introduction to R.pdf\"\n\nvitis show Mathematics --categories<\/code><\/pre>\n
<\/p>\n
\"Categories<\/p>\n
<\/p>\n
We can see that this file now has the category 'Mathematics\/Statistics' instead of 'Mathematics' (unnecessary links are tracked).<\/p>\n
<\/p>\n
It can be inconvenient to refer to the full path, let's create a 'global' alias:<\/p>\n
<\/p>\n
vitis assign Mathematics\/Statistics -a Statistics\n\nvitis show Statistics<\/code><\/pre>\n
<\/p>\n
\"Categories<\/p>\n
<\/p>\n
Not just regular files<\/h2>\n
<\/p>\n
Internet links<\/h3>\n
<\/p>\n
To unify the storage of any information, it would be useful to at least categorize links to Internet resources. And this is possible:<\/p>\n
<\/p>\n
vitis assign Habr ColorAnomaly -i https:\/\/habr.com\/en\/company\/sfe_ru\/blog\/437304\/ --yes<\/code><\/pre>\n
<\/p>\n
A file with the HTML page title and a .desktop extension will be created in a special location. This is the traditional shortcut format in GNU\/Linux. Such shortcuts receive an automatic category of NetworkBookmarks.<\/p>\n
<\/p>\n
Naturally, shortcuts are created to be used:<\/p>\n
<\/p>\n
vitis open ColorAnomaly<\/code><\/pre>\n
<\/p>\n
Executing the command opens the just-saved link in the browser. Categorized shortcuts to Internet sources can serve as a replacement for browser bookmarks.<\/p>\n
<\/p>\n
File fragments<\/h3>\n
<\/p>\n
It's also useful to have categories for separate file segments. Not a bad request, right? But the current implementation only affects regular text files, audio, and video files. Suppose you need to mark a specific part of a concert or a funny moment in a movie, then when using assign, you can use the flags \u2014fragname, \u2014start, \u2014finish. Let's save the intro from 'Duck Tales':<\/p>\n
<\/p>\n
vitis assign vitis assign -c Titles -f Duck_Tales\/s01s01.avi --finish 00:00:59 --fragname \"Duck Tales intro\"\n\nvitis open Titles<\/code><\/pre>\n
<\/p>\n
In reality, no files are cut; instead, a pointer file to the fragment is created, which describes the file type, file path, starting and ending points of the fragment. The creation and opening of fragment pointers are delegated to utilities I specifically made for these purposes \u2014 mediafragmenter and fragplayer. The first creates, the second opens. In the case of audio and video recordings, playing the media file from a specific position to a certain point is done using the VLC player, so it must also be present on the system. I initially wanted to base this on mplayer, but for some reason, it was very unsatisfactory with positioning at the right moment.
\nIn our example, a file 'Duck Tales intro.fragpointer' is created (it is placed in a specific location), and then the fragment is played from the beginning of the file (since \u2014start was not specified during creation) to the mark at 59 seconds, after which VLC closes.<\/p>\n
<\/p>\n
Another example \u2014 we decided to categorize a specific performance at a concert of some famous artist:<\/p>\n
<\/p>\n
vitis assign Lep\u0441 \"Save Our Souls\" -f Grigory Leps - Concert Sail - songs of Vladimir Vysotsky.mp4 --fragname \"Save Our Souls\" --start 00:32:18 --finish 00:36:51\n\nvitis open \"Save Our Souls\"<\/code><\/pre>\n
<\/p>\n
When opened, the file will be included at the required position and will close after four and a half minutes.<\/p>\n
<\/p>\n
How it all works + additional features<\/h1>\n
<\/p>\n
Storage of categories<\/h2>\n
<\/p>\n
At the very beginning of planning the organization of the semantic file system, three methods came to mind: through storing symbolic links, via a database, and through XML description. The first method won, as it is simple to implement and allows the user to view categories directly from the file system (which is convenient and important). At the beginning of usage vitis<\/em> In the user's home directory, a directory 'Vitis' is created along with the configuration file '.config\/vitis\/vitis.conf'. In ~\/Vitis, directories corresponding to categories are created, and symbolic links to the original files are created in these category directories. Category aliases are also just links to them. Of course, having a 'Vitis' directory in the home folder may not suit everyone. We can switch to any other location:<\/p>\n
<\/p>\n
vitis service set path \/mnt\/MyFavoriteDisk\/Vitis\/<\/code><\/pre>\n
<\/p>\n
At a certain point, it becomes clear that categorizing files scattered across different locations is rather pointless, as their location may change. Therefore, I initially created a directory where I simply dropped everything and assigned categories to it. Then I thought it would be nice to formalize this concept at a programming level. Thus, the notion of 'file space' emerged. vitis<\/em> it wouldn't hurt to set up such a place (where all the files we need will be stored) and enable autosave:<\/p>\n
<\/p>\n
vitis service add filespace \/mnt\/MyFavoriteDisk\/Filespace\/\nvitis service set autosave yes<\/code><\/pre>\n
<\/p>\n
Without autosave when using the 'assign' subcommand, the --save flag will be required if you wish to save the added file in the file space.<\/p>\n
<\/p>\n
Moreover, you can add multiple filespaces and change their priorities, which can be useful when there are many files stored on different media. I won't be discussing this option here; you can read more about it in the program's help.<\/p>\n
<\/p>\n
Migrating the semantic filesystem<\/h2>\n
<\/p>\n
One way or another, the Vitis directory and filespaces can theoretically move from place to place. To restore functionality, I created a separate utility link-editor<\/em>, which can batch edit links, replacing parts of the path with others:<\/p>\n
<\/p>\n
cp -r \/mnt\/MyFavoriteDisk\/Vitis\/ ~\/Vitis\nlink-editor -d ~\/Vitis\/ -f \/mnt\/MyFavoriteDisk\/Vitis\/ -r ~\/Vitis\/ -R\ncp -r \/mnt\/MyFavoriteDisk\/Filespace\/ ~\/MyFiles\nlink-editor -d ~\/Vitis\/ -f \/mnt\/FlashDrive-256\/Filespace\/ -r ~\/MyFiles -R<\/code><\/pre>\n
<\/p>\n
In the first case, after we moved from \/mnt\/MyFavoriteDisk\/Vitis\/ to the home directory, symbolic links related to aliases are edited. In the second case, after changing the location of the filespace, all links in Vitis are updated to the new ones according to the request to replace part of their path.<\/p>\n
<\/p>\n
Automatic categories<\/h2>\n
<\/p>\n
If you run the command vitis service get autocategorization<\/code>, you'll see that, by default, automatic category assignment is set up according to the format (Format and Type) and file extension (Extension).<\/p>\n
<\/p>\n
This is useful when, for example, you need to find something among PDFs or check what you have stored as EPUB and FB2; you can simply execute the request<\/p>\n
<\/p>\n
vitis show Format\/MOBI u: Format\/FB2<\/code><\/pre>\n
<\/p>\n
It so happened that standard GNU\/Linux tools like file or mimetype did not satisfy me precisely because they do not always correctly determine the format, which led me to implement my own solution based on file signatures and extensions. In general, the topic of file format detection is an interesting area for research and deserves a separate article. For now, I can say that while I might not have accounted for accurate recognition for all formats in the world, it is already working reasonably well. However, EPUB currently defines the format as ZIP (which is generally justified, but in practice, this should not be considered normal behavior). For a while, consider this feature experimental; please report any bugs. In unusual situations, you can always use categories based on file extensions, for example, Extension\/epub.<\/p>\n
<\/p>\n
If auto-categories are enabled by format, auto-categories that merge some formats will also be included, such as: 'Archives', 'Pictures', 'Video', 'Audio', and 'Documents'. Localized names will also be created for these subcategories.<\/p>\n
<\/p>\n
What is not mentioned<\/h1>\n
<\/p>\n
vitis<\/em> turned out to be a very versatile tool, and it\u2019s hard to cover everything at once. I will briefly mention what else can be done:<\/p>\n
<\/p>\n