🥇Storacle — decentralized file storage

Before I start, I need to leave a link to the previous article, so it's clear what exactly we're talking about.

In this article, I would like to discuss the layer responsible for file storage and how it can be used by anyone. Storacle is a standalone library, with no direct connection to music. It can organize the storage of any files.

In the previous article, I somewhat "dissed" ipfs, but this happened precisely in the context of the task I was addressing. Overall, I think this project is cool. I just prefer the ability to create different networks for different tasks. This allows for better organization of the structure and reduces the load on individual nodes and the network as a whole. You can even break down a network into pieces according to certain criteria, reducing the overall load within a single project if necessary.

So, storacle uses a mechanism spreadable to organize the network. Key features include:

Files can be added to the storage through any node.
Files are saved in their entirety, not in blocks.
Each file has its own unique hash based on its content for further handling.
Files can be duplicated for greater reliability.
The number of files on a single node is only limited by the file system (there is an exception, which will be discussed below).
The number of files in the network is limited by the capabilities of spreadable regarding the allowable number of nodes in the network, which in the second version will allow working with an infinite number of nodes (about this in another article).

A simple example of how this works from the program:

Server:

const  Node = require('storacle').Node;

(async () => {
  try {
    const node = new Node({
      port: 4000,
      hostname: 'localhost'
    });
    await node.init();
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

Client:

const  Client = require('storacle').Client;

(async () => {
  try {
    const client = new  Client({
      address: 'localhost:4000'
    });
    await client.init();
    const hash = await client.storeFile('./my-file');
    const link = await client.getFileLink(hash);
    await client.removeFile(hash);
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

An inside look

There's nothing supernatural going on under the hood. Information about the number of files, their total size, and other aspects is stored in an in-memory database and is updated upon deletion and addition of files, so frequent access to the file system is unnecessary. The only exception is when the garbage collector is enabled, which requires the circulation of files once certain storage sizes are reached, rather than prohibiting the addition of new files. In this case, it is necessary to traverse the storage, and working with a large number of files (> a million, for example) can lead to significant loads. It's better to store fewer files and run more nodes. If the 'cleaner' is disabled, there are no such issues.

The file storage consists of 256 folders and 2 levels of nesting. Files are stored in second-level folders. Thus, with 1 million files, there would be about 62,500 files in each folder (1,000,000 / sqrt(256)).

The names of the folders are generated from the file's hash to allow for quick access once the hash is known.

This structure was chosen based on a large number of various requirements for the storage: support for weak file systems where having many files in one folder is undesirable, quick traversal of all folders when needed, etc. It's a sort of golden mean.

Caching

When adding files and retrieving them, links to the files are written to the cache.
Thanks to this, there is often no need to traverse the entire network in search of a file. This accelerates link retrieval and reduces network load. Caching also occurs through HTTP headers.

Isomorphism

The client is written in JavaScript and isomorphic, allowing it to be used directly from the browser.
You can upload a file https://github.com/ortexx/storacle/blob/master/dist/storacle.client.js as a script and gain access to window.ClientStoracle or import it through a build system, etc.

Deferred Links

An interesting feature is the 'deferred link.' This is a link to a file that can be obtained synchronously, here and now, while the file will be pulled once it's found in the storage. This is very convenient, for instance, when you need to display some images on a website. You simply set the deferred link in the src and that's it. Many use cases can be imagined.

Client API

async Client.prototype.storeFile() — saving a file
async Client.prototype.getFileLink() — obtaining a direct link to a file
async Client.prototype.getFileLinks() — retrieving a list of direct links to the file from all nodes where it exists
async Client.prototype.getFileToBuffer() — get the file into a buffer
async Client.prototype.getFileToPath() — get the file into the file system
async Client.prototype.getFileToBlob() — get the file as a blob (for the browser version)
async Client.prototype.removeFile() — delete file
Client.prototype.createRequestedFileLink() — create a deferred link

Exporting files to another server

To transfer files to another node, you can:

Simply copy the entire storage folder along with the settings. (this may not work in the future)
Copy only the folder with the files. However, in this case, you will need to run the function node.normalizeFilesInfo(), to recalculate all data and enter it into the database.
Use the function node.exportFiles(), which will start copying the files.

Main node settings
When starting the storage node, you can specify all necessary settings.
I will describe the most basic ones; the rest can be found on GitHub.

storage.dataSize — size of the folder with files
storage.tempSize — size of the temporary folder
storage.autoCleanSize — minimum size of storage that needs to be maintained. If this parameter is specified, as soon as space starts to run low, the least-used files will be deleted.
file.maxSize — maximum file size
file.minSize — minimum file size
file.preferredDuplicates — preferred number of file duplicates in the network
file.mimeWhitelist — allowed file types
file.mimeBlacklist — disallowed file types
file.extWhitelist — allowed file extensions
file.extBlacklist — disallowed file extensions
file.linkCache — various link caching settings

Almost all parameters related to sizes can be specified in both absolute and relative values.

Working through the command line
The library can be used through the command line. To do this, it needs to be installed globally: npm i -g storacle. After this, you can execute the necessary actions from the directory with the project where the node is. For example, storacle -a storeFile -f ./file.txt -c ./config.js, to add a file. All actions can be found in https://github.com/ortexx/storacle/blob/master/bin/actions.js

Why you might need this

If you want to create some decentralized project in which it is planned to store and work with files using convenient methods. For example, the music project described at the link at the beginning of the article uses storacle.
If you are working on any other projects that require distributed file storage, you can easily set up your private network, flexibly configure nodes, and add new ones as needed.
If you simply need a place to store your website files and don’t want to write everything yourself, this library might be a better fit for you.
If you have a project where you're working with files but want to perform all operations from the browser, you can avoid writing server-side code.

My contacts:

Source: habr.com

Storacle — a decentralized file storage