{"id":80224,"date":"2020-05-04T13:42:20","date_gmt":"2020-05-04T11:42:20","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov"},"modified":"2020-05-04T13:42:20","modified_gmt":"2020-05-04T11:42:20","slug":"storacle-deczentralizovannoe-hranilishhe-fajlov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov","title":{"rendered":"Storacle \u2014 a decentralized file storage","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Storacle \u2014 a decentralized file storage\" src=\"\/wp-content\/uploads\/2020\/05\/333f7b998e00ce32367d97f7ccd05bef.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Before I start, I need to leave <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/495058\/\">a link to the previous article<\/a><\/noindex>, so it's clear what exactly we're talking about.<\/p>\n<p><\/p>\n<p>In this article, I would like to discuss the layer responsible for file storage and how it can be used by anyone. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/ortexx\/storacle\/\">Storacle is a standalone library<\/a><\/noindex>, with no direct connection to music. It can organize the storage of any files.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>In the previous article, I 'vented' a bit on <strong>ipfs<\/strong>, but this happened precisely in the context of the task I was addressing. Overall, I think this project is cool. I just prefer the ability to create different networks for different tasks. This allows for better organization of the structure and reduces the load on individual nodes and the network as a whole. You can even break down a network into pieces according to certain criteria, reducing the overall load within a single project if necessary.<\/p>\n<p><\/p>\n<p>So, storacle uses a mechanism <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/ortexx\/spreadable\">spreadable<\/a><\/noindex> to organize the network. Key features include:<\/p>\n<p><\/p>\n<ul>\n<li>Files can be added to the storage through any node.<\/li>\n<li>Files are saved in their entirety, not in blocks.<\/li>\n<li>Each file has its own unique hash based on its content for further handling.<\/li>\n<li>Files can be duplicated for greater reliability.<\/li>\n<li>The number of files on a single node is only limited by the file system (there is an exception, which will be discussed below).<\/li>\n<li>The number of files in the network is limited by the capabilities of spreadable regarding the allowable number of nodes in the network, which in the second version will allow working with an infinite number of nodes (about this in another article).<\/li>\n<\/ul>\n<p><\/p>\n<p>A simple example of how this works from the program:<\/p>\n<p><\/p>\n<p>Server:<\/p>\n<p><\/p>\n<pre><code class=\"javascript\">const  Node = require('storacle').Node;\n\n(async () =&gt; {\n  try {\n    const node = new Node({\n      port: 4000,\n      hostname: 'localhost'\n    });\n    await node.init();\n  }\n  catch(err) {\n    console.error(err.stack);\n    process.exit(1);\n  }\n})();<\/code><\/pre>\n<p><\/p>\n<p>Client:<\/p>\n<p><\/p>\n<pre><code class=\"javascript\">const  Client = require('storacle').Client;\n\n(async () =&gt; {\n  try {\n    const client = new  Client({\n      address: 'localhost:4000'\n    });\n    await client.init();\n    const hash = await client.storeFile('.\/my-file');\n    const link = await client.getFileLink(hash);\n    await client.removeFile(hash);\n  }\n  catch(err) {\n    console.error(err.stack);\n    process.exit(1);\n  }\n})();<\/code><\/pre>\n<p><\/p>\n<p><strong>An inside look<\/strong><\/p>\n<p><\/p>\n<p>Under the hood, there's nothing supernatural. Information about the number of files, their total size, and other details is stored in an in-memory database and updated when files are added or removed, so there's no need for frequent access to the file system. The exception is when the garbage collector is enabled, which requires file circulation when certain storage thresholds are reached, rather than blocking the addition of new files. In this case, it\u2019s necessary to scan the storage, and working with a large number of files (over a million, let's say) can lead to significant loads. It's better to keep fewer files and run more nodes. If the 'cleaner' is disabled, this problem does not exist.<\/p>\n<p><\/p>\n<p>The file storage consists of 256 folders and 2 levels of nesting. Files are stored in second-level folders. Thus, with 1 million files, there would be about 62,500 files in each folder (1,000,000 \/ sqrt(256)).<\/p>\n<p><\/p>\n<p>The names of the folders are generated from the file's hash to allow for quick access once the hash is known.<\/p>\n<p><\/p>\n<p>This structure was chosen based on a large number of various requirements for the storage: support for weak file systems where having many files in one folder is undesirable, quick traversal of all folders when needed, etc. It's a sort of golden mean.<\/p>\n<p><\/p>\n<p><strong>Caching<\/strong><\/p>\n<p><\/p>\n<p>When adding files and retrieving them, links to the files are written to the cache.<br \/>\nThanks to this, there is often no need to traverse the entire network in search of a file. This accelerates link retrieval and reduces network load. Caching also occurs through HTTP headers.<\/p>\n<p><\/p>\n<p><strong>Isomorphism<\/strong><\/p>\n<p><\/p>\n<p>The client is written in JavaScript and isomorphic, allowing it to be used directly from the browser.\u00a0<br \/>\nYou can upload a file <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/ortexx\/storacle\/blob\/master\/dist\/storacle.client.js\">https:\/\/github.com\/ortexx\/storacle\/blob\/master\/dist\/storacle.client.js<\/a><\/noindex> as a script and gain access to <strong>window.ClientStoracle <\/strong>or import it through a build system, etc.<\/p>\n<p><\/p>\n<p><strong>Deferred Links<\/strong><\/p>\n<p><\/p>\n<p>An interesting feature is the 'deferred link.' It's a link to a file that can be obtained synchronously, here and now, while the file will be retrieved once it is found in the storage. This is very convenient when, for example, you need to display some images on a site. Just set the deferred link in the src, and that\u2019s it. Many cases can be imagined.<\/p>\n<p><\/p>\n<p><strong>Client API<\/strong><\/p>\n<p><\/p>\n<ul>\n<li>async <strong>Client.prototype.storeFile()<\/strong> \u2014 saving a file<\/li>\n<li>async <strong>Client.prototype.getFileLink()<\/strong> \u2014 obtaining a direct link to a file<\/li>\n<li>async <strong>Client.prototype.getFileLinks()<\/strong> \u2014 retrieving a list of direct links to the file from all nodes where it exists<\/li>\n<li>async <strong>Client.prototype.getFileToBuffer()<\/strong> \u2014 get the file into a buffer<\/li>\n<li>async <strong>Client.prototype.getFileToPath()<\/strong> \u2014 get the file into the file system<\/li>\n<li>async <strong>Client.prototype.getFileToBlob()<\/strong> \u2014 get the file as a blob (for the browser version)<\/li>\n<li>async <strong>Client.prototype.removeFile()<\/strong> \u2014 delete file<\/li>\n<li><strong>Client.prototype.createRequestedFileLink()<\/strong> \u2014 create a deferred link<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Exporting files to another server<\/strong><\/p>\n<p><\/p>\n<p>To transfer files to another node, you can:<\/p>\n<p><\/p>\n<ul>\n<li>Simply copy the entire storage folder along with the settings. (this may not work in the future)<\/li>\n<li>Copy only the folder with the files. However, in this case, you will need to run the function <strong>node.normalizeFilesInfo()<\/strong>, to recalculate all data and enter it into the database.<\/li>\n<li>Use the function <strong>node.exportFiles()<\/strong>, which will start copying the files.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Main node settings<\/strong><br \/>\nWhen starting the storage node, you can specify all necessary settings.<br \/>\nI will describe the most basic ones; the rest can be found on GitHub.<\/p>\n<p><\/p>\n<ul>\n<li><strong>storage.dataSize<\/strong> \u2014 size of the folder with files<\/li>\n<li><strong>storage.tempSize<\/strong> \u2014 size of the temporary folder<\/li>\n<li><strong>storage.autoCleanSize<\/strong> \u2014 minimum size of storage that needs to be maintained. If this parameter is specified, as soon as space starts to run low, the least-used files will be deleted.<\/li>\n<li><strong>file.maxSize<\/strong> \u2014 maximum file size<\/li>\n<li><strong>file.minSize<\/strong> \u2014 minimum file size<\/li>\n<li><strong>file.preferredDuplicates<\/strong> \u2014 preferred number of file duplicates in the network<\/li>\n<li><strong>file.mimeWhitelist<\/strong> \u2014 allowed file types<\/li>\n<li><strong>file.mimeBlacklist<\/strong> \u2014 disallowed file types<\/li>\n<li><strong>file.extWhitelist<\/strong> \u2014 allowed file extensions<\/li>\n<li><strong>file.extBlacklist<\/strong> \u2014 disallowed file extensions<\/li>\n<li><strong>file.linkCache<\/strong> \u2014 various link caching settings<\/li>\n<\/ul>\n<p><\/p>\n<p>Almost all parameters related to sizes can be specified in both absolute and relative values.<\/p>\n<p><\/p>\n<p><strong>Working through the command line<\/strong><br \/>\nThe library can be used through the command line. To do this, it needs to be installed globally: <strong>npm i -g storacle<\/strong>. After this, you can execute the necessary actions from the directory with the project where the node is. For example, <strong>storacle -a storeFile -f .\/file.txt -c .\/config.js<\/strong>, to add a file. All actions can be found in <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/ortexx\/storacle\/blob\/master\/bin\/actions.js\">https:\/\/github.com\/ortexx\/storacle\/blob\/master\/bin\/actions.js<\/a><\/noindex><\/p>\n<p><\/p>\n<p><strong>Why you might need this<\/strong><\/p>\n<p><\/p>\n<ul>\n<li>If you want to create some decentralized project in which it is planned to store and work with files using convenient methods. For example, the music project described at the link at the beginning of the article uses <strong>storacle<\/strong>.<\/li>\n<li>If you are working on any other projects that require distributed file storage, you can easily set up your private network, flexibly configure nodes, and add new ones as needed.<\/li>\n<li>If you simply need a place to store your website files and don\u2019t want to write everything yourself, this library might be a better fit for you.<\/li>\n<li>If you have a project where you're working with files but want to perform all operations from the browser, you can avoid writing server-side code.<\/li>\n<\/ul>\n<p><\/p>\n<p>My contacts:<\/p>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/ortex\">@ortex<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"mailto:mywebstreet@gmail.com\">mywebstreet@gmail.com<\/a><\/noindex><\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/499954\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0435\u0436\u0434\u0435 \u0447\u0435\u043c \u043d\u0430\u0447\u043d\u0443, \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0441\u0441\u044b\u043b\u043a\u0443 \u043d\u0430 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0443\u044e \u0441\u0442\u0430\u0442\u044c\u044e, \u0447\u0442\u043e\u0431\u044b \u0431\u044b\u043b\u043e \u043f\u043e\u043d\u044f\u0442\u043d\u043e \u043e \u0447\u0435\u043c \u0438\u043c\u0435\u043d\u043d\u043e \u0440\u0435\u0447\u044c. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u0445\u043e\u0442\u0435\u043b \u0431\u044b \u0440\u0430\u0437\u043e\u0431\u0440\u0430\u0442\u044c \u0441\u043b\u043e\u0439, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0442\u0432\u0435\u0447\u0430\u0435\u0442 \u0437\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432, \u0438 \u043a\u0430\u043a \u044d\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u043e \u043b\u044e\u0431\u044b\u043c \u0447\u0435\u043b\u043e\u0432\u0435\u043a\u043e\u043c. Storacle \u2014 \u0441\u0430\u043c\u043e\u0441\u0442\u043e\u044f\u0442\u0435\u043b\u044c\u043d\u0430\u044f \u0431\u0438\u0431\u043b\u0438\u043e\u0442\u0435\u043a\u0430, \u043d\u0438\u043a\u0430\u043a\u043e\u0439 \u0441\u0432\u044f\u0437\u0438 \u0441 \u043c\u0443\u0437\u044b\u043a\u043e\u0439 \u043d\u0430\u043f\u0440\u044f\u043c\u0443\u044e \u043d\u0435\u0442. \u041c\u043e\u0436\u043d\u043e \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u0442\u044c \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435 \u043b\u044e\u0431\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432. \u0412 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80225,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80224","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0435\u0436\u0434\u0435 \u0447\u0435\u043c \u043d\u0430\u0447\u043d\u0443, \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Storacle \u2014 \u0434\u0435\u0446\u0435\u043d\u0442\u0440\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0444\u0430\u0439\u043b\u043e\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0435\u0436\u0434\u0435 \u0447\u0435\u043c \u043d\u0430\u0447\u043d\u0443, \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-04T11:42:20+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-04T11:42:20+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Storacle \u2014 decentralized file storage | ProHoster","description":"Before I begin, I must leave this.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Storacle \u2014 \u0434\u0435\u0446\u0435\u043d\u0442\u0440\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0444\u0430\u0439\u043b\u043e\u0432 | ProHoster","og:description":"\u041f\u0440\u0435\u0436\u0434\u0435 \u0447\u0435\u043c \u043d\u0430\u0447\u043d\u0443, \u0434\u043e\u043b\u0436\u0435\u043d \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/storacle-deczentralizovannoe-hranilishhe-fajlov","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-04T11:42:20+00:00","article:modified_time":"2020-05-04T11:42:20+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80224","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:22:48","updated":"2022-09-27 20:25:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/80224","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=80224"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/80224\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/80225"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=80224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=80224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=80224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}