I once set out to write an application for selecting music to listen to at home, outdoors, or during workouts, among other situations. I envisioned the architecture, sketched a prototype, and ultimately faced one "small problem."
But it was unclear where to source the actual music files. By this time, VKontakte had already closed its API, and the major music portals were similarly unresponsive, even returning songs in fragments to prevent parsing. The only remaining options were some random one-off sites filled with ads and junk, along with various dubious grabber programs and other "dirty" alternatives. In short, there were no genuinely good solutions. Of course, one could buy a subscription to something like Yandex Music or the like. Yet again, there was no open public API available, and you didn't have programmatic access to the music. A few large companies essentially restricted others' access to music. Why did this happen? Diving deeper, it became clear that the main issue was copyright. The current subscription model satisfies many commercial creators of musical works and the companies involved. Meanwhile, non-commercial and semi-commercial music also gets lumped into the general category. You either pay for all of it, or you don't listen to anything at all.
I started thinking about what to do with all this. How can we organize the free distribution of music? What would I do if I were creating music myself and wanted to earn from it? Would I appreciate it if my songs began to be distributed piratedly? What are the alternative solutions?
Ultimately, two main problems emerged that need addressing:
- Organizing the free distribution of music through methods that are convenient for the majority of people, including through software.
- Offering alternatives for music creators to earn revenue
A global decentralized music storage
Initially, I tried to find existing solutions and build everything on that foundation. After some time searching, the first appealing option I found was . I began to implement my idea, but after a while, I encountered several critical issues with this solution:
- IPFS is a storage solution for everything and anything. It contains images, music, videos, and anything you can think of. In general, it's a large planetary 'dump'. So when you launch your node, you immediately face a huge load. The machine struggles under the weight.
- It's some half-baked mechanism for collecting 'garbage'. I don't know how it is now, but back then, if you specified in the config that you wanted to limit storage to ten gigabytes, it meant nothing. The storage expanded, ignoring many configuration parameters. Ultimately, you needed a huge amount of hard drive space while IPFS figured out how to drop the unnecessary data.
- At the time I used the library (I don't know how it is now), the client had not implemented timeouts. You send a request to retrieve a file, and if it's not there, you just hang. Of course, people came up with various workarounds that partially solved the problem, but these were merely band-aids. Such things should come out of the box.
There were also many minor issues, and the impression I left with was clear: this cannot be used for a project. I continued searching for a storage solution, exploring various options, but found nothing suitable.
In the end, I decided it was worth trying to write a decentralized storage system myself. It may not aim to be interplanetary, but it will solve the specific task at hand.
Thus, we ended up with , , , , .
spreadable is the main, foundational layer that allows nodes to come together into a network. It includes an algorithm that I have only partially implemented, designed for around 10,000 servers. The full version of the algorithm is much more complex to implement and would require several additional months (maybe more).
I won't detail spreadable in this article; I'll write a separate one sometime. Here I will only highlight some particular features:
- It operates over http/https.
- You can create a separate network for a specific task, which significantly reduces the load on each individual project, compared to having them all in one network.
- A mechanism with timeouts and other minor details was originally thought out. And it works for all methods, both on the client side and in the node. You can flexibly manage parameters from your application.
- The library is written in Node.js. Performance bottlenecks are mitigated by its decentralized nature. The load can be 'spread out' by increasing the number of nodes. In return, there are many advantages: a large community, simplicity and convenience of use, an isomorphic client, no external dependencies, etc.
storacle — is a layer that inherits from spreadable, allowing files to be stored on the network. Each file has its own hash based on its content, which can later be used to retrieve it. Files are not divided into blocks but stored in their entirety.
metastocle — is a layer that inherits from spreadable, enabling the storage of data on the network but not files. The interface resembles that of NoSQL databases. For example, one can add a file to storacle, obtain its hash, and record it in metastocle linked to something.
museria — inherits from storacle and metastocle. This layer is directly responsible for storing music. The storage works only with mp3 files and ID3 tags.
The 'key' to a song is its full title in the form of Artist (TPE1) — Title (TIT2). For example:
- Brimstone — The Burden
- Hi-rez — Lost My Way (feat. Emilio Rojas, Dani Devinci)
You can learn in detail how song titles are formed . You need to check the function utils.beautifySongTitle().
A match based on keys is defined as a percentage set in the node settings. For example, a value of 0.85 means that if the key comparison function (song titles) detects a similarity of more than 85%, then it is considered the same song.
The similarity detection algorithm is also there, in the function utils.getSongSimilarity().
The cover for a song, for subsequent retrieval, can also be attached via tags (APIC). The utilities (utils) contain all the necessary methods for retrieving and processing tags.
An example of working with the storage through the client can be found in .
All the above layers are self-sufficient and can be used independently as lower layers for other projects. For instance, there is already a thought to create a layer for storing books.
museria-global — is a pre-configured git repository to run your own node in the global music network. Clone it, npm i && npm start and that's essentially it. You can set up more details, run it in Docker, etc. Detailed information is available on .
When a repository is updated, you need to update your node as well. If the major or minor version number changes, this action is mandatory; otherwise, old nodes will be ignored by the network.
You can work with songs either manually or programmatically. Each node runs a server for various tasks. Specifically, when visiting the default endpoint, you will receive an interface to work with music. For example, you can visit (the link may become outdated later; entry nodes can also be found in , or you can check updates on GitHub).
This way, you can search for and upload songs to the storage. Songs can be uploaded in two modes: regular and moderated. The second mode means that the process is handled by a person instead of a program. If you check this option when adding, you will need to solve a CAPTCHA. Songs can be added with priorities -1, 0, or 1. Priority 1 can only be assigned in moderated mode. Priorities are necessary for the storage to make more efficient decisions when you try to replace an existing song with a new one. The higher the priority, the greater the chance that you will overwrite the existing file. This helps combat spam and increases the quality of uploaded songs.
If you start adding songs to the storage, try to attach images (covers), even though this field is not mandatory. In 99% of cases, the first images found on Google by song titles are album covers.
Here’s how file addition technically works, in brief:
- The client receives the address of a free node, which will temporarily become the coordinator.
- The song addition function is triggered (by a person or code), and a request is made to add the song to the coordinator's endpoint.
- The coordinator calculates how many duplicates need to be retained (a configurable parameter).
- The most suitable nodes for storage are searched.
- The file is directly sent to these nodes.
Here’s how file retrieval technically works:
- The client receives the address of a free node, which will temporarily become the coordinator.
- The song retrieval function is triggered (by a person or code), and a request is made to retrieve it at the coordinator's endpoint.
- The coordinator checks for the link in the cache. If it exists and is valid, it is immediately returned to the client; otherwise, the nodes are queried to check for availability.
- The file is being obtained via the link, if found.
Alternatives for Music Creators
I've always been curious about how to objectively assess the value of many creative works. Why, for example, does someone put their music album for $10? Or for $20 or $100? Where is the algorithm? When we talk about a physical product or even many types of services, we can at least calculate the cost price and base our evaluation on that.
Okay, let's say it's priced at $10. Is this really effective? Suppose I listened to the album somewhere or a song from it and wanted to show my gratitude. But based on my feelings and personal capabilities, $3 is my ceiling. So what do we do? Most likely, I just won't do anything, like most people.
By setting a fixed price for creative work, you simply limit yourself and prevent a larger number of people from sending you smaller amounts of money, which could sum up to more than those who buy at your set price. It seems to me that creativity is precisely the sphere where donations should primarily govern. For this, we need to:
- Teach people to express gratitude in this way. Creators themselves should clearly show that they would like to receive donations, adding links to various payment methods everywhere, etc.
- We need more mechanisms to simplify and enhance these processes. For example, create a global website where people can donate for creativity using author links.
Let's say the link looks something like this:
http://someartistsdonationsite.site/category/artist?external-infoIf we narrow it down to musicians, then:
http://someartistsdonationsite.com/music/miyagi?song=blablaThe performer needs to verify their nickname and attach themselves to it.
In the museria client, we add a feature to generate such a link, and all projects using the repository can place donation buttons with these links next to the songs on their websites/apps. Users can quickly and easily make donations. Naturally, this approach can be applied in any project and category of creativity, not just through the repository.
Why, specifically for you, is a music repository important, and how can you participate?
- If you are working on a music-related project or planning to create one, this is what it was all designed for. You can use museria to store and retrieve songs, increasing the flow of songs online. If you have the capability to set up and maintain at least one personal node, it will be the best contribution to the development of the network.
- Perhaps you are ready to take on a different role: help with coding, or fill and moderate the database, spread information about the project to your acquaintances, etc.
- Maybe you liked the idea and are willing to contribute financially so that it can live and grow. The more nodes, the more songs.
- Or at some point, you might simply need to find and download a song. You can easily do this, for example, through .
The project is currently in its very early stages. A test network has been launched, nodes may frequently restart, require updates, etc. If there are no critical issues during the evaluation period, this same network will transform into the main one.
You can view external information about the node: the number of songs, free space, etc., via a link of the form http://node-address/status or http://node-address/status?pretty
My contacts:
Source: habr.com
