{"id":55408,"date":"2020-01-20T00:00:00","date_gmt":"2020-01-19T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie"},"modified":"2020-02-18T14:03:31","modified_gmt":"2020-02-18T11:03:31","slug":"effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","title":{"rendered":"Efficient storage of hundreds of millions of small files. Self-Hosted solution","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Efficient
\n
\nDear community, this article will focus on effective storage and retrieval of hundreds of millions of small files. At this stage, a final solution for POSIX-compatible file systems with full locking support, including clustered locks, is proposed, seemingly even without hacks.<\/p>\n

Therefore, for this purpose, I wrote my own specialized server.
\nIn the course of implementing this task, I managed to solve the main problem while simultaneously achieving savings in disk space and RAM, which was ruthlessly consumed by our clustered file system. Such a quantity of files is detrimental to any clustered file system. <\/a><\/noindex><\/p>\n

The idea is as follows:<\/p>\n

Simply put, small files are uploaded through the server, they are saved directly into an archive, and read from it as well, while larger files are stored alongside. The scheme: 1 folder = 1 archive, resulting in several million archives of small files instead of several hundred million files. And all this is implemented in full, without any scripts and not distributing files into tar\/zip archives.<\/p>\n

I will try to summarize, I apologize in advance if the post turns out to be extensive.<\/p>\n

It all started with the fact that I could not find a suitable server in the world that could save data received via the HTTP protocol directly into archives, avoiding the shortcomings inherent in ordinary archives and object storage. The reason for this search was a sprawling Origin cluster of 10 servers, which has already accumulated 250,000,000 small files, and the growth trend showed no signs of stopping.<\/p>\n

For those who do not like reading articles and prefer brief documentation:<\/b><\/p>\n

here<\/a><\/noindex> and here<\/a><\/noindex>.<\/p>\n

And docker at the same time, currently there is only an option to run it together with nginx just in case:<\/p>\n

docker run -d --restart=always -e host=localhost -e root=\\\/var\\\/storage \n-v \\\/var\\\/storage:\\\/var\\\/storage --name wzd -p 80:80 eltaline\\\/wzd<\/code><\/pre>\n
\nNext:<\/p>\n
When there are a lot of files, significant resources are required, and unfortunately, some of them go to waste. For example, when using a clustered file system (in this case, MooseFS), a file, regardless of its actual size, always occupies at least 64 KB. That means for files sized 3, 10, or 30 KB, 64 KB is taken up on the disk. If there are a quarter of a billion files, we lose between 2 to 10 terabytes. It won't be possible to endlessly create new files, as MooseFS has a limit: no more than 1 billion with one replica of each file.<\/p>\n
As the number of files increases, a lot of RAM is needed for the metadata. Additionally, frequent large metadata dumps contribute to SSD wear.<\/p>\n
wZD Server. Organizing the disks.<\/b><\/p>\n
The server is written in Go. First of all, I needed to reduce the number of files. How can this be done? Through archiving, but in this case without compression, as my files are solid compressed images. BoltDB came to the rescue, but it had to be refined, which is reflected in the documentation.<\/p>\n
So instead of a quarter billion files, in my case, there are only 10 million Bolt archives left. If I had the opportunity to change the current directory structure for filling files, it might be possible to reduce it to about 1 million files. <\/p>\n
All small files are packed into Bolt archives, automatically acquiring the names of the directories they reside in, while larger files remain next to the archives, as there's no point in packing them, which is configurable. Small ones are archived, large ones are left unchanged. The server operates transparently with both types.<\/p>\n
Architecture and features of the wZD server.<\/b><\/p>\n
\"Efficient
\n
\nThe server operates under Linux, BSD, Solaris, and OSX operating systems. I've only tested it for AMD64 architecture under Linux, but it should also work for ARM64, PPC64, and MIPS64.<\/p>\n
Key features:<\/b><\/p>\n