{"id":36999,"date":"2019-10-31T22:15:13","date_gmt":"2019-10-31T19:15:13","guid":{"rendered":"https:\/\/prohoster.info\/blog\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov\/"},"modified":"2019-10-31T22:15:13","modified_gmt":"2019-10-31T19:15:13","slug":"haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","title":{"rendered":"Hacks for working with a large number of small files","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

The idea for this article came up spontaneously from a discussion in the comments section of a previous article. A Few Words About Inodes<\/a><\/noindex>.<\/p>\n

\"Hacks
\n
\nThe issue is that the internal specifics of our services involve storing a vast number of small files. Currently, we have hundreds of terabytes of such data. We have encountered some obvious and not-so-obvious pitfalls and have successfully navigated through them.<\/p>\n

Therefore, I\u2019m sharing our experience, which may be helpful to someone.
\n<\/a><\/noindex><\/p>\n

The First Problem: \"No Space Left on Device\"<\/h2>\n

\nAs mentioned in the previously referenced article, the problem is that while there are free blocks on the file system, inodes have run out.<\/p>\n

You can check the number of used and free inodes with the command df -ih<\/code>:<\/p>\n

\"Hacks
\n
\nI won\u2019t reiterate the article; to summarize, the disk has both blocks specifically for data and blocks for metadata, which are the inodes (index nodes). Their number is determined at the initialization of the file system (specifically referring to ext2 and its successors) and does not change thereafter. The balance between data blocks and inodes is calculated from average data; in our case, with many small files, this balance should shift towards having more inodes.<\/p>\n

Linux already provides options with different balances, and all these pre-calculated configurations are in the file \/etc\/mke2fs.conf<\/code>.
\nThus, during the primary initialization of the file system via mke2fs, you can specify the desired profile.<\/p>\n

Here are a few examples from the file:<\/p>\n

    small = {\n        blocksize = 1024\n        inode_size = 128\n        inode_ratio = 4096\n    }\n\n    big = {\n        inode_ratio = 32768\n    }\n\n    largefile = {\n        inode_ratio = 1048576\n        blocksize = -1\n    }\n<\/code><\/pre>\n
\nYou can select the desired use case using the option «-T» when calling mke2fs. You can also manually specify the required parameters if there is no ready-made solution.<\/p>\n
More details are described in the manuals for mke2fs.conf<\/code> and mke2fs<\/code>.<\/p>\n
An aspect not covered in the previously mentioned article is that you can specify the data block size. Obviously, larger blocks make sense for large files, while smaller blocks are preferable for smaller ones. <\/p>\n
However, it\u2019s important to consider an interesting aspect such as the processor architecture.
\nAt one point, I realized that for large photo files, I needed a larger block size. It was in a home environment, on a home file storage system named WD based on ARM architecture. Without much thought, I set the block size to either 8k or 16k instead of the standard 4k, having previously measured the savings. Everything was great until the storage itself failed, even though the disk was intact. When I placed the disk in a regular computer with a standard Intel processor, I got a surprise: unsupported block size. We were stuck. The data was there and everything seemed fine, but it was impossible to read it. i386 processors and similar ones cannot work with block sizes that do not match the memory page size, which is exactly 4k. In short, the situation ended with the use of user-space utilities, which were slow and disappointing, but the data was saved. If anyone is interested \u2014 look up the name of the utility. fuseext2<\/code>. The moral: Either think through all scenarios in advance or don\u2019t act like a superhero and use standard settings for laypeople.<\/p>\n
UPD. As noted by the user berez<\/a><\/noindex> , I clarify that for i386, the block size must not exceed 4k, but it does not necessarily have to be exactly 4k; sizes of 1k and 2k are acceptable.<\/p>\n
So, how we solved the problems.<\/p>\n
First, we faced the issue when a multi-terabyte disk was filled with data, and we could not redesign the file system configuration.<\/p>\n
Second, a quick solution was required.<\/p>\n
As a result, we concluded that we needed to change the balance by reducing the number of files.
\nTo decrease the number of files, we decided to combine files into a single archive. Given our specifics, we grouped all files from a certain period into one archive and conducted archiving with a cron job nightly.<\/p>\n
We have chosen a zip archive. The previous article suggested tar, but it has one difficulty: it does not have a directory, and the files are processed in sequence (it's not called \"tar\" for nothing \u2014 it's short for \"Tape Archive\", a legacy from tape drives), that is, if you need to read a file at the end of the archive, you have to read the entire archive since there are no offsets for each file relative to the beginning of the archive. Therefore, this is a lengthy operation. In zip, everything is much better: it has that very directory and file offsets inside the archive, and the access time to each file does not depend on its location. In our case, we could set the compression option to \"0\", as all the files have already been compressed in gzip.<\/p>\n
Clients retrieve files through nginx, and according to the old API, only the file name is specified, for example:<\/p>\n
http:\/\/www.server.com\/hydra\/20170416\/0453\/3bd24ae7-1df4-4d76-9d28-5b7fcb7fd8e5\n<\/code><\/pre>\n
\nTo unpack files on the fly, we found and connected the nginx-unzip-module (https:\/\/github.com\/youzee\/nginx-unzip-module<\/a><\/noindex>) and configured two upstreams.<\/p>\n
As a result, we ended up with the following configuration:<\/p>\n
\"Hacks
\n
\nThe two hosts in the settings looked like this:<\/p>\n
server {\n  listen *:8081;\n\n  location \/ {\n    root      \/home\/filestorage;\n  }\n}<\/code><\/pre>\n
<\/p>\n
server {\n  listen *:8082;\n\n  location ~ ^\/hydra\/(\\d+)\/(\\d+)\/(.*)$ {\n    root      \/home\/filestorage;\n    file_in_unzip_archivefile \"\/home\/filestorage\/hydra\/$1\/$2.zip\";\n    file_in_unzip_extract \"$2\/$3\";\n    file_in_unzip;\n  }\n}\n<\/code><\/pre>\n
\nAnd the configuration of the upstreams on the higher-level nginx:<\/p>\n
upstream storage {\n  server server.com:8081;\n  server server.com:8082;\n}\n<\/code><\/pre>\n
\nHow it works:<\/p>\n