first edition - a server for efficiently storing a large number of files in a compact form that looks like a regular WebDAV server from the outside. A modified version is used for storage . The project code is written in Go and under the BSD license.
Server significantly reduce the number of small files on regular or clustered file systems with full locking support. The wZD developer-maintained cluster stores about 250 million small files spread across 15 million directories in a clustered FS .
wZD makes it possible to move (archive) the contents of directories to archives in the BoltDB format and then distribute these files from these archives (or put files into archives using the PUT method), significantly reducing the number of files in the FS and reducing the overhead of storing metadata. To increase the efficiency of processing large files, such files can be saved separately from Bolt archives. This approach allows you to organize the storage of a huge number of small files, without hitting the limit on the number of inodes in the file system.
The server can also be used as a NoSQL database for data in the key/value format (with sharding based on the directory structure) or for distributing pre-generated html or json documents from the database. In terms of performance, uploading and writing data using Bolt archives leads to an increase in latency by approximately 20-25% when reading and by 40-50% when writing. The smaller the file size, the smaller the difference in latency.
All :
- Multithreading;
- Multiserver, providing fault tolerance and load balancing;
- Maximum transparency for the user or developer;
- Supported HTTP methods: GET, HEAD, PUT and DELETE;
- Managing read and write behavior through client-side headers;
- Support for highly configurable virtual hosts;
- Support for CRC data integrity when writing / reading;
- Semi-dynamic buffers for minimal memory consumption and optimal network performance tuning;
- Delayed data packing;
- In addition, a multi-threaded archiver is offered to move files to Bolt archives without stopping the service.
Some limitations of the current release: no support for Multipart, POST method, HTTPS protocol, bindings for programming languages, recursive deletion of directories, no support for mounting a structure to a file system via WebDAV or FUSE, files are stored under one system user. The storage format is architecture specific and not portable between Little Endian and Big Endian systems. Despite the fact that the wZD server implements support for the HTTP protocol, you only need to run it under the guise of reverse proxies, such as nginx and haproxy.
Source: opennet.ru