first release — servers for efficient storage of a large number of files in a compact form, which externally appears as a regular WebDAV server. A modified version is used for storage . The project code is written in Go language and under the BSD license.
Server significantly reduces the number of small files on traditional or clustered file systems with full lock support. The wZD cluster, supported by developers, stores about 250 million small files distributed across 15 million directories in a cluster file system .
wZD allows you to move (archive) the contents of directories into archives in BoltDB format and then serve these files from these archives (or place files into archives using the PUT method), significantly reducing the number of files in the file system and lowering metadata storage overhead. To enhance efficiency in handling large files, such files can be stored separately from Bolt archives. This approach allows for the organization of a vast number of small files without hitting the inode limit in the file system.
The server can also be used as a NoSQL database for key/value data (with sharding based on directory structure) or for serving pre-generated HTML or JSON documents from the database. Regarding performance, the retrieval and writing of data using Bolt archives results in an increase in latency by approximately 20-25% during reading and by 40-50% during writing. The smaller the file size, the smaller the difference in latencies.
Key :
- Multithreading;
- Multi-server capabilities providing fault tolerance and load balancing;
- Maximum transparency for the user or developer;
- Supported HTTP methods: GET, HEAD, PUT, and DELETE;
- Control over reading and writing behavior through client headers;
- Support for flexibly configurable virtual hosts;
- Support for CRC data integrity during writing/reading;
- Semi-dynamic buffers for minimal memory consumption and optimal network performance tuning;
- Deferred data packaging;
- Additionally, a multithreaded archiver is offered for moving files into Bolt archives without stopping the service.
Some limitations of the current release: no support for Multipart, POST method, HTTPS protocol, bindings for programming languages, recursive directory deletion, lack of support for mounting the structure to the file system via WebDAV or FUSE, files are stored under a single system user. The storage format is tied to the architecture and is not transferable between Little Endian and Big Endian systems. Although the wZD server implements support for the HTTP protocol, it must be run only behind reverse proxies such as nginx and haproxy.
Source: opennet.ru
