for testing the second candidate in library releases with the implementation of a high-performance, compact embedded key-value database. The current version (0.5) is a technical release, marking the completion of any further developments and transitioning to the phase of public final testing and stabilization, leading to the formation of the first full release of the library. The code of libmdbx is licensed under the OpenLDAP Public License.
The MDBX library is a significantly reworked branch of — a transactional embedded key-value database based on without , which allows multi-threaded processes to work concurrently and efficiently with a locally-shared (non-network) database. In turn, MDBX is faster and more reliable than LMDB, while retaining all the key capabilities of its predecessor, such as and non-blocking reads with linear scaling across CPU cores.
The most important differences of MDBX compared to LMDB:
- Significantly more attention is paid to code quality, testing, and automated checks.
- Substantially more control during operations, from parameter checks to internal audits of database structures.
- Auto-compaction and automatic size management of the database.
- A unified database format for 32-bit and 64-bit builds.
- Estimation of the volume of samples by ranges.
- Support for keys with double size and user-defined page size for the database.
The released candidate for libmdbx is the result of the decision made in August 2019 to split the MDBX and MithrilDB projects. In this context, it was decided to eliminate (rational) maximum technical debt in libmdbx and stabilize the library. In fact, the work done in this direction is 2-3 times greater than originally estimated and planned:
- Support for macOS and 'second-tier' platforms: FreeBSD, Solaris, DragonFly BSD, OpenBSD, NetBSD. Support for AIX and HP-UX can be added if necessary.
- Code sanitization using Undefined Behavior Sanitizer and Address Sanitizer, all warnings during compilation with '-Wpedantic', all warnings from Coverity Static Analyzer, etc., have been resolved.
- Updating .
- Amalgamation of the source code for convenience in embedding.
- CMake support.
- Support for nested transactions.
- Using bootid to determine the fact of OS reboot (dirty database shutdown).
- End-to-end counting of updated/old pages and extended transaction information.
- MDBX_ACCEDE option for connecting to an already opened database in compatible mode.
- Using when available.
- Hot backup in pipe.
- Specialized optimized internal sorting algorithm (up to 2-3 times faster than 'qsort()' and up to 30% faster than 'std::sort()').
- Increased maximum key length.
- Automatic read-ahead management (database file caching strategy in memory).
- More aggressive and faster auto-compaction.
- More optimal B+ tree page merging strategy.
- Monitoring of non-local file systems (NFS, Samba, etc.) to prevent database corruption due to misuse.
- Expanded test suite.
Development of the 'next' version of libmdbx will continue within a separate project , while the development vector of the 'current' version of MDBX is aimed at freezing the feature set and stabilization. This decision has been made for three reasons:
- Complete incompatibility: achieving all planned features in MithrilDB requires a different (incompatible) database file format and a different (incompatible) API.
- New source code: the source code for MithrilDB is ensured to be license-independent from LMDB, and the project is planned to be published under a different (approved Apache 2.0 license rather than ).
- The separation allows avoiding potential confusion, bringing more clarity, and ensuring development independence for the projects.
MithrilDB, like MDBX, is also based on and will also feature extremely high performance, while eliminating a number of fundamental drawbacks of MDBX and LMDB. In particular, the problem of 'long reads', manifested as 'database bloat' due to garbage collection being blocked by long-reading transactions, will be resolved. Among the new features of MithrilDB, it is worth noting:
- Support for placing the database on multiple heterogeneous storage media: HDD, SSD, and non-volatile memory.
- Optimal strategies for 'valuable' and 'less valuable', for 'hot', 'warm', and 'cold' data.
- Use of Merkle tree for database integrity control.
- Optional use of WAL and significantly higher performance in write-intensive scenarios with guarantees on data integrity.
- Lazy catching of data on disks.
Source: opennet.ru
