Library libmdbx is a significantly revamped descendant of LMDB — an extremely high-performance, compact embedded key-value database.
The current version v0.5 is a technical release, marking the completion of any enhancements and the transition to the phase of public final testing and stabilization, leading to the formation of the first full release of the library.
LMDB is a fairly well-known transactional embedded DBMS of the 'key-value' class based on B+ tree without anticipatory journaling, which allows a set of multithreaded processes to work concurrently and extremely efficiently with a locally shared (non-networked) DB. In turn, MDBX is faster and more reliable than LMDB, while libmdbx retains all the key features of its predecessor, such as ACID and non-blocking reads with linear scaling across CPU cores, while also adding several new ones.
The description of differences and improvements of libmdbx compared to LMDB deserves a separate article (planned publications on 'Habr' and 'Medium'). Here it is appropriate to mention the most important and notable:
- Significantly more attention is paid to code quality, testing, and automated checks.
- Substantially more control during operations, from parameter checks to internal audits of database structures.
- Auto-compaction and automatic size management of the database.
- A unified database format for 32-bit and 64-bit builds.
- Estimation of range query sizes.
- Support for keys with double size and user-defined page size for the database.
The released libmdbx release candidate is the result of a decision (see below) to separate the MDBX and MithrilDB projects in August 2019. At the same time, it was decided to eliminate (rational) maximum technical debt and stabilize the library. In fact, much more has been done in the designated direction than was initially estimated and planned: 2-3 times more.
- Support for Mac OS and 'second-tier' platforms: FreeBSD, Solaris, DragonFly BSD, OpenBSD, NetBSD. Support for AIX and HP-UX can be added if necessary.
- Code sanitation using Undefined Behavior Sanitizer and Address Sanitizer has been conducted, all warnings during compilation with -Wpedantic have been eliminated, all warnings from Coverity Static Analyzer, etc.
- Updating API documentation.
- Amalgamation of the source code for ease of embedding.
- CMake support.
- Support for nested transactions.
- Using bootid to determine the fact of OS reboot (dirty database shutdown).
- Comprehensive counting of updated/old pages and extended transaction information.
- MDBX_ACCEDE option for connecting to an already opened database in compatible mode.
- Using OFD locks when available.
- Hot backup in pipe.
- A specialized optimized internal sorting algorithm (2-3 times faster than qsort() and up to 30% faster than std::sort()).
- Increased maximum key length.
- Automatic read-ahead management (database file caching strategy in memory).
- More aggressive and faster auto-compaction.
- More optimal B+ tree page merging strategy.
- Monitoring of non-local file systems (NFS, Samba, etc.) to prevent database corruption from improper usage.
- Expanded test suite.
Development of the 'next' version of libmdbx will continue within a separate project MithrilDB, while the development vector of the 'current' version of MDBX is aimed at freezing the feature set and stabilization. This decision has been made for three reasons:
- Complete incompatibility: achieving all planned features in MithrilDB requires a different (incompatible) database file format and a different (incompatible) API.
- New source code: the source code for MithrilDB is ensured to be license-independent from LMDB, and the project is planned to be published under a different (approved OSI license Apache 2.0, not OpenLDAP Foundation).
- Separation helps avoid potential confusion, bring more clarity, and ensure the independence of project development paths.
MithrilDB, like MDBX, is also based on B+ tree and will also feature extremely high performance, while eliminating a number of fundamental drawbacks of MDBX and LMDB. In particular, the problem of 'long reads', manifested as 'database bloat' due to garbage collection being blocked by long-reading transactions, will be resolved. Among the new features of MithrilDB, it is worth noting:
- Support for hosting databases on multiple heterogeneous media: HDD, SSD, and non-volatile memory.
- Optimal strategies for 'valuable' and 'less valuable', for 'hot', 'warm', and 'cold' data.
- Using Merkle tree for database integrity control.
- Optional use of WAL and significantly higher performance in scenarios with intensive writing and guarantees on data integrity.
- Lazy catching up of data on disks.
Source: linux.org.ru
