Library libmdbx is a significantly redesigned descendant of LMDB - an extremely high-performance, compact embedded key-value database.
The current version v0.5 is a technical release, marking the completion of any improvements and the transition to the public final testing and stabilization phase, with the subsequent formation of the first full release of the library.
LMDB is a fairly well-known transactional embedded DBMS of the key-value class based on tree B+ without proactive logging, which allows a swarm of multithreaded processes to work competitively and extremely efficiently with a locally shared (not network) database. In turn, MDBX is faster and more reliable than LMDB, while libmdbx retains all the key features of its progenitor, such as ACID and non-blocking read with linear scaling across CPU cores, and also adds a few new ones.
A description of the differences and improvements of libmdbx relative to LMDB deserves a separate article (it is planned to publish on HabrΓ© and Medium). Here it is appropriate to mention the most important and noticeable:
- Fundamentally, more attention is paid to code quality, testing and automatic checks.
- Significantly more control during operation, from parameter checking to internal audit of database structures.
- Auto-compactification and automatic database size management.
- A single database format for 32-bit and 64-bit assemblies.
- Estimation of the volume of samples by ranges (range query estimation).
- Support for keys twice the size of pancakes and user-selectable database page size.
The released libmdbx release candidate is the result of the decision (see below) to split the MDBX and MithrilDB projects in August 2019. In doing so, libmdbx decided to eliminate the (reasonable) maximum technical debt and stabilize the library. In fact, 2-3 times more work has been done in the indicated direction than was originally estimated and planned:
- Implemented support for Mac OS and second-tier platforms: FreeBSD, Solaris, DragonFly BSD, OpenBSD, NetBSD. Support for AIX and HP -UX can be added as needed.
- Sanitized code with Undefined Behavior Sanitizer and Address Sanitizer, fixed all build warnings with -Wpedantic, all Coverity Static Analyzer warnings, etc.
- Updating API descriptions.
- Amalgamation of the source code for ease of embedding.
- CMake support.
- Support for nested transactions.
- Using the bootid to determine if the OS has rebooted (dirty shutdown of the database).
- End-to-end count of updated/old pages and advanced information about transactions.
- MDBX_ACCEDE option to connect to an already open database in compatible mode.
- Using OFD locks upon their availability.
- Hot backup in pipe.
- Specialized optimized internal sorting algorithm (up to 2-3 times faster than qsort() and up to 30% faster than std::sort()).
- Increased maximum key length.
- Automatic control of read ahead (in-memory database file caching strategy).
- More aggressive and faster auto-compactification.
- A more optimal strategy for merging B+ tree pages.
- Control of non-local file systems (NFS, Samba, etc.) to prevent database corruption due to misuse.
- Expanded test suite.
Development of the "next" version of libmdbx will continue as a separate project MithrilDB, while the development vector of the "current" version of MDBX is aimed at freezing the feature set and stabilizing. This decision was made for three reasons:
- Total incompatibility: MithrilDB requires a different (incompatible) database file format and a different (incompatible) API to implement all the planned features.
- New source code: MithrilDB source code is license independent from LMDB, and the project itself is planned to be published under a different license (approved by OSI license Apache 2.0And not OpenLDAP Foundation).
- Separation avoids potential confusion, introduces more certainty and ensures the independence of the development path of projects.
MithrilDB, like MDBX, is also based on tree B+ and will also be characterized by extremely high performance, while eliminating a number of fundamental shortcomings of MDBX and LMDB. In particular, the problem of "long reads", which manifests itself as "swelling" of the database due to the fact that garbage processing is blocked by long reading transactions, will be eliminated. New MithrilDB features include:
- support for placing the database on several heterogeneous media: HDD, SSD and non-volatile memory.
- optimal strategies for "valuable" and "low-value", for "hot", "warm" and "cold" data.
- using Merkle tree to control the integrity of the database.
- optional use of WAL and significantly better performance in write-intensive scenarios with data integrity guarantees.
- lazy catch-up commits to disks.
Source: linux.org.ru