The libraries libmdbx 0.10.4 (MDBX) have been released, featuring a high-performance compact embedded key-value database, along with the related library libfpta 0.3.9 (FPTA), which implements a tabular data representation with secondary and composite indexes on top of MDBX. Both libraries are distributed under OSI-approved licenses. They support all current operating systems and architectures, including the Russian Elbrus 2000.
Historically, libmdbx is a deep reworking of the LMDB DBMS and surpasses its predecessor in reliability, feature set, and performance. Compared to LMDB, libmdbx places a strong emphasis on code quality, stable API operation, testing, and automated checks. A utility for integrity checking of the database structure is provided, along with some recovery capabilities.
Technologically, libmdbx offers ACID compliance, strict change serialization, and non-blocking reads with linear scaling across CPU cores. It supports auto-compaction, automatic database size management, and range query estimation. Since 2016, the project has been funded by Positive Technologies and has been used in its products since 2017.
A C++ API is offered for libmdbx, as well as bindings supported by enthusiasts for languages like Rust, Haskell, Python, NodeJS, Ruby, Go, and Nim. For libfpta, only the API description is publicly available in the form of a C/C++ header file.
The main innovations, improvements, and fixes added since the previous news on May 9:
- The ability to produce reproducible builds has been ensured.
- A bug was fixed that could cause a deadlock during transaction commits under very rare circumstances. The issue was identified by Positive Technologies specialists during internal testing of their products.
- Tests have been improved and test scenarios have been expanded to check all achievable non-isomorphic states of the page tree and GC contents within the database.
- An extra ‘noexcept’ was removed from the C++ API, additional overloads were added for the ‘cursor::erase()’ method, and the buffer implementation has been rid of the use of ‘std::string’ to ensure alignment (relevant for CLANG libstdc++).
- The regression in the dirty page spilling algorithm (selective eviction of modified database pages) has been fixed, which manifested as a rare unexpected MDBX_PROBLEM error when modifying data in huge transactions.
- A phase-checking review has been conducted, adding a number of checks to ensure stability during intentional database corruption.
- Minor warnings from UndefinedBehaviorSanitizer and Coverity Scan issues have been resolved.
- The check for the deprecated and no longer used internal flag 'P_DIRTY' in nested pages within database images created by older versions of the library has been corrected.
- The CMake scripts have been updated to improve the search for compiler components needed for LTO (link-time optimization).
- The maximum number of simultaneous readers has been increased to 32,767.
- Performance has been improved when using Valgrind and AddressSanitizer.
- On Windows, recursive SRW-lock usage has been eliminated when operating in MDBX_NOTLS mode (without using thread local storage), bootid generation has been fixed when the system time changes, WSL1 and WSL2 detection has been improved, and the ability to open databases on Plan 9 mounted via DrvFS has been added.
- In total, more than 160 changes have been made in 57 files, approximately 5,000 lines added, and about 2,500 lines removed.
A special thanks to the Erigon project team (Ethereum ecosystem) for their assistance in testing in extreme usage scenarios. Notably, over five months since the release of libmdbx v0.10.0, with database sizes of 1-2 TB in each Erigon installation (which is used on 7% of Ethereum nodes), only three reports of database corruption have been received, all due to external reasons rather than software errors: in two cases, the cause was RAM failures, and in the third, it was a data zeroing error in a specific storage subsystem configuration using BTRFS.
Source: opennet.ru
