The release of the libmdbx 0.11.7 library (MDBX) has taken place, featuring a high-performance compact embedded key-value database. The libmdbx code is distributed under the OpenLDAP Public License. All current operating systems and architectures are supported, including the Russian Elbrus 2000.
This release is notable for the migration of the project to the GitFlic service after GitHub's administration unexpectedly removed libmdbx along with numerous other projects on April 15, 2022, without any warning or explanation, simultaneously blocking access for many developers associated with companies under U.S. sanctions. From the users' perspective, all project pages, the repository, and forks suddenly turned into a "404" page, with no means of communication or inquiry.
Unfortunately, nearly all issues have been lost, which contained many questions and detailed answers, as well as numerous discussions. The loss of this information represents the only objective damage that GitHub's administration has inflicted on the project. Some copies of the discussions remain accessible in the archive at archive.org.
The loss of established CI scripts and infrastructure (available for OpenSource projects at no cost) compelled a review, unification, and resolution of some technical debt. CI has now been restored almost to its previous state, except for the builds and test runs for all BSD and Solaris variants. Notably, following GitHub's actions, no explanations or notifications were received, aside from reminders about payment obligations and attempts to charge fees.
Since the last news regarding the release of libmdbx v0.11.3, in addition to recovering from GitHub's actions, the following enhancements and fixes should be noted:
- A workaround for the detected incoherence issue/defect in the unified page and buffer cache in the Linux kernel has been added. In systems where the page and buffer cache are indeed unified, it makes no sense for the kernel to allocate memory for two copies of data when writing to a file already mapped in memory. Therefore, the written data becomes visible through memory mapping before the completion of the write() system call, even if the data has not yet been written to disk.
Overall, different behavior is unreasonable because with deferred merging, you still need to acquire locks for page lists, copy data, or adjust PTE. Therefore, the unwritten rule of coherence has been in effect since 1989 when unified buffer cache appeared in SRV4. Thus, identifying strange failures in overloaded scenarios with libmdbx required significant work. Initially, reproducing the problem was necessary, followed by verifying hypotheses and checking improvements.
It can now be confidently stated that the problem has been reliably identified, localized, and resolved, despite the complexity and specificity of the reproduction scenario. Additionally, the operation of the bypass mechanism was confirmed by one of the developers of Erigon (Ethereum), where in his case, the protection activation occurred as a regression due to an unnecessary assert check in a debug build.
It is important to note that in the context of widespread use of libmdbx in operational projects, ensuring reliable performance is fundamentally more important than determining whether it is an 'error or feature' and whether such coherence can be relied upon, especially not searching for the causes of incoherence within the Linux kernel. Thus, the focus here is on eliminating a problem that could affect users.
- The regression of the EXDEV (Cross-device link) error during hot database copying without compacting to another file system has been eliminated, both through the API and using the mdbx_copy utility.
- Kris Zyp implemented support for libmdbx in Deno. Kai Wetlesen packaged RPM for Fedora. David Bouyssié created bindings for Scala.
- The handling of the value set by the MDBX_opt_rp_augment_limit option during the processing of huge transactions in large databases has been corrected. Previously, due to a bug, unnecessary actions could occur, which sometimes affected performance in Ethereum implementations (Erigon/Akula/Silkworm) and Binance Chain projects.
- A number of issues have been fixed, including in the C++ API. Many build problems in rare and exotic configurations have been resolved. A complete list of all significant improvements is available in the ChangeLog.
- A total of 185 changes made across 89 files, with approximately 3300 lines added and about 4100 lines removed. More has been removed mainly due to cleaning up obsolete technological files related to GitHub and dependent services.
Historically, libmdbx is a significant rework of the LMDB database and surpasses its predecessor in terms of reliability, features, and performance. Compared to LMDB, libmdbx places great emphasis on code quality, stable API operation, testing, and automated checks. A database structure integrity checking utility with some recovery capabilities is included.
Technologically, libmdbx offers ACID compliance, strict change serialization, and non-blocking reads with linear scaling across CPU cores. It supports auto-compaction, automatic database size management, and range query estimation. Since 2016, the project has been funded by Positive Technologies and has been used in its products since 2017.
libmdbx provides a comprehensive C++ API, along with community-supported bindings for Rust, Haskell, Python, NodeJS, Ruby, Go, Nim, Deno, and Scala.
Source: opennet.ru
