Micron Technology, a company specializing in the production of DRAM and flash memory, new storage engine (Heterogeneous-memory Storage Engine), designed for use on SSDs based on NAND flash (X100, TLC, QLC 3D NAND) or permanent memory (NVDIMM). The engine is made in the form of a library for embedding into other applications and supports data processing in the key-value format. HSE code is written in C language and licensed under Apache 2.0.
Of the areas of application of the engine, it is used for low-level data storage in NoSQL DBMS, software storages (SDS, Software-Defined Storage) such as Ceph and Scality RING, platforms for processing large amounts of data (Big Data), high-performance computing (HPC), Internet devices of things (IoT) and solutions for machine learning systems.
HSE is optimized not only for maximum performance, but also for longevity across different classes of SSDs. High speed is achieved through a hybrid storage model - the most up-to-date data is cached in RAM, which reduces the number of accesses to the drive. As an example of integrating a new engine into third-party projects a variant of the document-oriented DBMS MongoDB, translated to use HSE.
Technologically, HSE relies on an additional kernel module , which implements a specialized object storage interface for solid-state drives, taking into account their capabilities and features, which allows you to get fundamentally different performance and durability characteristics. Mpool is also a development of Micron Technology, open at the same time as HSE, but separated into an independent infrastructure project. Mpool assumes the use ΠΈ , but only traditional SSDs are currently supported.
Performance testing with a package (Yahoo Cloud Serving Benchmark) showed significant performance gains when using 2TB storage with 1KB block processing. A particularly significant increase in performance is observed in the test with a uniform distribution of read and write operations (test "A" on the graph).
For example, MongoDB with the HSE engine turned out to be about 8 times faster than the version with the standard WiredTiger engine, and the HSE engine overtook the RocksDB DBMS by more than 6 times. Excellent performance is also seen in tests that involve 95% of reads and 5% of changes or additions (tests "B" and "D" in the graphs). The "C" test, which assumes only read operations, shows a gain of about 40%. The increase in the survivability of SSD drives during write operations compared to the solution based on RocksDB is estimated at 7 times.
Key features of HSE:
- Support for generic and extended operators for handling data in key/value format;
- Full support for transactions and with the ability to isolate storage slices through the creation of snapshots (snapshots can also be used to maintain independent collections in one storage);
- Ability to use cursors to traverse data in snapshot-based views;
- Data model optimized for mixed load types in one storage;
- Flexible storage reliability management mechanisms;
- Customizable data orchestration schemes (distribution across different types of memory present in the storage);
- A library with a C API that can dynamically link to any application;
- Ability to scale up to terabytes of data and hundreds of billions of keys in storage;
- Efficient processing of thousands of parallel operations;
- Significant increase in throughput, lower latency and increased write / read for various types of load compared to typical alternative solutions;
- The ability to use different SSD classes in the same storage to optimize performance and durability.
Source: opennet.ru