The release of the Dragonfly caching and in-memory data storage system has taken place. It manipulates data in a key/value format and can be used as a lightweight solution to speed up the operation of high-load websites, caching slow queries to the database management system and intermediate data in memory. Dragonfly supports Memcached and Redis protocols, allowing existing client libraries to be utilized and projects that use Memcached and Redis to be migrated to Dragonfly without code rework.
Compared to Redis, Dragonfly achieved a 25-fold increase in performance (3.8 million requests per second) under typical load conditions in an Amazon EC2 c6gn.16xlarge environment. In comparison with Memcached in an AWS environment c6gn.16xlarge, Dragonfly was able to execute 4.7 times more write requests per second (3.8 million versus 806 thousand) and 1.77 times more read requests per second (3.7 million versus 2.1 million).
In tests storing 5 GB of data, Dragonfly required 30% less memory than Redis. During snapshot creation with the 'bgsave' command, memory consumption increased, but at peak moments it remained almost three times lower than in Redis, and the snapshot writing operation itself was completed significantly faster (in the snapshot test, Dragonfly was written in 30 seconds, while Redis took 42 seconds).
High performance is achieved through a multithreaded architecture without resource sharing (shared-nothing), which implies that a separate isolated handler with its own portion of data is tied to each thread, operating without mutexes and spin locks. Lightweight VLL locks are used to ensure atomicity when working with multiple keys. An information storage structure called dashtable is utilized for efficient memory storage, implementing a type of partitioned hash tables.
Some capabilities of Dragonfly:
- Caching mode, which automatically ensures the eviction of old data for new entries when free memory is exhausted.
- Support for binding data to a lifespan, during which the data is considered valid.
- Support for background state dumps of the storage to disk for subsequent recovery after a restart.
- Availability of an HTTP console (bound to TCP port 6379) for managing the system and an API for providing metrics, compatible with Prometheus.
- Support for 185 Redis commands, which roughly corresponds to the functionality of Redis 5 release.
- Support for all Memcached commands except CAS (check-and-set).
- Support for asynchronous operations for creating snapshots.
- Predictable memory usage.
- Built-in Lua 5.4 interpreter.
- Support for complex data types such as hashes, sets, lists (ZSET, HSET, LIST, SETS, and STRING) and JSON format data.
- Support for storage replication to ensure fault tolerance and load balancing.
Dragonfly code is written in C/C++ and is distributed under the BSL (Business Source License). The BSL license was proposed by the co-founders of MySQL as an alternative to the Open Core model. The essence of BSL is that the code for extended functionality is initially available for modification, but for some time may be used for free only under additional conditions, which require the purchase of a commercial license to circumvent. Additional licensing conditions for the Dragonfly project stipulate a transition of the code to the Apache 2.0 license on March 15, 2028. Until that time, the license allows the use of the code only for providing the operation of one's services and products, but prohibits its application for creating paid cloud services that are an extension on Dragonfly.
Dragonfly version 1.0 is notable for implementing support for data replication from primary server to secondary. At the same time, Dragonfly can be configured to function as a secondary storage that receives data from a primary server based on either Dragonfly or Redis. The API for managing replication is compatible with Redis and is based on the use of ROLE and REPLICAOF (SLAVEOF) commands.
Source: opennet.ru
