The Dragonfly in-memory caching and data storage system has been released, which manipulates data in the key/value format and can be used as a lightweight solution to speed up the work of highly loaded sites, caching slow queries to the DBMS and intermediate data in RAM. Dragonfly supports the Memcached and Redis protocols, which allows you to use existing client libraries and transfer projects using Memcached and Redis to Dragonfly without reworking the code.
Compared to Redis, Dragonfly achieved a 2x performance improvement (6M requests per second) under typical workloads in an Amazon EC16 c25gn.3.8xlarge environment. Compared to Memcached in a c6gn.16xlarge AWS environment, Dragonfly was able to complete 4.7 times more write requests per second (3.8 million vs. 806k) and 1.77 times more read requests per second (3.7 million vs. 2.1 million).
In 5 GB storage tests, Dragonfly required 30% less memory than Redis. During the creation of snapshots by the βbgsaveβ command, memory consumption increases, but at peak moments it remained almost three times less than in Redis, and the snapshot write operation itself is much faster (in the test, a snapshot in Dragonfly was written in 30 seconds, and Redis - in 42 seconds).
High performance is achieved thanks to a multi-threaded architecture without sharing resources (shared-nothing), which implies that a separate separate handler with its own portion of data is attached to each thread, working without mutexes and spin-locks. Lightweight VLL locks are used to ensure atomicity when dealing with multiple keys. For efficient storage of information in memory, the dashtable structure is used, which implements a kind of partitioned hash tables.
Some features of Dragonfly:
- A caching mode that automatically replaces old data with new data after free memory is exhausted.
- Support for data-binding lifetimes during which data is considered up-to-date.
- Support for flushing the storage state to disk in the background for later recovery after a restart.
- The presence of an HTTP console (binds to TCP port 6379) for system management and an API for returning metrics, compatible with Prometheus.
- Support for 185 Redis commands, roughly equivalent to the functionality of the Redis 5 release.
- Support for all Memcached commands except CAS (check-and-set).
- Support for asynchronous operations for creating snapshots.
- Predictable memory consumption.
- Built-in Lua interpreter 5.4.
- Support for complex data types such as hashes, sets, lists (ZSET, HSET, LIST, SETS and STRING) and JSON data.
- Storage replication support for failover and load balancing.
The Dragonfly code is written in C/C++ and distributed under the BSL (Business Source License). The BSL was proposed by the MySQL co-founders as an alternative to the Open Core model. The essence of BSL is that the code of extended functionality is initially available for modification, but for some time it can be used free of charge only subject to additional conditions, which require the purchase of a commercial license to bypass. The additional license terms of the Dragonfly project mandate that the code be migrated to the Apache 2.0 license on March 15, 2028. Until that time, the license allows the use of the code only to ensure the operation of its services and products, but prohibits the use to create paid cloud services that act as an add-on to Dragonfly.
Dragonfly version 1.0 is notable for the implementation of support for data replication from the primary server to the secondary. At the same time, Dragonfly can be configured to be used as a secondary storage that accepts data from the primary server based on both Dragonfly and Redis. The replication management API is compatible with Redis and is based on the use of the ROLE and REPLICAOF (SLAVEOF) commands.
Source: opennet.ru