The first release of the Dragonfly in-memory data caching system is available, supporting Memcached and Redis protocols while allowing for much higher performance and lower memory consumption for queries. The system manipulates data in a key/value format and can be used as a lightweight solution to enhance the performance of high-traffic websites by caching slow database queries and intermediate data in memory. The Dragonfly code is written in C/C++ and is distributed under the BSL (Business Source License).
The BSL license was proposed by the co-founders of MySQL as an alternative to the Open Core model. The essence of BSL is that the code with extended functionality is initially available for modification, but may be used for free for a limited time only under additional conditions, bypassing which requires purchasing a commercial license. Additional licensing conditions for the Dragonfly project stipulate that the code will transition to the Apache 2.0 license only on June 1, 2027. Until that time, the license allows the code to be used solely for the operation of its own services and products, but prohibits its application for creating paid cloud services that serve as an extension of Dragonfly.
According to the developers and demonstrated tests, Dragonfly claims to be the fastest in-memory storage system. Compared to Redis, Dragonfly has achieved a 25-fold increase in performance and a threefold reduction in memory consumption under typical load conditions. server Dragonfly can handle millions of requests per second; for instance, in the Amazon EC2 c6gn.16xlarge environment, it achieved a performance of 3.8 million requests per second.
In tests storing 5 GB of data, Dragonfly required 30% less memory than Redis. During snapshot creation using the 'bgsave' command, memory consumption increases, but at peak moments, it remains nearly three times lower than Redis, and the snapshot writing operation itself is performed significantly faster (in the test, the snapshot in Dragonfly was written in 30 seconds, while Redis took 42 seconds).
High performance is achieved through a multithreaded architecture without resource sharing (shared-nothing), which implies that a separate isolated handler with its own portion of data is tied to each thread, operating without mutexes and spin locks. Lightweight VLL locks are used to ensure atomicity when working with multiple keys. An information storage structure called dashtable is utilized for efficient memory storage, implementing a type of partitioned hash tables.
The features available in the first release include support for the RESP2 protocol and 130 Redis commands, which roughly corresponds to the functionality of the Redis 2.8 release. Additionally, Dragonfly supports all memcached commands except for CAS (check-and-set), provides asynchronous operations for snapshot creation, offers predictable memory consumption, includes a built-in Lua 5.4 interpreter, and supports complex data types such as hashes, sets, and lists (ZSET, HSET, LIST, SETS, and STRING).
A separate caching mode is available, which automatically evicts old data with new data after free memory is exhausted. It is possible to bind a time-to-live to the data, during which the data is considered valid. The storage state can be flushed to disk in the background for subsequent recovery after a restart. An HTTP console (bound to TCP port 6379) and an API for delivering metrics compatible with Prometheus are provided for system management. Future versions will expand support for Redis commands and implement storage replication capabilities for fault tolerance and load balancing.
Source: opennet.ru
