Developers from Cloudflare discussed their work on optimizing the performance of disk encryption in the Linux kernel. As a result, they prepared for the subsystem and Crypto API, which allowed them to more than double the throughput in synthetic tests for reading and writing, as well as reduce latencies by half. When tested on actual hardware, the overhead from encryption was reduced to almost the level observed when operating with a disk that does not use data encryption.
Cloudflare uses dm-crypt for data encryption on the storage devices used for content caching in its CDN network. Dm-crypt operates at the block device level and performs encryption for write input/output requests and decryption for read requests, acting as a layer between the block device and the file system driver.
To assess the performance of dm-crypt using the package measurements were taken of the performance with encrypted and unencrypted partitions on a RAM disk, located in RAM to eliminate fluctuations in disk performance and focus on code performance. For unencrypted partitions, the read and write performance remained at 1126 MB/s, but when encryption was enabled, the speed dropped by 7 times to 147 MB/s.
Initially, there was suspicion of ineffective algorithms being used in the kernel's cryptosystem. However, the tests utilized the fastest aes-xts algorithm with a 256-bit encryption key, whose performance during the 'cryptsetup benchmark' was more than twice that of the results obtained when testing the RAM disk. Experiments with dm-crypt flags for performance tuning resulted in no improvement: using the flag '--perf-same_cpu_crypt' even reduced performance to 136 MB/s, and specifying the flag '--perf-submit_from_crypt_cpus' only increased it to 166 MB/s.
A deeper analysis of how it works revealed that dm-crypt is not as simple as it seems — when a write request is received from the FS driver, dm-crypt does not process it immediately but places it in the 'kcryptd' queue, which is not processed right away, but rather at a more convenient time. From the queue, the request is sent to the Linux Crypto API for encryption. However, since the Crypto API uses an asynchronous execution model, encryption is also not performed immediately, passing through another queue. Once encryption is complete, dm-crypt may attempt to sort the pending write requests using a search tree. . At the end, a separate kernel thread again picks up the accumulated input/output requests with a certain delay and sends them to the block device stack.
When reading, dm-crypt initially adds a request to the 'kcryptd_io' queue to retrieve data from the storage device. After some time, the data becomes available and is placed in the 'kcryptd' queue for decryption.
Kcryptd sends a request to the Linux Crypto API, which decrypts the information in asynchronous mode. Requests do not always get through all queues, but in the worst-case scenario, a write request can sit in queues up to 4 times, while a read request can sit up to 3 times. Each entry into a queue leads to delays, which are the key reason for the significant drop in dm-crypt performance.
The use of queues is driven by the need to work under interruption conditions. In 2005, when the current queue-based model of dm-crypt was implemented, the Crypto API was not yet asynchronous. After transitioning the Crypto API to an asynchronous execution model, a form of dual protection became essential. Queues were also introduced to save kernel stack consumption, but this optimization became less relevant after its increase in 2014. An additional queue, 'kcryptd_io,' was introduced to overcome the bottleneck that led to memory allocation wait times when handling a high number of requests. In 2015, a sorting phase was added because encryption requests on multiprocessor systems could complete out of the order in which they were sent (instead of sequential disk access, requests were accessed randomly, and the CFQ scheduler also did not work efficiently). Currently, using SSDs has made sorting unnecessary, and the CFQ scheduler is no longer used in the kernel.
Considering that modern drives have become faster and smarter, the resource allocation system in the Linux kernel has been re-evaluated, and some subsystems have been redesigned, engineers from Cloudflare in dm-crypt, a new operating mode has been implemented that eliminates unnecessary queues and asynchronous calls. This mode is enabled with a separate 'force_inline' flag, turning dm-crypt into a simple proxy that encrypts and decrypts incoming requests. Interaction with the Crypto API has been optimized by explicitly selecting encryption algorithms that operate in synchronous mode and do not use request queues. For synchronous operation with the Crypto API, a module has been introduced that allows the use of FPU/AES-NI for acceleration and directly passes through encryption and decryption requests.
As a result, testing with a RAM disk managed to more than double the performance of dm-crypt — performance increased from 294 MB/s (2 x 147 MB/s) to 640 MB/s, which is very close to the performance of raw encryption (696 MB/s).
During load testing on real servers, the new implementation demonstrated performance very close to the configuration operating without encryption, and enabling encryption on servers with Cloudflare's cache had no impact on response time. In the future, Cloudflare plans to submit the prepared patches for inclusion in the main Linux kernel, but they will need to be revised first, as they are optimized for specific loads and do not cover all use cases, such as encryption on low-powered embedded devices.
Source: opennet.ru
