Developers from Cloudflare about work to optimize the performance of disk encryption in the Linux kernel. As a result, they prepared for subsystem and Crypto API, which more than doubled the throughput for reading and writing in the synthetic test, as well as halving latency. When tested on real hardware, the overhead from encryption was reduced to almost the level observed when working with a disk without using data encryption.
Cloudflare uses dm-crypt to encrypt data on drives used to cache content on the CDN. Dm-crypt works at the block device level and encrypts I/O write requests and decrypts read requests, acting as a layer between the block device and the file system driver.
To evaluate the performance of dm-crypt using the package The speed of working with encrypted and unencrypted partitions on a RAM disk located in RAM was measured to eliminate disk performance fluctuations and focus on code performance. For unencrypted partitions, read and write performance was kept at 1126 MB / s, but when encryption was enabled, the speed decreased 7 times and amounted to 147 MB/s.
Initially, there was a suspicion of using inefficient algorithms in the kernel cryptosystem. But the tests used the fastest aes-xts algorithm with 256 encryption keys, the performance of which when running the "cryptsetup benchmark" is more than twice as high as the result obtained when testing the RAM disk. Experiments with the dm-crypt flags for performance tuning did not give any result: when using the "--perf-same_cpu_crypt" flag, the performance even decreased to 136 MB/s, and when the "--perf-submit_from_crypt_cpus" flag was specified, it increased only to 166 MB/s.
A deeper analysis of the logic of work showed that dm-crypt is not as simple as it seems - when a write request is received from the FS driver, dm-crypt does not process it immediately, but places it in the βkcryptdβ queue, which is not parsed immediately, but upon the occurrence convenient moment. From the queue, the request is sent to the Linux Crypto API to perform encryption. But since the Crypto API uses an asynchronous execution model, encryption is also not performed immediately, but bypassing another queue. After encryption is complete, dm-crypt may attempt to sort pending write requests using a search tree . At the end, a separate kernel thread again, with a certain delay, picks up the accumulated I / O requests and sends them to the block device stack.
When reading first, dm-crypt adds a request to the "kcryptd_io" queue to get data from the drive. After some time, the data becomes available and placed in the "kcryptd" queue for decryption.
Kcryptd sends a request to the Linux Crypto API, which decrypts the information asynchronously. Requests do not always pass through all queues, but in the worst case scenario, a write request settles in queues up to 4 times, and a read request up to 3 times. Each hit in the queue introduces delays, which are the key reason for the significant performance degradation of dm-crypt.
The use of queues is due to the need to work in the conditions of interruptions. In 2005, when the current queue-based dm-crypt model was implemented, the Crypto API was not yet asynchronous. After the transfer of the Crypto API to an asynchronous execution model, essentially double protection began to be applied. Queues were also introduced to save consumption of the kernel stack, but after its increase in 2014, the optimization data lost its relevance. An additional "kcryptd_io" queue was introduced to overcome a bottleneck that causes memory allocation waits when a large number of requests come in. In 2015, a sorting phase was additionally introduced, since encryption requests on multiprocessor systems could be completed out of order (instead of sequential disk access, access was performed in random order, and the CFQ scheduler did not work efficiently). Currently, sorting has lost its meaning with SSDs, and the CFQ scheduler is no longer used in the kernel.
Given that modern drives have become faster and smarter, the resource allocation system in the Linux kernel has been revised and some subsystems have been redesigned, Cloudflare engineers dm-crypt has a new mode of operation, free from the use of unnecessary queues and asynchronous calls. The mode is enabled by a separate flag "force_inline" and brings dm-crypt to the form of a simple proxy that encrypts and decrypts incoming requests. Interaction with the Crypto API has been optimized by the explicit choice of encryption algorithms that work in synchronous mode and do not use request queues. For synchronous work with Crypto API was a module that allows you to use FPU / AES-NI for acceleration and directly forwards encryption and decryption requests.
As a result, when testing a RAM disk, we managed to more than double the performance of dm-crypt - performance increased from 294 MB / s (2 x 147 MB ββ/ s) to 640 MB / s, which is very close to the performance of bare encryption (696 MB /s).
In load testing on real servers, the new implementation showed performance very close to the configuration running without encryption, and enabling encryption on servers with Cloudflare cache did not affect the response speed in any way. In the future, Cloudflare plans to release the prepared patches into the main Linux kernel, but before that they will need to be reworked, as they are optimized for a certain workload and do not cover all areas of application, for example, encryption on low-power embedded devices.
Source: opennet.ru