Jens Axboe, creator of io_uring and the CFQ, Deadline, and Noop I/O schedulers, proposed a patch for inclusion in the QEMU emulator codebase that reduces latency during fdmon (file descriptor monitoring) in "aio=io_uring" mode by 50-80 times when the system is idle.
The problem was manifested due to the translation of the ppoll() operation into a sleep state with a timeout of 499 ms, despite the presence of I/O. To resume the main event processing loop, which was suspended due to ppoll(), a patch was proposed that adds a call to the aio_notify() function in the SQE (Submission Queue Entry) creation function.
The issue arose during regression testing of io_uring with virtual machines different block devices. Jens noted the random occurrence of timeouts when using AHCI/SATA devices in "aio=io_uring" mode, while tests with virtio-blk or nvme devices consistently completed successfully in about a second. It is noted that the problem affects all types of block devices, but for AHCI/SATA devices, latency is most pronounced due to the use of MMIO.
Jens also described his experience debugging the problem using the AI assistant Claude. After identifying a scenario that reproduced the conditions for a timeout, he provided Claude with the existing debugging data, gave access to the virtual machine, and suggested determining the likely causes of the identified failure.
Claude decided to check if the performance would slow down when using a virtio-blk device and ran a destructive scenario provided by the developer that reproduced the issue. During the test, the first 128 MB of content from the block device /dev/vda in the virtual machine was deleted. After that, Claude concluded that the problem was not with virtio-blk. When Jens pointed out to the AI assistant the deletion of part of the content from /dev/vda, it responded, "Yes, I did that," and after a request to fix it, restored the functionality of the virtual disk /dev/vda. It is noted that using the AI assistant helped to better understand the nuances of various event processing cycles in QEMU.
It is noteworthy that the issue was quite difficult to detect, as the slowdown is not recorded in synthetic tests due to the fact that the occurrence of the failure is influenced by the awakening of the event processing cycle with ppoll due to other activity, and synthetic I/O tests do not process the received data. The slowdown became more noticeable with the addition of several calls to usleep() to simulate data processing.
Before the fix on the idle system: time sudo ./iotest /dev/sda Executed in 25.76 secs fish external usr time 6.19 millis 783.00 micros 5.41 millis sys time 12.43 millis 642.00 micros 11.79 millis
After the fix on the idle system: time sudo ./iotest /dev/sda Executed in 1.30 secs fish external usr time 2.14 millis 0.14 millis 2.00 millis sys time 16.93 millis 1.16 millis 15.76 millis
Source: opennet.ru
