Jens Axboe, creator of io_uring and the CFQ, Deadline, and Noop I/O schedulers, proposed a patch for inclusion in the QEMU emulator codebase that reduces latencies by 50-80 times when using fdmon (file descriptor monitoring) in "aio=io_uring" mode and when the system is in an idle state.
The issue arose due to the translation of the ppoll() operation into a sleep state with a timeout of 499 ms, despite the presence of input/output. To resume the main event processing loop, which was being paused due to ppoll(), a patch was proposed that adds a call to the aio_notify() function in the SQE (Submission Queue Entry) creation function.
The problem emerged during regression testing of io_uring with virtuálne mašini various block devices. Jens noticed random timeouts occurring when using AHCI/SATA devices in "aio=io_uring" mode, while configurations with virtio-blk or nvme devices always successfully completed tests in about a second. It is noted that the issue affects all types of block devices, but the appearance of delays is most pronounced for AHCI/SATA devices due to the use of MMIO.
Jens also described his experience debugging the problem using the AI assistant Claude. After defining the scenario that reproduced the conditions for a timeout, he provided the existing debug data to Claude, granted access to the virtual machine, and suggested identifying the probable causes of the detected failure.
Claude decided to check if performance would slow down when using a virtio-blk device and ran the proposed destructive scenario that reproduced the problem. During the test, the first 128 MB of content was deleted from the block device /dev/vda in the virtual machine. After this, Claude concluded that the problem was not with virtio-blk. When Jens pointed out to the AI assistant the deletion of part of the content from /dev/vda, it responded, "Yes, I did that," and after being requested to fix it, restored the operability of the virtual disk /dev/vda. It is noted that using the AI assistant helped to better understand the specifics of various event processing cycles in QEMU.
Este demn de menționat că problema a fost destul de greu de descoperit, deoarece în testele sintetice, întârzierile nu sunt înregistrate datorită influenței asupra apariției erorilor, generat de trezirea ciclului de procesare a evenimentelor cu ppoll din cauza unei alte activități, iar testele sintetice de intrare/ieșire nu procesează datele primite. Întârzierea a devenit mai vizibilă odată cu adăugarea mai multor apeluri usleep() pentru a simula procesarea datelor.
După corectare, în sistemul în stare de inactivitate (idle): time sudo ./iotest /dev/sda Executat în 25.76 secs fish extern usr time 6.19 millis 783.00 micros 5.41 millis sys time 12.43 millis 642.00 micros 11.79 millis
După corectare, în sistemul în stare de inactivitate: time sudo ./iotest /dev/sda Executat în 1.30 secs fish extern usr time 2.14 millis 0.14 millis 2.00 millis sys time 16.93 millis 1.16 millis 15.76 millis
Sursa: opennet.ro
