I very very regularly have nvme timeout on me in a guest, for instance with
[ 14.310896] NET: Registered PF_QIPCRTR protocol family
[ 40.388311] nvme nvme0: I/O 128 QID 4 timeout, completion polled
[ 40.388343] nvme nvme0: I/O 96 QID 5 timeout, completion polled
[ 40.388400] nvme nvme0: I/O 32 QID 15 timeout, completion polled
[ 70.595937] nvme nvme0: I/O 96 QID 1 timeout, completion polled
[ 70.595969] nvme nvme0: I/O 129 QID 4 timeout, completion polled
[ 70.904742] vmxnet3 0000:03:00.0 ens160: intr type 3, mode 0, 9 vectors allocated
(from `dmesg`; when reproducing, simply `dmesg --level warn --follow`
This adds a delay of 30 seconds to file system operations.
This seems to be new ever since I attempted to maximize I/O performance, by
- using the VMware NVMe virtual controller
- with independent disks
- and having that on a very fast SSD
This is somewhat reproducible for me in two scenarios
- boot Linux guest (see above for example)
- run the very fast ripgrep (`rg`) on the full depth of the file system (e.g. `cd / ; rg x > /dev/null`, to hit plenty of "things" for heavy I/O
Each of these timeouts means waiting for 30 seconds. That is not at all good.
The host operating system at no time issues any warning or errors.
Hardware (host):
- Dell Inspiron 7610 notebook == Tigerlake 8 core / 16 execution units
- 64 GB of memory
- moved OEM SSD to secondary PCI 3.0 slot
- installed 2 TB WD SN850X PCI 4.0 SSD into primary PCI 4.0 slot - this combo of SSD and PCI slot is about the fastest you can get in a simple laptop
Software (host):
- Windows 11 Pro (fully up-to-date)
- VMware Workstation 17.0.2
Hardware (guest):
- 32 GB of memory
- 16 cores
- independent NVMe pointing to the fast WD SN850X disk
Software (guest)
- Fedora Linux 38 (== 6.2.13-300.fc38.x86_64, but this happened with earlier kernels, too)
- ... anything that puts load onto the I/O subsystem, e.g. starting the KDE desktop, running ripgrep ...
How can I fix the timeouts?
