I have problems finding the root cause to why several of our VM's have extreme latency on IO. Example
Our virtual Exchange 2003 server have Average Disk sec/Reads and Disk sec/writes with latency on about 60-80ms. Normal peaks are 200ms, with the occational peak over 1000ms.
A backup attempt of a virtual fileserver (2003 server) were aborted when total bandwith never came over 1.2MB/s
vmkernel logs show a lot of abort and async io error messages
Jan 16 19:01:27 esx1 vmkernel: 4:04:58:31.887 cpu0:1028)SCSI: 3731: AsyncIO timeout (5000); aborting cmd w/ sn 133580, handle 3044/0x7a02a40 Jan 16 19:01:27 esx1 vmkernel: 4:04:58:31.887 cpu0:1028)LinSCSI: 3616:
Aborting cmds with world 1024, originHandle 0x7a02a40, originSN 133580 from
vmhba0:2:3
Jan 16 19:01:27 esx1 vmkernel: 4:04:58:31.887 cpu0:1028)LinSCSI: 3632: Abort failed for cmd with serial=133580, status=bad0001, retval=bad0001
These messages indicate that the ESX is waiting 5000ms for a response from the storage. When it gets none it sends an abort to storage. However this fails as well. ESX then sends a reset and this finally is successful.
Our virtual environment consist of two HP DL servers with QLA4052 iSCSI HBA's connected to the SAN through a HP Ethernet switch. SAN consist of 11 7200rpm FCATA disks. Most vmkfs volumes are RAID5 with 9 disks or RAID10.
We also have a test environment (put together to resolve this issue) of one HP DL server with both iSCSI and FC HBA. Both test and production run ESX v 3.0.2. All hardware is on the vmware HCL.
I know the storage is not enterprise class, but we have a standard diskless windows 2003 server that boots from SAN and have all its disks there, and it perfoms on par or better than the servers that have internal 10K rpm RAID10 SAS disks. A review by the vendor also show that the SAN is lightly loaded with peaks at about 80% of the IOPS it can handle.
When monitoring filetransfers between the vm's and ohysical servers we see behaviour we dont understand and dont see on physical to physical server filetransfers;
# Smaller files (less than 400MB) uses about 12% of the available 1GB bandwith on the NIC througout the transfer.
# Larger files (1GB+) start out at same level, but within a short time (30-120sec) transfer drops to 1%. Transfer speed gradually picks up but are never close to 12%.
To me this looks like an issue with ESX 3.0.2 and the SAN, since 3.0.2 vm's perform very well when connected to something else than the SAN, and the SAN performes well with servers not running ESX 3.0.2. Am I the only one that have ssen this behaviour?
