High Read Latency for VM on iSCSI Storage

[satish_halemani]

Hi,

We have a 2012 Domain Controller (4 vCPU/6GB Memory) VM on iSCSI storage (NetApp) configured using Software iSCSI adapter. The ESXi host is on version 5.5 and HP Gen9 hardware. No iSCSI network bindings configured and no jumbo frames as well as storage doesn't support it. The link speed is 1GB. 2 NICs are configured in Active/Active mode for the iSCSI vmkernel port.

The domain controller takes nearly 1hr to boot after each reboot. We checked the storage performance (on VM/Datastore/ESXi host) using vCenter and also ESXTOP. We observed that the Read Latency was spiking to over 250ms at regular intervals. Write latency is within acceptable limits(15ms). The strange thing is the Write Rate for the VM is less than Read Rate. No packet drops or high latency when we ping to the storage. Storage team also checked and reported no issues from their end.

Please suggest what could be the issue and what we can check further to isolate the issue.

Thanks in advance.

Regards,

Satish

[cyberpaul · Accepted solution]

Hi Satish,

this sounds a lot like the "delayed ack issue". For some iSCSI arrays it is advisable to disable Delayed ACK feature otherwise you might experience high read latency. I've seen this with Dell EqualLogic arrays.

Unfortunately, it is a struggle to turn that feature off once it has been enabled (and it is enabled by default). There are some blog posts on how to do it, like this one:

Disable Delayed Ack for iSCSI on ESXi, not always working | Virtual Allan

Also, please make sure there is no IO limit set for the given VM. IO scheduler in vSphere 5.5 behaves terribly with IO limit set and disk latency can spike a lot. If this is the case, either set the IO limit to "unlimited" or switch back to 5.1 scheduler. I can provide more detailed information if this is the case.

Let us know how it looks!

Pavel

[MBreidenbach0]

Is the NetApp 7-mode or Clustered Data ONTAP ? iSCSI Multipathing is very different for these two NetApp OS variants.

You may want to read NetApp Technical Reports:

TR 3749 (7-mode + vSphere)

TR 4068 (cDOT + vSphere 5)

TR 4333 (cDOT + vSphere 6)

TR 4597 (ONTAP + vSphere)

[satish_halemani]

Thanks for the suggestion. It did turn out that the was replication running on the storage end which was causing the issue.