VMware Cloud Community
Marc_P
Enthusiast
Enthusiast

Event: Device Performance has deteriorated. I/O Latency increased

Hi,

Since upgrading to vSphere 5 I have noticed the following errore in our Events:

Device naa.60a980004335434f4334583057375634 performance has deteriorated. I/O latency increased from average value of 3824 microseconds to 253556 microseconds.

This is for different devices and not isolated to one.

I'm not really sure where to start looking as the SAN is not being pushed as these messages even appear at 4am in the morning when nothing is happening.

We are using a NetApp 3020C SAN.

Any help or pointers appreciated.

64 Replies
olaf_eichorn
Contributor
Contributor

Hi folks, had the same problems on local store on a brand new Dell-Server.
I found this article
http://www.vmdamentals.com/?p=2052
from Eric Zandboer and changed Disk.DiscMaxIOSize from default 32767 Kb to 128 Kb
- and this was it!

Reply
0 Kudos
JustinCooper
Contributor
Contributor

I did the same thing to address the issue, but the ii latency issues have returned. The host reboot is most likely what alleviated it, but for what reasons I am not entirely sure. It could be a failover issue in my iSCSI port groups that gets corrected when the connections are refreshed.

Reply
0 Kudos
The_Outsider
Contributor
Contributor

I too have been experiencing this error, of recent in a VMware 5.1 environment - and it started after the systems were brought up after a power outage.

What I noticed were - the messages largely refer to SATA based LUNs. And there is hardly any traffic on them.

I see this discussion thread has been active for sometime, with no concrete conclusion other than the subtle suggestion of ignoring the alerts.

Reply
0 Kudos
dwilliam62
Enthusiast
Enthusiast

For Equallogic iSCSI SAN there is now a best practices guide for ESX deployments.

http://en.community.dell.com/techcenter/extras/m/white_papers/20434601/download.aspx

Re: Local storage.  I would strongly suggest upgrading to a l current version of ESX v5.0.  (Or better 5.1).   The build numbers are way over 1 million compared to the 400K you reported.

A common cause of latency is having multiple VMDKs on a single Virtual SCSI adapter inside the VM.   ESX allows up to 4x Virtual adapters per VM.   More adapters mean more concurrent IO operations are possible.  A single adapter can only talk to one "drive" / VMDK at a time.  If you have more than 4x, don't put your busiest disks on a single controller.  Spread them out over the 4x adapters.  

If you search for Virtual SCSI adapters and paravirtual SCSI adapters you will find more info.

Regards,

Reply
0 Kudos
ErwinInfracontr
Contributor
Contributor

Hi, experiencing the same issue. This problem occurs when copying something from inside the VM. I copy a file, which is 3GB, from C:\ to C:\Temp inside the OS of the VM. Then the warnings start to appear. I also experienced this when installing/upgrading the VMware Tools of the VM. It is really frustrating, because I absolutely see no slow performance inside the VM's. 

We upgraded firmware, OS, drivers of all parts. No result.

Erwin

Reply
0 Kudos