VMware Cloud Community
BearHuntr
Contributor
Contributor
‎07-15-2010 11:13 AM

Weird Physical Device Read/Write Latency spikes.

So, let me preface this with the fact that I am by no means an expert in SAN and storage technology, so bear with me if I'm not quite explaining this correctly.

When looking at the performance reports in vCenter for a cluster of 4 ESX hosts, we are seeing a repetitive spike in Physical Device Read and Write Latency. It seems to happen about every 8.5 hours and seems to be affecting some of our VM servers and users are reporting slowness during these times. This is also viewable in Analyzer reports on the SAN (EMC Clariion CX3-80) for any VM Datastore LUNs. It does not show up for any Physical Windows or UNIX boxes that are connected to the SAN. It also shows up at the same time for our VDI environment, which is a completely different set and type of disks on the same SAN and a separate cluster of 2 ESX hosts with its own vCenter server. Attached is a an example of what we see. I'm not sure if anyone can help me with this or if it is even something that is considered a problem, but we came across this in our troubleshooting efforts and it seems completely odd. Thanks in advance for any help you can offer!!

Scott.!file:///C:/Users/segber/AppData/Local/Temp/moz-screenshot.png!

Preview file
150 KB
0 Kudos
2 Replies
BearHuntr
Contributor
Contributor
‎07-29-2010 06:28 AM

It turned out that these spike were apparently caused by Symantec Endpoint Protection downloading updates from the AV server all at the same time on every VM server. I changed the Symantec Communication policy settings to check for updates every 24 hours instead of contantly and set the randomization to 6 hours so that they would be staggered.

0 Kudos
JMB11
Contributor
Contributor
‎07-29-2010 07:04 AM

That's a sensible, practical workaround.

The fundamental problem you've run into is that A/V software wasn't originally designed for virtualized environments -- it assumes it's got free reign to do heavy local disk I/O, which obviously doesn't work well in a VDI / SAN situation.

This is a well-known problem -- for some background, see for example http://virtualization.info/en/news/2010/07/vmware-to-release-an-antivirus-framework-trendmicro-worki....

0 Kudos