Poor storage performance is generally the result of high I/O latency. vCenter or esxtop will report the various latencies at each level in the storage stack from the VM down to the storage hardware. vCenter cannot provide information for the actual latency seen by the application since that includes the latency at the Guest OS and the application itself, and these items are not visible to vCenter. vCenter can report on the following storage stack I/O latencies in vSphere.
Storage Stack Components in a vSphere environment
GAVG (Guest Average Latency) total latency as seen from vSphere
KAVG (Kernel Average Latency) time an I/O request spent waiting inside the vSphere storage stack.
QAVG (Queue Average latency) time spent waiting in a queue inside the vSphere Storage Stack.
DAVG (Device Average Latency) latency coming from the physical hardware, HBA and Storage device.
To provide some rough guidance, for most application workloads (typically 8k I/O size, 80% Random, 80% Read) we generally say anything greater than 20 to 30 ms of I/O Latency may be a performance concern. Of course as with all things performance related some applications are more sensitive to I/O latency then others so the 20-30ms guidance is a rough guidance rather than a hard rule. So we expect that GAVG or total latency as seen from vCenter should be less than 20 to 30 ms. as seen in the picture, GAVG is made up of KAVG and DAVG. Ideally we would like all our I/O to quickly get out on to the wire and thus spend no significant amount of time just sitting in the vSphere storage stack, so we would ideally like to see KAVG very low. As a rough guideline KAVG should usual be 0 ms and anything greater than 2ms may be an indicator of a performance issue.
So what are the rule of thumb indicators of bad storage performance?
• High Device Latency: Device Average Latency (DAVG) consistently greater than 20 to 30 ms may cause a performance problem for your typical application.
• High Kernel Latency: Kernel Average Latency (KAVG) should usually be 0 in an ideal environment, but anything greater than 2 ms may be a performance problem.
So what can cause bad storage performance and how to address it, well that is for next time…
And as a side note: Check out your local VMUG (VMware User Group). The VMUG community has more than 75,000 members with more than 180 local groups across 32 countries. Many local area VMUGS have free user conferences that are a great opportunity to learn from in-depth technical sessions, demonstrations and exhibits and network with other VMware customers and partners. That is where I’ll be this week, presenting Storage Troubleshooting and Performance Best Practices at the Denver VMUG.
Hey FB,
There should be a place somewhere in unisphere that tells you what your Disk pool's raid system is. It probably is Raid5 or Raid6
What I ment by the backplane is this:
Now when you create a LUN you have to assign it to a ServiceProcessor on the SAN. This Service Process will serve any request to that LUN always unless a failure occurs. So in this case I made all odd numbers assigned to SPA in the diagram, and all even numbers assigned to SPB. So if a request came in for DatastoreSAS01 to say save a file a VM running on DatastoreSAS01, the request would select the path that would lead to SPA then it would save the file in one of the disk drawers where that VM's files are located.
However in the event that SPA Fails, SPB spins up another virtual service controller per say, or takes over all of SPA's LUN responsibilities so there is no outage or downtime. However to gain access to the drawers that SPA is connect too, you can see there is no direct connection. Each SP SAS cables go up to the drawers accordingly. However each drawer and SP has a backplane which connects them so you can still gain access to the Disks in the event of a failure.
So in this picture I'm exaggerating a bit as it woudln't have to go up the entire shelf to get to the other side, it would simply find the best backplane to take, however we found when testing the VNXe3100-3150's that transfer speeds dropped by more then 50% when you lost a SP. Our speeds went from 100mbp/s transfering files between VM's to 30mbp/s when transfering file. It something you can test pretty easily just pull one of your SP into maintence mode during an outage windows and you can test this as well:
Now going back to my original comment, if for some reason your pathing was not working "correctly" and it was having to transverse the backplane somehow to get to LUNS that SPA controls through SPB becuase that is the path it took it could count for some of your latency, however in most cases as long as the connections are setup correctly VMware's Multipathing selects the proper path assuming your settings are to what the vendor reccoemends which you have already double checked.
Thanks Fpicabia and let us know how it works out as I'm curious to the resolutions 
Just out of curiosity do any of your ESXi hosts have any SSD disks? as if you get the SAN working better but it still falls short of your performance expectations its possible you could look into a local caching option to help spare the SAN some IOPS and drop the latency. Anyhow good luck! keep us posted