Please note that totalTime is the time from when the command entered the VMkernel queue till it completed while issueTime is the time the I/O spent only in the VMkernel queue. So if we subtract the issueTime from the totalTime we get the time the I/O spent in the driver which is what you want here.
Also, it should be noted that these values are in CPU cycles (TSC values), so you need to divide with your CPU speed (e.g. 3000000000 for a 3Ghz CPU) to get it in seconds.
Lastly, please note that figuring what's really your bottleneck can be really hard, since queueing is expected if the VM's are issuing a lot of I/O's. Here are a few things to keep in mind:
\- The bottom line (of th /proc node(s) SV mentioned) shows 'active' and 'queued' values (for the given instant when you look at the proc node). The active value reflects how many commands are active in the driver for this LUN (and it can never go past the queue depth reported at the very top). The queued value tells you how many I/O's are issued that had to be queued in the VMkernel, and so queued+active is the amount of total I/O outstanding from all your VM's. If your system is busy it should show a (almost) constantly max'ed active value.
\- If the queued value is high due to the VM's constantly pushing lot's of I/O's that fine, but if it is constantly high due to bad performance from your storage it is bad - so you need to look at the average I/O turnaround time (the time spent in the driver from above) and try to determine whether that number is too high or too low, but really you also want to consider the average I/O sizes here (which is "(KBwritten + KBread) / cmds").
\- Note that writes are typically faster than reads due to caching.
\- There is a HUGE difference between sequential and random I/O's and you may not know much about what is issued (and likewise with the I/O size as mentioned above). It is therefore preferrable to test performance with VM's running a benchmark like 'iometer' on an otherwise idle system. Here you can pick the I/O size(s) and pattern (random vs. sequential percentages) so you take some of the guesswork out of the equation.
I am sure this did not make your life any easier, but doing fair and serious benchmarking is actually a fine art and without doing it carefully you may get less than ideal results and jump to the wrong conclusions.
As a very last thing, in case you find a performance issue, be aware that putting many VM's with large disks on the same (huge) VMFS may actually hurt performance because the I/O pattern (as seen from the array) is random I/O's scattered \_far_ apart on the volume (since a VM with a large disk tends to concentrate I/O to only a small part of it). If that is the case you may benefit from grouping the disks in the array into several smaller RAID groups and then have a VMFS with less VM's per volume.
/Thor
NB: I work for VMware, but opinions expressed above are my own and not necessarily those of my employer.
Message was edited by:
donbaek - had to add my disclaimer here 
