disk queues

[theJoysOfESX]

I am in the process or sizing our environment for the purchase of a storage array/SAN. On our physical servers I have collected data on disk read and writes per second, kb per second, and disk queue lengths. I have been unable to find a way of measuring disk queues on the ESX host. Is this possible?

I realise I could run perfmon counters on all guests but this will take a while to setup and require a lot of number crunching.

[masaki]

You can read this:

http://www.vmware.com/pdf/wwc_performance.pdf

where iometer is said to be a good tool for disk I/O performances.

IMHO you don't need to do in deep analysis because you are going towards a better technology (SAN) with much more performances.

So whatever performances you have today, tomorrow will be better.

😆

the important thing is to distribute workloads on LUNs.

[theJoysOfESX]

Hi,

Thanks for the response. I suppose I could re-phrase my question:

How do you know when storage is a performance bottleneck?

On a physical server running windows I use perfmon to monitor the disk queue. If its consistently high I know the disk is the bottleneck. This is ultimately what I want to do in ESX. I know I can setup perfmon counters in each VM, but that would mean setting it up across 100 VMs, instead of 6 ESX servers.

[davidbarclay]

Have you looked at "esxtop", a command line utility?

Run esxtop, then push "d" to get disk metrics for that host with a breakdown per VM.

Dave

[masaki]

Through the VI client you can look at HOST and vm performances.

Disk I/O in KB or MB is presented. If you need a more deep analysis you must use esxtop and a good guide:

http://www.vmware.com/pdf/vi_performance_tuning.pdf

[wobbly1]

you can also monitor your san switches as well as look at tools available on the array - evaperf for example

[theJoysOfESX]

I looked at esxtop in VI3 and that does provide the queue info, sorry forgot to mention I am currently running 2.5. Also from my experience of esxtop it is good for a realtime view. However is there a way to record this data at regular intervals?

While I have the immediate issue of performance and I need 'proof' of this, I am think about this for capacity management purposes where I could see usage and growth trends, and ultimately predict when I need to buy more hardware etc.

[masaki]

with esx 2.5:

Optionally Enable VMKUSAGE and/or VMKTREE:

a) enable vmkusage:

vmkusagectl install

b) download vmktree from link http://tihlde.org/~larstr/vmktree/

Install with:

Installation/upgrade procedure:

tar xzvf vmktree-0.2.19.tgz

cd vmktree-0.2.19

./install.sh

Now you can access this application on https://your-server/vmktree/

VMKusage installs a local db that you can see on http://yourserver/vmkusage

[theJoysOfESX]

Masaki. I am running vmkusage on all my ESX servers, but there is no disk queue information. I haven't installed vmktree, but do you know if this does provides disk queue info?

Also as I upgrade to VI3 I will loose vmkusage and have to rely on VC2 for performance trending.

Am I trying to do something unusual in trying to identify when storage is a bottleneck? It seems very difficult to say 'for that amount of IO's the number of spindles I have is not sufficient', and disk queues seem to be the best indication whether storage is a bottleneck or not.

Wobbly1 - Currently the VMs are running on local storage so evaperf and switch info isn't available :-(. It will be in the future though .

[masaki]

You should look for some info under /proc

Like cat /proc/stat that gives some info on disk I/O

or /proc/vmware/procstats.

[VirtualNoitall]

Hello,

On your esx server run "man esxtop" and look at the batch section. You can set this to run in the background and log just the info you are interested in.

Running perfmon in a virtual machine is not a good way to get performance data due to time keeping in virtual machines

Message was edited by:

VirtualNoitall

[theJoysOfESX]

Virtualnoitall, I did experiment with batch section but esxtop in 2.5 doesn't have disk queues.

When you say running perfmon in VMs is not a good way, what problems does it cause? I am interested because I have been running perfmon counters in all my VMs for a couple of weeks, but I haven't analysed the data yet (there are 83 million rows in the DB!).

[VirtualNoitall]

Hello,

Have a look here: http://www.vmware.com/pdf/vmware_timekeeping.pdf. specifically the Time Measurements Within a Virtual Machine section.