Highlighted
Enthusiast
Enthusiast

ESXTOP & Queues

Hi

It drives me slightly crazy, maybe you can help me out...

Based on KB: 1267

I set the queue to 64 and reviewed it:

l2xmaxqdepth               int   64     Maximum queue depth to report for target devices.

cat /proc/scsi/qla2xxx/8

Device queue depth = 0x40 = 64

So thats the device driver queue also known als LUN queue, correct?

But then there are some other queues:

esxtop - d:disk adapter:

                                        AQLEN

vmhba0 -                       0    31      0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

vmhba2 -                       1   975     5.99     0.00     5.99     0.00     0.02     0.00     0.00     0.06     0.01     0.07     0.00

vmhba3 -                       0  2176     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

The value listed under AQLEN is the queue depth of the storage adapter. This is the maximum number of ESX VMKernel active commands that the adapter driver is configured to support.

esxtop - u: disk device

                                                     DQLEN

mpx.vmhba32:C0:T0:L0       -               1     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

mpx.vmhba34:C0:T0:L0       -               1     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

naa.xxxxxxxxxxxxxxxx       -            128     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

The value listed under DQLEN is the queue depth of the storage device. This is the maximum number of ESX VMKernel active commands that the device is configured to support.

And last but not least:

http://communities.vmware.com/docs/DOC-6490

The kernel queue can be thought of as kind of an overflow queue for the  device driver queues.

AQLEN means that the HBA can handle up to 2176 active commands.

DQLEN limits the commands per storage device.

l2xmaxqdepth limits the active commands per LUN.

Where is the difference? Shouldn't be the values the same? This is where I'm confused the most!!

So basiacally I can multiply the LUN queue with the number of connected datastores to know how much active commands my host is allowed to issue down to the storage. This value will be limit by AQLEN to 2176 per vmhba.

But wait, there is more!?

I can also limit the number of commands within the HBA BIOS.

Here I can set the "Execution throttle" which will further reduce the AQLEN down to let's say 256 or even lower.

Regards

Patrick

0 Kudos
7 Replies
Highlighted
Enthusiast
Enthusiast

No ideas?

0 Kudos
Highlighted
VMware Employee
VMware Employee

Do you have Storage IO Control enabled?

Highlighted
VMware Employee
VMware Employee

Patrick wrote:

AQLEN means that the HBA can handle up to 2176 active commands.

DQLEN limits the commands per storage device.

l2xmaxqdepth limits the active commands per LUN.

Where is the difference? Shouldn't be the values the same? This is where I'm confused the most!!

So basiacally I can multiply the LUN queue with the number of connected datastores to know how much active commands my host is allowed to issue down to the storage. This value will be limit by AQLEN to 2176 per vmhba.


I agree it is confusing but keep in mind:

  • 1 HBA --> many LUNs
  • 1 LUN --> many hosts / HBAs

If I can find the time I will try to describe this a bit better in a blog post.

0 Kudos
Highlighted
Enthusiast
Enthusiast

Just booted my lab.

At the first glance esxtop showed me a DQLEN of 32 for all disks. Then I verified that SIOC is diabled on all datastores. Afterwards I checked esxtop again and one disk showed me a DQLEN of 128, the other disks remained on 32.

//EDIT: The one disk who is flapping between 128 and 32 is a local RAID volume. So the l2xmaxqdepth/qla2xxx setting should'nt take effect here at all!? I completly overlooked this.

I guess the last time I set the value to 32 and rebooted the system the changed didn't take effect, why ever. Now the DQLEN and the setting in the qla2xxx drivers are identical and I’m less confused.

But as long as SOIC is disabled, the DQLEN should be the same as l2xmaxqdepth.

If SIOC is enabled, SOIC can dynamically ajust the DQLEN value?

Regards

Patrick

0 Kudos
Highlighted
VMware Employee
VMware Employee

For LUNs presented from a SAN via a HBA, the DQLEN will 32 by default.

For the LUN with a DQLEN of 128, is this a local disk?

You can also see different DQLEN in the case of 'gateway' disks, for instance EMC's LUNZ devices.

HTH

Cormac

http://cormachogan.com
0 Kudos
Highlighted
Enthusiast
Enthusiast

Cormac schrieb:

For the LUN with a DQLEN of 128, is this a local disk?

Yes.

But why is it flapping between 32 and 128?

Patrick

0 Kudos
Highlighted
VMware Employee
VMware Employee

I've not seen this before.

Are you sure it is the same device and not just an esxtop refresh issue where the NAA ids are changing order/place?

To answer your other questions, the change to the HBA device queue depth will not have any effect on your local disk which uses a compeltely different driver.

My only other thought was adaptive queue settings - but what you describe as 'flapping' does not match the behaviour of adaptive queue depths - see KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100811...

HTH

Cormac

http://cormachogan.com
0 Kudos