SAN performance issue

[ckboon]

Hi,

During anti-virus signature updates for all VMs, their performance are badly affected. Togging thru applications in the virtual desktop is very slow. VirtualCentre's performance report does not show a problem for CPU & memory for all VMs. Disk I/O was higher during the update period.

If I change the HBA queue depth, would it help? What is the default queue depth number, 16 or 32?

Is there a recommended number to use?

What is the max?

BTW, my SAN is a Sun StorageTek 6140, using SATA disk.

Thanks for the help.

Regards,

Alan

[depping]

For QLogic the default is 32, recommended is 64.

But it could be more than just that, how about the VM/VMFS ratio? Sata could be a problem etc.etc.

Duncan

[ckboon]

SATA may be the problem when concurrent read and write exceeds certain number of I/O. During normal ops, the performance is very good. Only during anti-virus updates then we will hit the problem.

For now, I will try changing the queue path number to 64 and test it out.

Thanks.

[larstr]

Use esxtop to see the queue depth on your san controller. Having some queueing is ok, but if you have a lot of queueing on a regular basis, it will be bad for you. Increasing the queue depth from 32 to 64 will help if you're having a lot of queueing on a regular basis, but if you're not having a lot of queueing it will make things slower.

Lars

[ckboon]

Stupid question. How do I read the output of esxtop?

I chose the d option and it gave me this output:

9:35:49am up 5 days, 11:51, 148 worlds; CPU load average: 0.44, 0.42, 0.42

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN WQLEN ACTV QUED %U

vmhba0 - - - - 1 1 1 3 238 0 0 0 0

vmhba1 - - - - 1 1 4 23 4096 0 0 4 0

vmhba2 - - - - 1 0 0 0 4096 0 0 0 0

vmhba3 - - - - 1 1 0 0 4096 0 0 0 0

vmhba4 - - - - 1 0 0 0 4096 0 0 0 0

Which Queue Depth should I look at - AQLEN (storage adapter queue depth) or WQLEN (World queue depth)?

And what should the numbers be anyway?

BTW, what block size should I set when I create a LUN on a SAN storage?

Thanks for the help.

Regards,

Alan

[larstr]

Is QUED = Queue Depth? I take it 0 is very bad?

0 is actually very good as it shows that there's currently no queueing going on. The vmworld session IP42 - "VMware ESX Server Storage Performance - A Scalability Study" goes into details on this topic and was a very good session.

Lars

[glynnd1]

Alan,

How did you resolve this issue? Also, what anti virus software are you using?

I ask as we have found a similar issue with Symantec when it does updates, and are interested in know how others have dealt with this.

David

[pandes3]

Hi glynnd1,

Did you find solution to your problem ? If so please can you share it ?

Regards

S

[bobross]

This is (nearly without doubt) a SATA side effect. SATA is notoriously weak at random I/O, which is prevalent during anti-virus signature ops. In fact, increasing Q depth may actually hurt overall response times, as the I/O will merely wait for service from a spindle. This is especially true if the disks in your array do not support native command queueing (NCQ). But even if they do, it is very likely that most of the slowness you see is I/O wait at the controller head, as it has nowhere to perform the I/O until the (slow) SATA spindles can handle the request.

It never ceases to amaze me how many folks think they can get by with SATA, only to see something like this - when push comes to shove, the complaints come in. I don't mean to sound preachy, but you get what you pay for! Our shop learned the same lesson the hard way, and we replaced all our SATA with FC (SAS would have done the same) and now the I/O flies.

[PaulSvirin]

Maybe you can try queue depth throttling:

---

iSCSI SAN software

http://www.starwindsoftware.com

[TheEsp]

Hi

What storage array do you have ? As the previous posters have stated your que depth is ok , your RAID group is probably working way to hard for the I/O your workloads are demanding.

Your SAN performance tools are going to be really helpful in seeing how your cache and controllers and RAID groups are working.