3.0.1 --> 3.0.2 upgrade disk performance drops 84%

[Jae_Ellers]

While doing some unrelated work on the test systems I noticed their local disk was very slow. So we added BBWC to those systems. While waiting for the hardware to show up we upgraded the systems to 3.0.2 to work on our roll-out.

After looking at the numbers I'm shocked. I ran IOMeter at 32k block with 100%, 50%, 0% read tests. I ran local disk and san tests. Initial SAN tests were right where I'd expect them on an older, 2Gb machine. After the upgrade the SAN througput and IOPs were down 60-84%. You can see the BBWC increased performance (marginally) on the local disks.

Anyone seen this? Has anyone looked? I'm going to research HBA firmware and see if I need to update the test systems, but any ideas or confirmation would be appreciated.

Backend is 2 EVA 8000s. Gear is HP DL380 G3 w/ 2Gb Qlogic cards. Results are attached.

[Morten_Dalgaard]

Hi.

The reason for your lower scores is the fact that you used a 64 queue depth on your 3.0.1 tests, but a 1 queue depth on your 3.0.2 tests.

[Jae_Ellers]

Outstanding catch Morten. This test was run by another admin and I assumed she used the same test configurations. Just goes to show the truth in the old saw about assume.

I'll retest soon and post results, but I bet you're right.

[Morten_Dalgaard]

I just wish i knew how to get performance like that.

We run HP bl465c blades, with Qlogic HBAs, hooked up to an EVA8000 @ 2gbps.

While the Windows blades are able to get nearly 400Mbytes/sec (with 2 port load balancing), VMware ESX 3.0.2, and RHEL 4 are only able to manage around 80Mbytes/sec. We have this general problem with Linux machines on the SAN.

esxtop shows 0 queue on the HBAs, and 0 load (?!). The guest OS runs at about 80% iowait.

I suspect the Qlogic Linux driver, however i hear that Emulex cards have the same problem in Linux.

Any suggestions would be greatly appreciated.

[Jae_Ellers]

You should be getting performance like that. The test platform is a DL380 G3 with a single port qla 2340 2Gb card. We're running 2x EVA 8k as well w/ 4Gb Brocade.

How much other stuff is happening on the EVA? What are your IOPS & data rates on your disk groups overall?

[Morten_Dalgaard]

The EVA is pretty much idle, the Windows test, and the Linux/VMware tests are done under the same conditions, with the same hardware. That is what leads me to believe that it can only be something driver/OS specific, and not hardware/load specific.

Do you have any options set for the QLA driver in VMware? Like Queue options? I run without any options.

[Jae_Ellers]

I set the following now, but it was vanilla when my original performance tests were run.

#!/bin/sh

esxcfg-advcfg -s 0 /Disk/UseDeviceReset

esxcfg-advcfg -s 1 /Disk/UseLunReset

service mgmt-vmware restart

/usr/sbin/esxcfg-module -s "qlport_down_retry=14 ql2xmaxqdepth=64" qla2300_707

/usr/sbin/esxcfg-module -g qla2300_707

/usr/sbin/esxcfg-boot -b

/usr/sbin/esxcfg-boot -q vmkmod | /bin/grep qla2300_707

-=-=-=-=-=-=-=-=-=-=-=-

Check my blog:

-=-=-=-=-=-=-=-=-=-=-=-