Dramatic pvscsi performance reduction in RHEL 6 vs 5?

[hostasaurus]

Hi all, we have an vsphere ESXi 4.1 cluster talking 10 gig iscsi to an EMC CX4.  For the past year we've been deploying servers based on centos 5 x86_64 where I'd do the install of boot and root using the emulated 'lsi logic sas' controller and ext3 filesystem.  I'd also add a paravirtual controller and second hard disk in vsphere, after the OS is installed I'd install the vmware tools and ext4 drivers then format the second drive ext4 and mount it under /var for data.

With this configuration, I have been able to consistently achieve about 260 MB/sec write speeds, 500 MB/sec read speeds and anywhere from 7000 to 11,000 IOPS depending on the number of drives in the LUN on the EMC.

So, out comes RHEL/CentOS v6 x86_64 and low and behold, pvscsi is built in; that's great news.  I've deployed a few of those where now all I have to do is go back to provisioning one hard drive to the guest because the kernel has the pvscsi loaded at boot.  I'm still doing a /var partition in linux for the data but now all the partitions are ext4 and pvscsi instead of just /var.

I never bothered to do any performance testing when we began to deploy these new guests because I didn't notice anything odd and figured it wouldn't perform any differently than v5.  Well, we started to see issues when one of the guests got loaded up with I/O and not an amount that would have ever caused issues before.  I started doing some testing and with the exact same guest configuration (memory and vCPU) on the same ESXi host with its data on the same LUN on the EMC, I can't do better than about 70 to 80 MB/sec write, 250 MB/sec read, 800 IOPS.  So my write has dropped by nearly 75%, read by 50% and IOPS are down tenfold.

I can't find anything to explain this other than the pvscsi module installed in CentOS 5 by vmware tools reports itself as 1.0.1.1 and the kernel module included in CentOS/RHEL 6 reports itself as 1.0.1.0-k.  I can't find much of any reference online to the versions or their differences so I have no idea if that could be the problem but I can't come up with any other ideas.

Anyone else seen this issue?  Any ideas for me to try?

Thanks!

[hostasaurus]

I haven't made any progress on this with vmware support and have not been able to get the pvscsi kernel library from version 5 (of the OS) to work in version 6 since that's still what I think the issue is.  Support did mention there being a known performance issue with RHEL 6.1 and pvscsi, which alarms me even more since I'm already having horrible performance on CentOS 6.0 as they do not have RHEL 6.1 rebuilt yet, so now it sounds like things could get even worse.  They said the solution to that issue is to run the older kernel until it gets figured out but could not give me any specifics.  They did point me to the ESXi v5 tools install pdf:

http://packages.vmware.com/tools/docs/manuals/osp-esxi-50-install-guide.pdf

which has a useful article on how to override kernel modules with ones from the tools package instead.  I've been trying to get that to work using the pvscsi.ko from the centos 5 tools kernel mod package but have not had any success getting a bootable system yet, or one that uses the other module file...

So no ideas at this point other than going back to CentOS 5 until someone else figures it out.

[eliotw]

Have you tried changing the default I/O elevator/scheduler being used ? We discovered that RedHat recommends "noop" for virtualized environments after some testing with RHEL5 indicated the default "cfq" elevator was limiting some types of IO (direct/raw, non-AIO like generated by EMC's iorate test tool and dd) to a single outstanding IO (confirmed via esxtop). This drastically reduced the I/O throughput we could acheive. RH indicated "cfq" was behaving as expected but it did not do the same in bare metal installs. We could not get an adequate explanation from them. Doesn't exactly match your configuration but it might be worth testing "noop" to see if there is any change.

How to check/change the elevator in use from RH:

To check what kind of scheduler is being used for particular drive, use this command:

cat /sys/block/<disk>/queue/scheduler

Example: Note “[noop]”, this denotes what scheduler is active currently.

# cat /sys/block/sda/queue/scheduler

[noop] anticipatory deadline cfq

An example that sets the sda I/O scheduler to noop on the fly

# echo noop > /sys/block/sda/queue/scheduler

To set the scheduler in Grub.

If we take noop as the target default scheduler for the system, the /boot/grub/menu.lst kernel entry would look like this:

title CentOS (2.6.18-128.4.1.el5)

root (hd0,0)

kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/VolGroup00/LogVol00 elevator=noop

initrd /initrd-2.6.18-128.4.1.el5.img

Having the elevator entry in place, the system will set the I/O scheduler to the specified one on every boot.