VMware Cloud Community
chill
Hot Shot
Hot Shot
‎11-27-2008 10:57 AM

RHEL 5 (32 bit) performance issues

I have a RHEL 5 (32 bit) Virtual Machine hosted on an IBM x3850 (ESX 3.5 UPdate 3). The VM is assigned 1 gig of memory, 1 vCPU, 50 Gig Disk (which is located on our SAN). Almost all of our VM's are Windows vm, only 2 linux. The other Linux VM is running fine. This one, the performance is terrible - Disk and CPUperformance in particular. My background is more on the Windows side, so I'm looking for some help to boost the performance of the RHEL 5 VM.

Let me know what other information you require to help with this.

Thanks in advance

If you find this information helpful or correct, please consider awarding points.
0 Kudos
26 Replies
Texiwill
Leadership
Leadership
‎11-28-2008 10:23 AM

Hello,

Moved to Virtual Machine and Guest OS forum.

I would start looking at /var/log/messages within your VM as well as determine which processes are running, etc. Is VMware Tools installed? You will need to debug the VM before we can look at other issues. What is running within this VM?


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
ToS_45
Contributor
Contributor
‎04-08-2009 06:14 AM

Hi,

We are experiencing the same problem with our performance of RHEL5 against VM on ESX.

We are dealing with Redhat and VMware support for several months now, but no solution is found and it seems that nobody has faced the problem !

We pointed out that the same poor performances exists in previous versions of RHEL (4, 3) if running on multiple vCPU and SMP kernel. After booting a machine with 1 vCPU and the UP kernel, we get normal performances.

But, as RHEL5 is no longer offering UP kernel, and as we cannot compile our kernel ourselves if we want to have support... We are stucked with very poor performances with I/O schedulers, I/O wait and so on... We get 40% performances less than a RHEL4 or RHEL3.

Our configuration :

RHEL 5 with one vCPU and 2GiB of RAM, 35 GiB of vDisk (on SAN).

ESX : HP DL380G5 with 2 Xeon Quad Core @2.66GHz and 16GiB of RAM

Does anyone have ever faced this problem ?

Regs,

0 Kudos
RParker
Immortal
Immortal
‎04-08-2009 07:44 AM

We are dealing with Redhat and VMware support for several months now, but no solution is found and it seems that nobody has faced the problem !

Not true! We have the same problem, you are not alone. My solution was to move the Linux VM's off the SAN, which of course isn't an option for everyone.

SAME ESX, SAME Hardware, SAME VM, they ONLY thing I did was migrate FROM SAN to Local disk. That's it. Everything works great now. So I understand your pain.

What's even more weird is the SAN works BETTER for Windows VM's than when Windows VM's were on the SAME local disk. Very perplexing.. So I just swapped places.

And just to add even more complexity / Weirdness... SUSE Linux doesn't have a problem either way, it's only Redhat....

0 Kudos
ToS_45
Contributor
Contributor
‎04-09-2009 06:51 AM

Hi,

unfornately, putting our VM on local storage didn't solve our performance issues (and was only tried as a test purpose).

Regs,

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 07:38 AM

Please also verify if there is any path thrashing happening if the VM is on LUN. If path thrashing is happneing then it will have lot of IO realted perofrmance issue. Also please check if you have set proper I/O scheduler. Red Hat recommends noop for VM and in RHEL 5 by default its cfq, so you can also try changing that. Also if you are running oracle on the VM that you can also try deadline I/O scheduler.

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 07:41 AM

Please also verify if there is any path thrashing happening if the VM is on LUN. If path thrashing is happneing then it will have lot of IO realted perofrmance issue. Also please check if you have set proper I/O scheduler.

How do we find out this information? That's sounds promising, can you point me to some information, or post it?

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 07:48 AM

The LUNs from a SAN device if configured in the 'AVT' mode, can cause path/LUN trashing effecting I/O performance. Solution: Present the LUNs with host type set to 'LNXCL' and 'AVT' Disabled.

0 Kudos
ToS_45
Contributor
Contributor
‎04-10-2009 07:52 AM

for the noop or deadline I/O Schedulers you can check how your linux kernel booted via the pseudo-file /proc/cmdline

if you want to modify these paremeters, you can edit your /boot/grub/grub.conf or either modify dynamically (for test purposes) at boot by editing your kernel bootline

I will try to find more about the 'avt' or 'lnxcl' parameters on my SAN...

Regs,

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 07:57 AM

also if its RHEL 5 you can simply cat /sys/block/sda/queue/scheduler to check in which scheduler is you machine curretly and you can change it to any desired I/O scheduler by echo > deadline /sys/block/sda/queue/scheduler. This is changed on the fly however to make it permanant you need to add the entry in /etc/grub.conf

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 07:58 AM

by the way which SAN you r using?

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 08:03 AM

> by the way which SAN you r using?

Netapp 3070

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 08:07 AM

Multipathing Checklist

This appendix provides a checklist of multipathing setup requirements for different storage arrays.

Component

Comments

All storage arrays

Write cache must be disabled if not battery backed.

Topology

No single failure should cause both HBA and SP failover, especially with active-passive storage arrays.

IBM TotalStorage DS 4000 (formerly FastT)

Default host type must be LNXCL.

Host type must be LNXCL.

AVT (Auto Volume Transfer) is disabled in this host mode.

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 08:08 AM

ok, i have IBM & EMC, check in the SAN there should be a option where you can select that to which host (windows,AIX,LINUX,Solaris-and you shoud find there LNXCL) you are going to present the LUN.

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 08:11 AM

none of these options exist and there are no settings for Netapp LUNs.

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 08:20 AM

there should be some option in NetApp, please check with vendor

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 08:51 AM

there should be some option in NetApp, please check with vendor

I don't have to check. I know there are NO options for LUNs on Netapp. I even posted a link that says the same thing, no specific requirements for Netapp, no tweaks, no settings, nothing.

They LUNS are either there or they are not. That's it.

0 Kudos
kkshirsagar
Contributor
Contributor
‎04-10-2009 09:01 AM

ok then can you please run the command "time dd if=/dev/zero of=test count=500000" inside the linux VM and paste the output. This command will create 256MB data file named test and display the time and write speed took by the command?

0 Kudos
RParker
Immortal
Immortal
‎04-10-2009 09:06 AM

500000+0 records in

500000+0 records out

real 0m2.216s

user 0m0.094s

sys 0m2.046s

500000+0 records in

500000+0 records out

real 0m12.162s

user 0m0.408s

sys 0m11.348s

first is SAN (Netapp) second is Local disk. Interesting part is, the Redhat works BETTER on Local disk than it does on SAN, even though this test doesn't prove it. The other poster said that moving his disks to local didn't make a diff however, but in our case it did.

0 Kudos