VMware Communities > VMTN > VMware Infrastructure™ > VI: ESX 3.5 > Discussions

This Question is Possibly Answered

1 "correct" answer available (10 pts)
1 2 Previous Next
15 Replies Last post: Oct 12, 2007 12:00 AM by caciolli
Reply

Any feedback on IBM's new Technote claiming poor performance with VI3?

Sep 11, 2007 8:37 PM

Click to view benny.hauk's profile Hot Shot benny.hauk 202 posts since
Dec 6, 2004
After a few months of throttled CPU on all our IBM Notes mail server VMs running on VI3, we've all but given up on running any Domino server (mail server or otherwise) on VMWare. We'd been hopeful when seeing the same thing in ESX2.5 that things would improve in VI3 but they haven't. We finally found this technote from IBM basically saying that they see the same thing (SAN/No SAN, 2.5/VI3) and can't find a workaround:

http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg21252786

And their technote isn't talking about Domino either. They reproduce the CPU throttling within the VM (backedup by esxtop) using simple Windows copy (not dealing with network copies here, only disk IO) commands.

They seem to be saying that a significant portion of the IO work typically done by the physical hba has to be done in software when virtualized in ESX (aka, CPU cycles).

Anyone out there in the know disputing the claims IBM is making? We only found the technote because we've seen the exact same thing everyday. In the back of my mind I figured there was probably something that could be tweaked but we hadn't managed to find anything yet. Once I saw the technote from the super-highly-intelligent IBM eggheads, I didn't feel so bad ;)

I've seen some other postings like this one in which no one has given much explanation or been able to attribute seemingly huge IO problems to misconfiguration (only small differences seem to be tweakable):

http://www.vmware.com/community/click.jspa?searchID=-1&messageID=503100

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 3, 2007 11:35 AM
Click to view mfleener's profile Novice mfleener 15 posts since
Mar 21, 2005

I'm not here to dispute anything. I would ask some questions that IBM did not make clear to me in the technote. They did not spell out what type of I/0 was inside the physical ESX host they tested. I would also be instrested in these specfics: NIC count/speed HBA count/speed. Also what kinda of load was on CPU 0 (the CPU that the VMware kernal uses), was a min CPU/Memory reservation set for the console OS? I'm of the mind set to give more resources to the console OS. I run a reservation of 1024MB and 800MHz for the console just to be safe. Also something to look at is %ready for the console and various VMware processes.

Anyway its something you can look at in your enviroment if you have not already.

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 3, 2007 3:03 PM
in response to: mfleener
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator
You would think they would simply open a case with VMware support and get their response on their issue. I see no mention of them contacting VMware to find out why they are seeing the results that they are seeing.
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 4, 2007 5:38 AM
in response to: esiebert7625
Click to view benny.hauk's profile Hot Shot benny.hauk 202 posts since
Dec 6, 2004
Good point. They seemed to do a pretty thorough job of testing then at the end of the day made some assumptions and moved on. Would have been nice to bring this to VMWare support (both as an ordinary support customer and as "big blue") or at least state that you tried to or whatever the case may be. Maybe I'll have time one of these days to try and reproduce some of the simple failure cases they conducted and then give VMWare a call myself, since we seemed to see the same thing IBM noticed when running heavy Domino I/O.
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 4, 2007 5:48 AM
in response to: benny.hauk
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator
I know alot of people that run some heavy I/O VM's like Domino/Exchange/SQL servers and have not had issues. If it was indeed that big a problem you would think more people would be complaining about this particulraly to VMware. I've been a Domino admin for over a dozen years and I know the Domino server can be a very big cpu/memory/disk consumer. There are so many server tasks that Domino runs (it seems like more and more each version, back in R3 it was so much simpler) that it can require some serious hardware to run on.I would not even recommend virtualizing Domino for large installations only small/medium installs.
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 4, 2007 5:50 AM
in response to: benny.hauk
Click to view mfleener's profile Novice mfleener 15 posts since
Mar 21, 2005

Benny,

Just for fun, do you mind sharing with us what you give to the console OS for Proc/Mem plus any reservation numbers. Count/Type NIC/HBAs would also be helpful.

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 5, 2007 10:33 AM
Click to view hgaidhani's profile Novice hgaidhani 14 posts since
Aug 22, 2007
VMware

You may want to look at the VMworld session "WV21: IBM Lotus Domino and Lotus Sametime on VMware Infrastructure 3".


Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 9, 2007 12:08 PM
in response to: hgaidhani
Click to view benny.hauk's profile Hot Shot benny.hauk 202 posts since
Dec 6, 2004
The session is only viewable by attendees. Anyone care to share notes they took at VMWorld?
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 10, 2007 12:18 PM
in response to: benny.hauk
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator

You have mail....


Also have you seen this IBM technote...

http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg21267837

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Thanks, Eric
Visit my website: http://vmware-land.com
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 10, 2007 1:01 PM
in response to: esiebert7625
Click to view benny.hauk's profile Hot Shot benny.hauk 202 posts since
Dec 6, 2004
I hadn't read that one yet. Pretty standard VMWare doc, really (no practical advise there that isn't in any best practice guide for ESX). Impractical stuff like this is kind of annoying:

To achieve the best performance:
1) Create a single virtual disk per LUN (assign all available space on the LUN to a single virtual disk)
2) Utilize a separate LUN and Virtual Disk for the operating system, page file, application executable code
3) Utilize a separate LUN for the Domino data files
4) Utilize a separate LUN for Domino transaction logs

However, good news: I'm having conversations with the guy from IBM that wrote the original technote I posted and he also was the one who made that presentation at VMWorld. I'll share the results of our work once we are finished.

Question for a moderator: Can I post details about a particular presentation from VMWorld if I get the content directly from the presenter and not from the password-protected VMWorld site?

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 10, 2007 1:04 PM
in response to: benny.hauk
Click to view oreeh's profile Guru oreeh 9,815 posts since
Nov 30, 2005
Moderator
<Moderator hat on>

You should ask the original author and VMware - not us moderators.

</Moderator hat off>
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 10, 2007 1:46 PM
in response to: benny.hauk
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator

I don't see why not, the presentations are not public yet but the information in them isn't confidential. Just like if you went to VMworld and came back and told your co-workers about everything you had seen and heard there. The presentations should all be available to the public shortly I expect. There is a thread on this in the vmworld forum...


http://www.vmworld.com/vmworld/message/1307#1307

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Thanks, Eric
Visit my website: http://vmware-land.com
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 10, 2007 3:42 PM
in response to: benny.hauk
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator
Well I just listened to the VMworld presentation and it basically confirmed that CPU utilization is much higher on virtualized Domino systems compared to physical systems. They are working with VMware to find out the reason why. A couple other things that stood out, VMFS performance was much better then RDM performance with Domino, also the key performance measure for Domino is user response time and a virtualized Domino server response time was almost equal and in a few cases better then the physical server.

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Thanks, Eric
Visit my website: http://vmware-land.com
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 11, 2007 6:03 AM
in response to: esiebert7625
Click to view benny.hauk's profile Hot Shot benny.hauk 202 posts since
Dec 6, 2004
Glad that you pointed that out - we experience the same thing:

I see high CPU on heavily used notes servers (mail servers with 100-200 users) BUT we can't see poor response time on the client end at all - furthermore we can't see a noticeable difference in response time regardless of whether we are cpu-throttled on a VM or on a physical server using very little cpu. That is a very good thing because it means we can keep our existing virtualized notes servers around while we figure something else out.

It should be noted that we still need to figure something else out (as an alternative to virtualizing Domino servers) if for no other reason than those VMs are using more than their fair share. Virtualization has lots of benefits, but if one VM hogs the cpu resources that 4-6 better-chosen VMs would use, then it makes more fiscal sense to put the one VM on a low-end physical box (where in our case won't need the CPU resources the VM counterpart did) and then virtualize the 4-6 other VMs in its place. I understand that Domino server then can't participate in VMotion and all that other cool stuff (cheap SAN connectivity for instance), but in the end it comes down to money, right? If a choice has to be made, I'll virtualize the 4-6 and deploy the resource hog on hardware.

Benny
Reply Re: Any feedback on IBM's new Technote claiming poor performance with VI3? Oct 11, 2007 6:26 AM
in response to: benny.hauk
Click to view esiebert7625's profile Guru esiebert7625 6,790 posts since
Oct 23, 2006
Moderator
Agreed, not every server is a good virtualization candidate. Domino is kind of unique because it has clustering and virtualization (server partitioning) built right into the application so if you use physical servers you still have those capabilities.

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Thanks, Eric
Visit my website: http://vmware-land.com
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

1 2 Previous Next
Actions