VMware Cloud Community
JonRoderick
Hot Shot
Hot Shot

Vmotion problems with more than 6GB RAM - experiences?

I'm starting to see some performance impact on VMs with > 6GB RAM - have a dedicated pair of 1GB NICs for vMotion in my ESX 3.5 U3 setup.

Anyone for any similar experiences - I'm heading towards telling them app team to go physical but don't want to do that if I can avoid it.

Cheers

Jon

Forgot to say, these are Windows VMs.

Message was edited by: JonRoderick

Reply
0 Kudos
11 Replies
wila
Immortal
Immortal

Hi Jon,

Not all windows are the same.. are these windows 2003 servers? If so what type..

What's your host.. how much memory does it have. Do the hosts have memory to spare or are they swapping memory?



--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
JonRoderick
Hot Shot
Hot Shot

Yeah, left stuff out!

Windows Server 2003 64 bit - hosts are 24GB and not swapping but running 70/80%.

THe guests in question are using the RAM too - i.e. 6/12GB allocated and 6/12GB being used (not 12GB allocated and only 2GB used!).

Jon

Reply
0 Kudos
wila
Immortal
Immortal

Heheh.. well you probably don't have to tell us everything, but a bit might help in order to get the right idea.

The thing I would at first look for is if the VMs are ballooning as this could adversily effect your VMs performance, especially if those are VMs on which you need high performance. You can find this out on the performance tab in the VI client.

I'm not aware of VMs getting slower when they are over a certain size in RAM, never heard about it before anyways. Not that that means everything Smiley Happy



--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
JonRoderick
Hot Shot
Hot Shot

My only thought was whether the slowdown was being caused by the amount of data being piped over the vmotion vswitch - 1GB link and 6/12GB RAM - any potential probs?

Jon

Reply
0 Kudos
wila
Immortal
Immortal

Are those guests bounced around the hosts alot then? Normally it should be fine with dedicated VMotion nics or are those NICS located on the same LAN as your guests.

It shouldn't really matter.. but beware the "shouldn't" 😉

--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
JonRoderick
Hot Shot
Hot Shot

Those guests are being vmotioned say 2 or 3 times a week - not too much but the app team do see the VM performance drop off a cliff when it vmotions.

I'll check things out in ESXTOP and do some manual vmotions I think...

Cheers.

Reply
0 Kudos
RParker
Immortal
Immortal

slowdown was being caused by the amount of data being piped over the vmotion vswitch - 1GB link and 6/12GB RAM

There is no 'slowdown'. You have to think about this from a software point of view. Virtual Machines are software emulated hardware. You are forcing ALL the software emulation across 1 piece of hardware SHARED among 20-30 other VM's. So there are 2 CPU, but 1 Memory bus, 1 SCSI controller, and your Virtual Switch I bet is barely used, and the CPU cycles are time slices.

So when you increase the RAM in a VM, you are using software to provide your container for what Hardware would normally provide. And since it's emulated, you don't have the benefit of the cache and the hardware speed, software is a fraction of the hardware. CPU is the only Virtualization that is able to passthrough to the VM, but that is done in time segments you don't get 100% CPU 100% of the time. So the more RAM you put in a VM the more software driven translation that takes place, and thus since it's slower than hardware, it takes longer to load the memory at boot (which takes longer) and manipulate the memory, and flush the memory (which takes longer) when you shutdown. Lot's of memory in a VM is more noticeable because you are feeling the difference between software and hardware. So that's why it SEEMS slow, but it never was fast to begin with, it's only now that more programs are being used that you notice the impact. So it hasn't 'slowed' it was never up to hardware speeds, and this is the best it's going to get regardless of what Virtualization you use.

There is only so much a single machine with 2 sockets, 1 memory bus, 1 SCSI bus can handle during a given period. And even if you had 1 VM ALL of the hardare calls are done at a software emulated level, since you don't have direct access to the hardware.

Reply
0 Kudos
JonRoderick
Hot Shot
Hot Shot

RParker - as ever, to the point Smiley Happy

Sounds as though you're saying I need to consider very carefully whether or not VMs with large amounts of RAM are suitable for ESX (or virtualisation of any kind).

In your experience, what are the largest VM specs that can be satisfactorily virtualised on ESX?

Jon

Reply
0 Kudos
JonRoderick
Hot Shot
Hot Shot

Also, am I to expect the vmotion operation to degrade as the amount of RAM in the VM is increased? I would've expected to see some analysis of this done by VMware - unless it was and the results were poor and so weren't published...:)

Jon

Reply
0 Kudos
wila
Immortal
Immortal

Yes, makes a good point and it also made me remember that this isn't true for all CPUs and why newer CPUs are better, for example the RVI feature on AMD processors helps quite a bit here. See:

Third-Generation AMD Opteron™ Processor Key Architectural Features

and

Performance of Rapid Virtualization Indexing (RVI)

Not sure if intel processor have such a feature already as well.



--

Wil

_____________________________________________________

Visit the new VMware developers wiki at http://www.vi-toolkit.com

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
Kahonu
Enthusiast
Enthusiast

The more RAM you have, the more motioning VMotion has to do. So naturally a 500mb VM will VMotion faster than a 6gb one.

Reply
0 Kudos