VMware Cloud Community
abugeja
Hot Shot
Hot Shot
Jump to solution

Increased Memory Usage on ESXi 4.1 from ESX 4.0

Hi All,

Recently we have migrated around 130 virtual machines from a ESX 4.0 U2 Cluster to esxi 4.1 u1 cluster and noticed the memory usage is much high on the hosts compared to 4.0.

The 4.0 u2 cluster had 255GB of RAM between 9 hosts and the new cluster has 355GB ram between 7 hosts.

4.0 u2 memory usage was around 60-70% and now on 4.1 they are on 90% usage. Can someone please explain this. Find it hard to understand that there is 100GB more RAM available on the new cluster but its using all of it.

ESX 4.0 u2 hosts were HP BL460 G1 - 32GB each

ESXi 4.1 u1 are HP BL460 G6 - 48GB each. 

Thanks

Reply
0 Kudos
1 Solution

Accepted Solutions
Orcunus
Enthusiast
Enthusiast
Jump to solution

I believe it'a all about TPS (Transparent Page Sharing) and your new generation CPU's. This phenomenon is independent from the version of vmkernel. Let me explain briefly,

New generation CPU's, such as Nehalem, have a feature called Hardware-Assisted Memory Management Unit which includes a cache buffer called TLB (Translation Lookaside Buffer). I will not dive into details but if vmkernel detects that your CPUs have that kind of feature, it decides to use large memory pages (2MB) instead of small memory pages (4KB) and also not to use TPS as long as a contention on memory resources occurs. That's why you encounter higher memory usage than you expect to see.

As a workaround (not a best-practice but applicable), you can force vmkernel to work with small pages and trigger TPS. There is an advanced parameter for that.

  • Mem.AllocGuestLargePage=0

You can try this on a test environment and see if it works for you.

View solution in original post

Reply
0 Kudos
6 Replies
Virtualinfra
Commander
Commander
Jump to solution

Is that the Vmware tools on the virtual machine is updated.. If not?, Update the Vmware tools on all the migrated virtual machine.. Monitor the performance.. memory usage will be normal..

Thanks & Regards Dharshan S VCP 4.0,VTSP 5.0, VCP 5.0
Reply
0 Kudos
Orcunus
Enthusiast
Enthusiast
Jump to solution

I believe it'a all about TPS (Transparent Page Sharing) and your new generation CPU's. This phenomenon is independent from the version of vmkernel. Let me explain briefly,

New generation CPU's, such as Nehalem, have a feature called Hardware-Assisted Memory Management Unit which includes a cache buffer called TLB (Translation Lookaside Buffer). I will not dive into details but if vmkernel detects that your CPUs have that kind of feature, it decides to use large memory pages (2MB) instead of small memory pages (4KB) and also not to use TPS as long as a contention on memory resources occurs. That's why you encounter higher memory usage than you expect to see.

As a workaround (not a best-practice but applicable), you can force vmkernel to work with small pages and trigger TPS. There is an advanced parameter for that.

  • Mem.AllocGuestLargePage=0

You can try this on a test environment and see if it works for you.

Reply
0 Kudos
Virtualinfra
Commander
Commander
Jump to solution

@ Orcunus

TPS is introduced in ESX(i) 4.0. as its mentioned that the upgrade is from ESX 4.0 to ESXI 4.1, this shouldn't impact.

Take a look at the below KB for TPS..

http://kb.vmware.com/kb/1021095

Thanks & Regards Dharshan S VCP 4.0,VTSP 5.0, VCP 5.0
Reply
0 Kudos
Orcunus
Enthusiast
Enthusiast
Jump to solution

If you upgrade your hardware, it may impact even if you don't upgrade your vmkernel version. Please take a look at this blog.

http://blogs.vmware.com/uptime/2011/01/new-hardware-can-affect-tps.html

Regards,

Ozan Orçunus

Reply
0 Kudos
abugeja
Hot Shot
Hot Shot
Jump to solution

Hi All,

Thanks for your comments. We changed the  Mem.AllocGuestLargePage=0 setting in the dev lab and the host is now using less memory. Applied it also to a few hosts in production and noticed the same results.

What are the pros & cons to the allocation of guest large page?

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

What you are seeing is the use of large page files.  VMware has started allocating a lot more memory up front which helps get better performance out of the VMs.  Here is a KB explaining the larer memory allocation.

Use of large pages can cause memory to be fully allocated

When momory begins to be over allocated you will see indiviual VMs set to use the smaller page file size.  They will continue to use the smaller page file until hey are migrated off of the host or rebooted.  So there is no need to chance the settings on the individual VMs as you will be undercutting technology that helps them run faster, and the lower use VMs will be forced into that state if the memory is needed elsewhere.  You will also notice that there is increased use of memory balooning.  This is by design as well and has been shown to increase performance when used properly

I'll attach a graphic from a training manual that shows at what points the memory usage changes start to happen.

Reply
0 Kudos