VMware Cloud Community
rrosenkoetter
Enthusiast
Enthusiast

Memory Resource Allocation - Serious Bug?

So I'm looking over Memory Resource Allocation, and I notice most VMs have the Limit set to the memory assigned to the VM (i.e. 1024 MB or 2048 MB). A few apparently random VMs have the limit set to Unlimited though. Anyone know what causes this? And if it would cause any problems?

Second thing I notice, and this could be a real problem... There are a couple of VMs that I built with 1024 MB of memory, and then later I increased their memory to 2048 MB. Well, the Resource Allocation number didn't change! So the limit is still set 1024 MB...

What happens when a VM that has been assigned 2048 MB (and that's what the VM's OS thinks it has), has a Resource Allocation limit of 1024 MB? I'm thinking some serious swapping is going on (which could explain some performance issues I've been having). Anyone aware of this bug? I'm using VC 2.0.2 50618

33 Replies
wapiti10
Enthusiast
Enthusiast

My understanding is that the limit is the maximum that the VM will be allowed to use. Therefore, even though the guest OS thinks it has 2048, what it was allocated. ESX will only allow that machine to use 1024, if that is what the limit is set at. That means less RAM thus perhaps creating a performance issue.

Dallas

Dallas
Reply
0 Kudos
RParker
Immortal
Immortal

Yeah as the other poster pointed out, that "unlimited" is a bit of a misnomer. It allows you to adjust the memory on the fly within the VM limits

So if you have a guest with 2Gb of RAM allocation, and you want to "throttle" that memory back to 1Gb, you can do it without changing the settings of the VM (which requires you to power off). So "unlimited" means there basically is no such throttling and the memory is allowed to use as much as the memory allocation assigned to the VM.

The guest swapping depends on the OS, and what you have inside the VM. You shoud monitor the VM with the meters to see if you really need that much RAM. We typically only allocated 768 Meg of RAM at most, except for special circumstance. Some of my VM's are at 512 Meg, rarely do I assign more than 1G, and I watch the swap file to see if it grows. Server 2003 will and must use a swap file. So even the existence of a swap file doesn't necessarily mean it's in use, you have to watch the growth of the file, if it significantly increases, then you can bump the RAM, but the swap file is always used by Windows 2003 Server, XP and Vista, you can turn it off.. but you could have unexpected results doing so.

Reply
0 Kudos
dlafrance
Contributor
Contributor

I am experiencing a very similar problem where various vms have cpu resources set to 23100 whereas the others are just unlimited. I agree and do think this is a problem for I had never set any limit or reservation on these vms. At first I thought it was due to how the vms were converted from physical because a portion were done manually and the others using VMware Converter, but I found that there is no rhyme or reason here.

Hopefully someone who's experienced this before will know the cause.

Reply
0 Kudos
krc1517
Enthusiast
Enthusiast

Anyone ever find out what was causing this?

I have about of 50 hosts that we see this on all the time. We'll spin out a VM form a template, check the VM to ensure that limits are not set for CPU and Mem and let DRS handle the rest after that.

After a day or sometimes weeks to months, we'll get calls saying the VM is very slow. Check the perf graphs, sure enough ballooning about 1/2 the memory. Check the CPU or Memory limits and sure enough, configued to some value vs unlimited like we origianlly configured the VM. Usually, memory is the one that causes problems. It usually seems to get scaled back to about 1/2 of what you have provisioned for the VM. CPU will get limited by a value, but it is usually the sum of all the CPUs in MHz of the host.

The host hardware is a mix of BL685c G1 w/64GB or 128GB or RAM. All running verion 3.5.0 130756

Thanks,

KC

Reply
0 Kudos
DCTony
Contributor
Contributor

Still seeing this issue. Anyone come up with a resolution?

Reply
0 Kudos
whinshaw
Enthusiast
Enthusiast

I am having this very problem now. Running vSphere 4 Update 1 on my templates. Any news on why this is still happening?

If you find this information useful, please award points for

"correct"

or "helpful".

Wes Hinshaw

www.myvmland.com

If you find this information useful, please award points for "correct" or "helpful". Wes Hinshaw www.myvmland.com
Reply
0 Kudos
Troy_Clavell
Immortal
Immortal

typically you will see these limits set if you have your templates configured as such. Convert the template(s) you use to deploy your guest with to a VM, and check to see there are any limits set.

Reply
0 Kudos
whinshaw
Enthusiast
Enthusiast

I built all my server 2008 templates and I never set a a memory allocation. Also, when I deploy servers from templates it may or may not have the allocation set. I converted all my templates and they "all" had a reservation set. So if I know I never set it, then when did this happen? It looks like some of my 2003 templates are in the same boat. Still looking into it.

If you find this information useful, please award points for

"correct"

or "helpful".

Wes Hinshaw

www.myvmland.com

If you find this information useful, please award points for "correct" or "helpful". Wes Hinshaw www.myvmland.com
Reply
0 Kudos
whinshaw
Enthusiast
Enthusiast

Ok, well it looks like I have figured it out on my own. From what I have been reading this can happen when you upgrade hardware (for instance add memory). We added memory to our hosts and doubled it. This and upgrade to a new version of ESX can likely cause the issue. If you have a hunch you are having memory balooning issues, you may want to consort vFoglight or any other troubleshooting software that has predefined triggers.

If you find this information useful, please award points for

"correct"

or "helpful".

Wes Hinshaw

www.myvmland.com

If you find this information useful, please award points for "correct" or "helpful". Wes Hinshaw www.myvmland.com
Reply
0 Kudos
Halesh
VMware Employee
VMware Employee

So I'm looking over Memory Resource Allocation, and I notice most VMs have the Limit set to the memory assigned to the VM (i.e. 1024 MB or 2048

MB). A few apparently random VMs have the limit set to Unlimited though. Anyone know what causes this? And if it would cause any problems?

Limit will not cause any problmes, this is the feature in the ESX resource management to limit the VMs not to cross the assigned resource. Unlimited also will not cause any problems, guest does not know about the 'unlimited' and it is really limited to the memsize for that VM. Unlimited helps in adding the memory to the VM in feautre like hot add, etc.

Second thing I notice, and this could be a real problem... There are a couple of VMs that I built with 1024 MB of memory, and then later I increased

their memory to 2048 MB. Well, the Resource Allocation number didn't change! So the limit is still set 1024 MB...

Yes, if you are limiting memory(Ex: 1024 MB in your case) , you need to increase the limit also along with memsize. So it is better to use unlimited. Unlimited is just terminology, but guest will never consume resource more than is is assigned.

What happens when a VM that has been assigned 2048 MB (and that's what the VM's OS thinks it has), has a Resource Allocation limit of 1024 MB?

I'm thinking some serious swapping is going on (which could explain some performance issues I've been having). Anyone aware of this bug? I'm using

VC 2.0.2 50618

Yes, and this not the bug. As you are limiting 1024MB , more than 1024 MB RAM memory will not be given to that VM. So rest will be reclaimed by memory swap. This is expected. If you dont want to make swapping of that VM you need to remove limits.

Thanks.

Reply
0 Kudos
allencrawford
Enthusiast
Enthusiast

Well, according to this KB article (http://kb.vmware.com/kb/1003470) which I've experienced on more than one occasion, you certainly can have issues by setting a memory limit less than what you've got allocated to the VM.

But, whatever causes this buggy behavior seems to still exist. It isn't being propagated via templates in our environment (already checked that) but I'm still not clear on what causes it. I've got a PowerCLI script that I run occasionally to reset all the VMs back to default "Normal" share levels and "unlimited" on the memory limits. Yet, after moving to a 4.1 vCenter Server, just about every single VM is set to have a memory limit equal to its assigned memory. Obviously that isn't going to hurt anything, until you go to upgrade the memory of course because you have to remember to change the memory limit too. But who's going to do that when they think everything is already set to "unlimited"?

Odd issue #2 that I see is that even after resetting all the limits (again) if I check the "Resource Allocation" on a cluster within vCenter, everything looks great (unlimited). But check the "Resource Allocation" tab when connected directly to a host in that same cluster, and all powered on VMs show a memory limit matching their assigned memory. Only the powered off VMs show "unlimited."

Oh, and someone on this thread said "just change the memory limit when you upgrade the memory on a VM." That's easy to say, but the other problem I had was when I was upgrading the memory in our MS-SQL VM that hosts the vCenter database. Since that VM was down, so was vCenter, so I had to connect directly to an ESXi host. I was able to bump the memory up just fine, but the limit would not change. I went from 8GB to 12GB and the memory just stayed at 8GB. The only time I was able to change it was once vCenter was back up and I did it via the vCenter connection (i.e. not directly attached to the host). It accepted "Unlimited" when i did that. Something is clearly not right years later...

Two notes:

1) I have opened an SR for this (finally, as we've had the issue since the vCenter 2.5.x days at least)

2) We did not upgrade to vCenter 4.1, we literally installed a fresh instance of vCenter 4.1 and migrated hosts to it via some PowerCLI scripts. And all of our hosts are still ESXi 4.0 U1 (except for a few ESX 3.5 U4 hosts that can't be upgraded yet).

Reply
0 Kudos
RParker
Immortal
Immortal

This and upgrade to a new version of ESX can likely cause the issue. If you have a hunch you are having memory balooning issues, you may want to consort vFoglight or any other troubleshooting software that has predefined triggers.

It's NOT a bug, you didn't read the part about.. it's not unlimited from the HOST its unlimited from the range you defined in the VM configuration.

2GB is the MOST that VM can be allocated (setting the memory limit). If you see "unlimited" that means the VM can use ALL of it's configured memory. This is because if you change the memory config later to something like 4G you don't need to change it in 2 places, since it's already "unlimited".

It just means the VM is "unlimited" with respect to the configured RAM, not ALL the memory the host can provide. So there is not problem, there is no bug, and there is NO issue.

Reply
0 Kudos
allencrawford
Enthusiast
Enthusiast

Well, except for the part where something is causing this value to be changed on lots of VMs...

Reply
0 Kudos
RParker
Immortal
Immortal

Well, except for the part where something is causing this value to be changed on lots of VMs...

I have noticed myself, it's triggered by a clone or sVmotion, that is not a mystery, the original question wasn't about WHAT was causing it to change, but WHY the value is unlimited.. and NOT set the value in the memory configuration.

THAT part is NOT a bug, it's by design. Could be a pre-built limit so that VM's don't start grabbing memory just because someone changed the memory amount, that could pose a lot of issues, so by setting a limit it keeps this from happening.

Besides that slider CAN be adjusted while the VM is running... so yes maybe it's an annoyance, but it doesn't cause downtime.

Reply
0 Kudos
allencrawford
Enthusiast
Enthusiast

Hmmm, I'll have to check it after my next svMotion. I've been doing a lot of them recently as we work on getting off an old SAN. However, I'd still consider it a mystery. Why would I want my memory limit to change after a clone or an svMotion? Or really ANY time where I wasn't the one making the change?

But, the original poster sure seems to be talking about WHAT causes the values to seemingly randomly change to me (even re-read it). It later morphed into a discussion as to WHY and HOW memory limits work (the part you correctly replied to).

Reply
0 Kudos
krc1517
Enthusiast
Enthusiast

I've been having this problem since before my original reply to this post back in 4/09

We've been bit by the template being configured with a limit in the past..a limit that we didn't intentionally set. Fixed that.

Once again, a random sampling, very large sampling at this point, have some value set as something other than unlimited for memory limit.

I read somewhere that removing a host and adding it back in will cause this value to change.

This could be true but I wasn't able to reproduce it this afternoon but much like allencrawford states, logging directly into the host and looking at the resource allocation for memory shows a value other than unlimited on all the VMs.

Looking at some esx3.5 hosts, i see the VMs are set as unlimited for memory limit

Looking at esx4 and 4.1, they are set to what the RAM is/was configured at time of provisioning.

We've done rolling upgrades through our clusters so a lot of the VMs were once on esx 3.5 and are now on 4.1. But in some case, the VMs were provisioned just a few days ago, with no limit set, but now have a limit set.

I too run a powershell script to change the values back to unlimited because too many times, the VM has been changed to a value that is less than the allocated amount of RAM and it starts to balloon as it should. This causes lots of performance issues.

What else could be causing this value to be set?

Reply
0 Kudos
nickg201110141
Contributor
Contributor

I too have seen this issue. We had a VM with poor performance and increased RAM, twice. However today we discovered the limit has not changed since the first time we increased RAM. This VM came over from 2.5 to 4.1 and was vmotioned many times.

Reply
0 Kudos
Gene_H
Enthusiast
Enthusiast

AllenCrawford,

Any chance you could post the PowerCLI scirpt?

Gene

Reply
0 Kudos
craigamason
Contributor
Contributor

I now have my second case open with VMWare support over this issue. (They didn't believe me the first time..) The case has been sent to engineering, along with about 1GB of gzipped trivia logs.  I will be happy to post whatever VMWare Support comes up with.

Reply
0 Kudos