Insufficient resources to satisfy configured failover level for HA?

[cyrus_ho]

I have two ESX servers

ESX1 = 20GB RAM (current memory usage: 7.40GB with 16VMs running, highest "configured memory"'s VM = 4GB RAM

ESX2 = 20GB RAM (current memory usage: 10.12GB with 16VMs running), highest "configured memory"'s VM = 4GB RAM

Number of host failures the cluster can tolerate = 1

When I tired to power on one more VM, it prompted the message "Insufficient resources to satisfy configured failover level for HA" which do not allow me to continue starting up it. Actually I don't know why the message happened when I was trying to power on the 33rd VM.

I tired to search some resources and found out that I need to add more RAM or I need to check the "Allow VMs to be powered on even if they violate availability constraints" from the HA setting.

Also, according to this website: http://www.vmwarewolf.com/ha-failover-capacity/ . I can't apply the calculation correctly to my existing environment. (Maybe I got some wrong in here)

Here is my Calculation:

20GB RAM / 4GB RAM = 5 (Does it means I have 5 slots available per ESX host in the cluster??)

Applying the Step 4 mentioned:

Since I have 2 hosts and the configured failover capacity for the

cluster is 1, it left only 1 hosts in a failure situation. Hence

the total number of VMs that can be powered on this server is 5

VMs. (i.e. 5 multiplied by 1 = 5)???? Only 5 VMs I can power on?But I am currently running 32 VMs simultaneously

I think my calculation is going wrong, however I don't know what is the memory resouce allocation and calculation of the HA requirement?

Does anyone can tell me why I can't power on more than 32 VMs??

Many Thanks~

[weinstein5]

Welcome to the forums - You are heading down the right path to figure out why this is happening - one question do you have any reservations set on your VMs or resource pools for either memory or cpu? Because the slots are based on reservations and not the assigned memory assigned memory - what is the pool of CPu cycles you have available? - check out page 75 of the resource management guide - http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_resource_mgmt.pdf

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct

[mike_laspina]

Hello,

The maximum VM's per host could be why. Normally you can have 128VM's / vCPU.

e.g.

32 X 4 vCPU = 128

[cyrus_ho]

I didn't set any reservation in the resource pool. All are set as "Unlimited".

Sorry, what is the meaning of pool of CPU cycles?

My CPU in both ESX are Xeon X5355 2.66Ghz and it showed "8CPU x 2.659 Ghz", I think it is enough for processing all VMs

[TomHowarth]

What ESX version are you running and what VirtualCenter version are you running? there is a known bug with VirtualCenter 2.5 U2 regarding 2 node HA clusters that was fixed in the U3 release.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth

VMware Communities User Moderator

[cyrus_ho]

My ESX version is 3.5.0, 120512

And the VirtualCenter 2.5 Update 3 (build 119598)

The upgrade was performed with in-place upgrade without any problem occured.

[cyrus_ho]

Sorry, i don't know what you mean that about your calculation.

How can you find this formula: 32 x 4 vCPU = 128???

[weinstein5]

Actually according to the maximums document - http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_config_max.pdf - the maximum of vcpus per host is 192 vCPUs and 170 VMs -

The p[ool of CPU cycles is basically the speed of each CPU multiplied by the number of cores - If reservations are for your VMs even if it is only a single VM the calculation is done this way to determine available HA capacity - say one of your VMs has a memory reservation set to 1 GB then HA assumes all VMs will will need 1 GB so in your case if there is a fail over then you will 1 GB x 32 or 32 GB on the remaining server - this calculation is done the same way for CPU cycles - the only other thing I can think of without knowing the CPU loads I believe what is going on is HA is looking at a worst case scenario in other twice your highest loaded host so 2 x 10.1 or 20.2 which is more than what a single server has -

[mike_laspina]

Sorry,

I was not as clear as I should have been.

There is a maximum limit per host for vCPUs. Initially 3.5.0 was limited to 128 vCPUs per host, thus the calc is based on a possible config of 4 vCPUs per VM.

And if that were the case here you would be at the limit.

I checked the current docs and the maximum has changed to 192 vCPU's per host.

It just seems that your limit is homed around the base 2 value 32 so the thinking is that it may be limited to the original value or the likes of.

I will check the avanced settings to see if there are some config limits that could result in your event.

[mike_laspina]

Hello,

There is an advanced config limit of 128 vCPU under the Misc section. So looks like the value is defaulted to the previous limit of 128 vCPUs per host.

Misc.RunningVCpuULimit

[cyrus_ho]

As you said previous before, I have 32VMs are running currently and the maximum vCPU is 4 these 32 VMs. When I try to power on the 33rd VM, the limit is occured.

Therefore you assume the limit should be caused by the default settings (Misc.RunningVCpuULimit = 128 vCPU) from the ESX

The formula is (32 VMs x 4 MAXvCPU = 128 vCPU limitation per host)

Is it right? Sorry for my repeatedly question, because I need to clarify the workaround to resolve this problem.

How can I check current value of this parameter?

Also how can I set this value?

Does it need restart to take affect?

[cyrus_ho]

I located the value under the ESX > Configuration > Advanced Settings > Misc

Does it need to restart to take effect?

[mike_laspina]

Yes thats correct

How can I check current value of this parameter?bq. Select the ESX host in VC and goto to the configuration tab -> Advanced Settings -> Misc and scroll to the Misc.RunningVCpuLimit entry.

Also how can I set this value?

Change the value to 192.

Does it need restart to take affect?

Never had to go there, But I would say that you would not need to restart the host since they usually note it if it does

[cyrus_ho]

Changed to both configurations of ESX server, but the error message was still prompted.

[mike_laspina]

Hmmm! Well it one of two possibilities.

1) It needs a reboot or mgmt-vmware service cycle. ( it would not surprise me)

2) I am not on the right track.

service mgmt-vmware restart

I would also start digging for clues in the host logs

/var/log/vmkernel.? etc..

/var/log/vmware/hostd.log etc

[TomHowarth]

In light of the conversations that have been going on here, Have you upgraded your host from an earlier version? that could account for the issue. if not I would attempt a rebuild of the Hosts.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth

VMware Communities User Moderator

[cyrus_ho]

The are no any possible solution for my current situation.

I suppose it should caused by the out of CPU resources because when I try to reserve the CPU for one VM, it prompt me the same message. However, I can reserve the RAM for the VM.

Thanks for all

[cyrus_ho]

Sorry I can't reboot my server at this moment

However, when I was viewing the log, it showed the following messages

/var/log/vmkernel

Nov 1 00:09:47 esx1 vmkernel: 17:05:21:37.640 cpu0:1041)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)

/var/log/vmware/hostd.log

Current value 147144 exceeds soft limit 122880.

Current value 146628 exceeds soft limit 122880.

Current value 146628 exceeds soft limit 122880.

So, I think it may be out of CPU resources issue.