Cannot start VM's with error message cannot allocate memory

[johanvalck]

Hello,

We are running an ESX3.5 update 5 host (upgrade was done about 1 week ago) with 10gb mem and only 3 and sometimes 4 running VM's. We only use local storage on this host.

Today I wanted to start up a VM by using the VIC and noticed an error before it even really tried to boot. Also other VM's encountered the exact same problem, so it isn't related to one specific VM. I also tried to boot a VM from the console. And still no luck and I get this message:

VMControl error -999: Unknown error: The poweron operation failed unexpectedly

I found in the /var/vmware/hostd/ these errors, they are repeated every minute:

Exception while processing request: EndofStream

Client sent us an invalid SOAP request: at line number 1, no element found

PendingRequest: HTTP Transaction failed, closing connection. Reason: Broken pipe, class type is: N7Vmacore15SystemExceptionE

Also I noticed that when I check the host sumary in the VIC almost all memory is used, but when I check the individual VM's memory usage it doesn't even come close to the total memory usage.

I'm afraid to restart the whole host, because the VM's currently running are running just fine, without problems.

Any idea what this could be?

Regards, Johan

[AndreTheGiant]

Have you tried to open a connection with the VI client to the ESX host?

In this case the power-on works?

Andre

[Narkis]

Does the VM host SQL or any other high resource utilize application? As you said the VM is powered off modify the H/W and reduce the allocated memory.

Login to console try esxtop then press m for memory utilization report. or follow the KB article. Thanks!

Please find the KB http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=10232&slic...

[johanvalck]

Thanks for the replies so far.

I can only connect with the VI client to the host directly, because I don't run VC server. I have only 1 host, no vmotion, etc.

I have 3 VM's which are powered off, 2 are linux webservers and 1 is a clean basic installation of Windows 2003. All these 3 VM's having the same problem, they cannot boot. I tried also from the console "vmware-cmd /path/to/vm-server.vmx start", but without luck.

Are there any other logs I can check which might give me some more information? I checked /var/logs/vmware/hostd* and there I found the error messages I mentioned in my first post.

My guess is there must be eating something all memory whicht the VM's are also using, but on the host itself the command "free" shows nothing really strange, and in VC I don't see a VM eating all the memory.