Just applied update Build: 82664 ESXi 3.5 and now I'm only able to start one VM. I have tested changing the settings to see if I could launch one additional VM, but I am unable. The error message in the subject line is all I get.
There seems to be a problem with ESXi in the way (that's my opinion) it handles memory for virtual machines. If you look at Configuration \ Memory you will see memory split between system and virtual machines. The virtual machines number includes the memory that the VMs will actually use, plus memory overhead to manage the VMs.
The Memory Reservation on the Resource Allocation tab reflects the portion of memory that ESXi thinks it can give to VMs (i.e. Virtual Machines memory - VM memory overhead). With ESX regulary that number is around 200 MB but with ESXi its up around 700 MB. Not such an issue when you have 16 GB, but it is with 1 / 2 GB. To free up some more memory
1) Go to Configuration \ system resource allocation
2) Click on advanced.
3) Find the VIM object and change the reservation from 512 MB to 192 MB. (see image).
That'll impact the Memory Reservation number of the Resource Allocation tab (give it a minute or two).. You can probably go lower that 192 (you may need to modify reservations on sub-objects. I wouldn't recommend this on a production system.
I tinkered and found I was able to increase my Memory Reservation to as much as 479MB by reducing the VIM setting to as little as 133MB (it wouldn't accept lower than that for me). I totally agree I wouldn't do that for production and I bumped it back up a bit. I'm just trying to get familiar with ESXi right now and the systems I plan to use for initial production have 4 - 8GB. Those aren't huge boxes either but I think they'll suffice for the migration of several VMs that I'm currently running with VMware Server under Windows. I'm also planning to test and compare with iSCSI and NFS datastores on the backend for shared storage.
Message was edited by: KhanStPreest
Thanks Dave! I as well was testing out ESXi on a non-production box with only 1 G of RAM. For the life of me I couldn't figure out why I couldn't start 2 vm's that only had 128 mb each. Heck, I couldn't even start 2 vm's that only had 64mb each. I was rather upset cause VMware is all about how ESX allows you to run "more on less" and how it has such a small foot print. I was about ready to write it off as a bunch of hoopla when I came across your post. Following your directions I'm now running 2 vm's with 256mb each and if my Resource Allocation tab is right, I might be able to fire up 1 or 2 more with 256mb each before it starts to die on me again. Excellent post and frankly VMWare needs to fix this. If I'd been a bit busier today or a bit less persistant, and not found this thread, I would have totally written off VMWare.
Excellent Post Dave!!!! Can't thank you enough. VMWare... fix this issue! I think your going to see a LOT of people testing this software on boxes that only have 1 or 2 G of memory and I'd bet many people will be like me and totally write off the software if they can't get it to run easily.
Today, I also had the same problem on my VMESXi3.5 U2 released at 2008/08/13.
I reduced vim's memory reservation value to 192MB as you said and got solved. Now I can run 2 VMs concurrently on 1GB HP server.
I have one more questions about vim. What is vim's purpose? Any info docs is available?
If you take a look at the processes under VIM you'll see the various management agents for ESXi - vpxa - VirtualCenter agent, busybox - service console interface, SFCB - CIM server, dcui - the ESXi user console interface (ALT-F2), etc.
Ok, well I guess the 10 million dollar question is _why_? Why does VMWare provide around 730 MB of memory overhead for VM's on ESXi and ONLY 200 MB for VM's overhead on ESX??
I see everyone happy in changing the reservation from 512 MB to 192 MB on the VIM object and happy for the extra resulting memory reservation showing up on the Resource Allocation tab, but hasn't anybody bothered to ask why did VMWare design ESXi like this in the first place???
Why did VMWare give a little over 700 MB for VM overhead in ESXi and not just 200 MB as in ESX???? And should we KEEP the 512 MB reservation on the VIM object since it was there by default? There MUST be a reason why it's 512MB and not 192 MB.... right???
Nice work Dave.
I agree with the other poster that this will be a stubling block for people testing ESXi as that is what I was doing and was about to give up until I found your posted solution.
But I suppose one should not treat VM technology with the same attitude as simple pc.
Thanks to Dave for his informed responses and to you all for asking the question. Was just about to set up ESXi on a new PC at home and would have hit this snag pretty quickly with my 2GB. Cheers for the proactive, preventative maintenance.
I also encounter this problem and changing the advanced vim setting worked.
After I restart the server it defaults back to 1024MB reservation.. I'm
guessing this was intended to happen. Is there anyway to prevent / permanently
change this setting?
running 3.5 U4 with 2GB memory.