Hi
I'm new to VMware and EsXi, so far I only used Linux KVM.
I've got a HPE Microserver GEN10 with only 8GB of RAM. I'm using ESXi 7 as a hypervisor. Most of the time I'm running a small single Linux guest on it. Because I'm using hardware pass through all the memory of the guest has to be reserved. Since it's in most cases the only guest, I'd like to assign 4GB of RAM to it. But for some unknown reason ESXi lets me only reserve 2GB of RAM for it. If I run other guests without memory reservation and hardware pass through, ESXi lets them use up to 7GB of the 8GB of physical memory.
How can I reserve more memory to a guest?
What are the limitations and where can I find the corresponding documentation?
Is there a way around reserving all of the memory for a guest with hardware pass through?
Thanks for your help!
Meinrad
The best place to start if you are looking for documentation is https://docs.vmware.com/en/VMware-vSphere/index.html When it comes to reservations there is no absolute limit. The only limit is what you have available in your ESXi host: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-117972E3-F5D3-4641...
In this example, I have configured 8 GB memory for this VM. When I want to reserve memory, the maximum for this reservation is 8 GB. If I want to reserve more I must increase the amount of assigned memory first.
it probably will not allow you to reserve more as there are a couple of other things that require memory, like the hypervisor itself, and the overhead reservation for the VMs.
Why would you set a reservation? ESXi is pretty good at memory management, just let it do it's thing!
A reservation is a promise, i.e. that while the memory might not be consumed / resident for the "reserver" right now, any allocation would be granted and if necessary forcefully taken away (reclaimed) from someone else. ESXi has a certain amount of default reservations for the purpose of stable operations, you are of course more likely to collide with those on a host that has as much memory as my phone.
The max you can reserve, is whatever is currently not reserved in the user pool (where the VMs go):
vsish -e get /sched/groups/4/stats/capacity | awk -F '[ :]' '$4 ~ /mem-unreserved/ {print $5/1024" MB"}'
Note that you _shouldn't_ reserve all of that in any environment that you care about because under some circumstances ESXi might need to reserve more (e.g. heavy IO workloads) and if it can't, will PSOD.
To list all current reservations > 9 MB:
memstats -r group-stats -s gid:name:min:conResv:availResv:memSize -u mb | sed -n '/^-\+/,/.*\n/p' | awk 'NR == 3 || length($3) !~ 1 {print $0}'
@deppingany form of passthrough requires memoy preallocation (since 6.5) which in turn requires a full reservation.
Ah, missed the "passthrough" part of the post 🙂
Just to be open and honest, 8GB does not give you a lot of memory to work with in general to be fair...
Definitely agree but there is an argument to be made that for corner (or edge, wink wink) cases like these we could at least look at our default reservations and potentially reduce the large consumers (hostd / vpxa / logger) based on actual estimated consumption or fully disable other services if not in use (e.g. iSCSI). A fine line to walk for sure and should definitely not be done on servers that don't fit into a shoe box.
@vbondzio Thanks for the detailed answer.
Your commands showed me that the host and the system alone use about 5.5GB. And that I technically could increase the reserved momory of my VM to almost 3GB.
$ vsish -e get /sched/groups/4/stats/capacity | awk -F '[ :]' '$4 ~ /mem-unreserved/ {print $5/1024" MB"}'
1007.29MB
$ memstats -r group-stats -s gid:name:min:conResv:availResv:memSize -u mb | sed -n '/^-\+/,/.*\n/p' | awk 'NR == 3 || length($3) !~ 1 {print $0}'
---------------------------------------------------------------------------------
gid name min conResv availResv memSize
---------------------------------------------------------------------------------
0 host 7606 6599 1007 4149
1 system 1586 1570 1023 1431
...
627625 vmm.206715 2048 0 3055 2048
627626 vmmanon.206715 35 0 1041 18
---------------------------------------------------------------------------------
@depping I know, but since I only need very limited amount of memory I thought 8GB is going to be enough. I was not expecting ESXi to reserver 6GB RAM for it self.
Keep in mind that ESXi is designed and intended to be used in datacenters, where memory/cpu resources are typically not a problem. Hence the reservation of resources etc.
I just looked over the code (very briefly and very likely very incompetently) and I don't see how you could disable the need for pre-allocation on pass through. You could try to manually set "sched.mem.prealloc" and "sched.mem.prealloc.pinnedMainMem" to false in the vmx, I'm not sure if that will be honored. In any case, it would be "sowas von unsupported", if it works at all.
There is also a way to reduce the reservation of certain worlds, we do mention the use of "localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig" in a bunch of KBs, it only takes a bit of trial and error and interest to deduce the usage and reduce the reservations of worlds where the memSize is waaaaay below the min.
Again, in no way recommended and you should really get two DIMMs of 8 GB instead of doing that, my answer is really just to satisfy your curiosity.