Skip navigation
2019

Memory resource provisioning is one of the biggest challenges for IT administrators and virtualization designers. Although the provision of CPU is the major factor in the virtual infrastructure, most of the time there is lesser attention to CPU, because many applications that are deployed in virtualization need more memory, not the prreclaim.jpgocessor. And also today CPU technologies are very powerful, but applications like SQL and Oracle for some of their processes need more memory. Lack of enough memory resources may restrict the development of many virtual infra, so what should we do in this situation, before providing more physical resources for our hypervisors?

There are many technologies to handle the problem of memory over-commitment  such as Swapping and Ballooning and VMware ESXi use some of them to confronting with these issues. in this post I will review act and importance both mentioned mechanism:

1. Virtual Memory Ballooning is just a memory reclaiming technique that lets the VMkernel retrieve idle memory pages. When the ESXi host has less than 6% free memory (actually >=6%), Ballooning will come into the ring to handle out of memory problem! If a VM has many idle pages, its host borrows them to use as the temporary overhead for the VMs with more memory demand because probably they have some memory-intensive processes.

When a virtual machine wants to release some of the old used page files dedicated from host physical memory, does not remove it exactly, just change the address space pointer of the allocated memory list to the free memory list. So VM with Balloon Driver (vmmemctl.sys) can decide which pages (idle pages) can be reclaimed back to the host (up to 65% of VM guest memory) and which ones are needed for itself yet (already used pages) without involving host in this decision procedure. Now time for inflating step to happen If you disable memory ballooning driver inside the guest OS, VM will not be aware of host memory state and amount of available or unused physical memory and hypervisor cannot understand how much memory can take it back for other VMs memory requests.

2. Host Swapping is another mechanism used in low memory status, but unlike the Ballooning, it does not relate to VM guest OS. Host Swapping (or vRAM) has occurred If the host has less than 2% free memory and needs to provide more memory resources for memory-intensive VMs, so hypervisors should have enough swap space. although swapping is a reliable way against memory over-commitment, but it may cause total performance degrading because the efficiency of generated swap files is highly depended on IOPS rate of datastore specified for swapping, So if we don't provision suitable Disks (Like SSD) for swap files, it can lead to low performance.

There is one crucial point: If Memory Compression could happen, there is no need for hypervisor swap procedure execution. (I will discus more about it later on another post)

Remember, ESXi Host's memory will stay in share state until total required memory for them (memory requests) is greater than available physical memory (over-commitment), So if we want to prevent this situation before ballooning activation, we need to do these jobs:

1. Estimate carefully about overall required memory and also provide at least 25% more memory in the design phase.ballooning.png

2. Use VMware DRS in cluster-level resources to distribute VMs between hosts and balance resource usage, then no need to worried about low-memory. It's recommended to provision equal memory for all of ESXi hosts.

3. Use Resource Pools, especially in cluster-level and reserved estimated memory for mission-critical application and high resource-demand VMs and ensure they access to enough resources. Also, reserved memory is protected against memory reclaiming, even contention happened.

4. Do Not disable Balloon Driver, because it's the best intelligent memory reclamation technique. but if you want to do that, you can do it via the vSphere Client setting (Remove sched.mem.maxmemctl entry from .vmx file) or Guest OS Windows Registry (Set value of HKLM\SYSTEM\CurrentControlSet\services\VMMEMCTL from 2 to 4)

 

Link of post on my personal blog: Undercity of Virtualization: Memory Virtualization: Management & Reclaiming - Section I

 

 

 

VMware vCenter Server Appliance Management Interface (VAMI) is a very useful web console environment for VCSA management & maintenance operations,vSAN-Error.png that has been existing on HTTPS://VCSA_URL:5480. One of these tools is Backup and with these tools, you can take a specified backup of vCenter data consists of Inventory & Configuration (Required) and Stats, Events & Tasks (Optional)

VMware published VCSA6.7 (version 6.7.0.14) with these protocols for Backup: FTPS, HTTPS, SCP, FTP, HTTP. But announced NFS & SMB will be supported after VCSA6.7U2 (version 6.7.0.3). We had two big problems with these useful tools,  One of them is related to VSAN Health Service.

 

Whenever the Backup task has been started, it was stopped immediately and generated a warning about VSAN Health Service, because it seems to be crashed. (VCSA Management GUI exactly will tell us this happened.) Sadly if you try to start this service (even with --force option) it leads to another failed attempt and result is something like this:

 

So after many retrying for starting this service, I decided to check the files structure of this service in path of /etc/VMware-vsan-health and compare them with a fresh-installed vCenter Server.

vSAN-files.png

Also, there are two files that could to be related to the cause of this issue: logger.conf  file that has been absolutely empty in the troubled vCenter Server and VI result shows nothing, whereas in the healthy VCSA you can see something like below results:

VSAN-logger.png

When I checked the vsanhealth.properties, it shows communication of this service is worked with HTTPS, so its connections need to have an SSL structure. Then I found the second file: fewsecrets, it contains something like two hash streams. So I decided to risk and remove this file and the logger.conf file too (of course after getting its backup). At last, after some minutes the next try of service start was successful.

vSAN-Started.png

 

Remember that you need always to check DNS (FWD, RVS), NTP, Certificate and Firewall, Especially if you setup the vSphere environment with an External Platform Service Controller. I will explain the second problem in another post.

Link to my personal blog: Undercity of Virtualization: VCSA Backup Failed because of VSAN