excessive VM paging and Shared SAN storage

pearlyshells · ‎02-26-2009

Is there a correlation between excessive paging on a VM and the SAN storage for the VMs VMDK file?

williambishop · ‎02-26-2009

I'm a bit confused by your question, but I'll bite. If you have excessive paging, you will impact your vmdk. Your vmdk does not cause you paging, that is a result of the OS itself, and it's configuration.

--"Non Temetis Messor."

pearlyshells · ‎02-26-2009

I apologize. The question came up during a conversation we were having in the office. One of the administrators commented that when a VM experiences EXCESSIVE paging that it can run havoc on the SAN LUNs. It was my impression that memory and memory issues within a VM has no bearing on the SAN. Outside of a storage area for vmdk's, I didn't think that O/S paging would be a consequence to the SAN....it might be if the Host memory experienced paging, I think.

So, the question really is to validate what our administrator said.

vSeanClark · ‎02-26-2009

<span class="jive-thread-reply-body-container">I didn't think that O/S paging would be a consequence to the SAN..

If a VM's Guest OS (XP, Windows Server 2003..) runs our of memory and needs to page memory out to disk, this will definitely affect the SAN, because it is writing to disk (disk being SAN). This is why it is probably to your benefit to give a Windows VM plenty of memory and let it use what it needs to (the unused memory just sits in a pool for all VMs to use). On the Linux side, you will want to be a little more stingy though. Linux will use most of the memory you give it for file cache (by default).

If the paging/swapping happens because you've only given your VM 1GB of RAM and it's capable of having 2GB, then you would be unnecessarily impacting your SAN with page file activitiy. Add memory and keep SAN happy.

Sean Clark - -

Sean Clark - http://twitter.com/vseanclark

pearlyshells · ‎02-27-2009

thanks. shows how little I know . I thought that the paging was to the vDD of the VM and that the SAN was not in the loop. So, how does memory limits and reservations help in avoiding this problem?

Erik_Zandboer · ‎02-27-2009

They don't. You configure an amount of memory to the VM. That is what the VM "sees" as total memory amount. If it needs more, it will start paging (using the SAN disks). Limits and reservations only determine how memory behaves on an ESX level, ESX can "clip" memory to the VM (the VM thinks it has 2GB of RAM, but it only gets 1GB for example due to the limit of 1GB). Paging then occurs even sooner. Reservation is just so that ESX keeps an amount of memory free for that particular VM in case it needs it.

You might try and create a separate disk on the VM where you put the pagefile. The you are at least able to split the "normal" workload from the paging workload on the SAN. You could even consider putting the pagefile disk on local storage (although you will have to give up on HA/DRS/VMotion then).

Visit my blog at http://erikzandboer.wordpress.com

Visit my blog at http://www.vmdamentals.com

pearlyshells · ‎02-27-2009

thanks very much for the information. I totally missed the boat on this when I went to the class

Ken_Cline · ‎02-27-2009

This is why it is probably to your benefit to give a Windows VM plenty of memory and let it use what it needs to (the unused memory just sits in a pool for all VMs to use).

Yes - and no. Remember that when you allocate RAM to a VM, the allocation has to be available (somewhere) before admission control will allow the VM to start. Any RAM that is reserved must be available in unreserved physical RAM on the ESX host. Any memory between Reservation and Allocation will be accounted for in the per-VM .vswp file that is created on VM power-up.

Also, if you reserve vRAM, once the guest OS touches the reserved memory, it is no longer available for sharing with other VMs. This is of particular interest when Windows is the guest OS, since the first thing Windows does on power-up is to go out and touch every available memory address.

So...from a "simplicity of administration" point of view, it may be a good idea to give Windows VMs lots of RAM; however, from a "judicious management of capacity" point of view, you really need to be careful and allocate only what the workload demands. This will likely take some tuning over time - and will probably be less than you originally think you're going to need.

If the paging/swapping happens because you've only given your VM 1GB of RAM and it's capable of having 2GB, then you would be unnecessarily impacting your SAN with page file activitiy. Add memory and keep SAN happy.

Just be careful not to fall into the trap of "I had 2GB on my physical server, I need 2GB on my VM". You also have to look at the additional space required by a larger vRAM footprint. You have the .vswp file I mentioned earlier, but you could also have a suspend file (if you allow your VMs to be suspended - typically only VDI environments), plus if you're allowing users to take snapshots that include RAM state - you have to plan for those files. All of these will live on the SAN (by default) - and SAN storage aint cheap...

Ken Cline

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/

Ken_Cline · ‎02-27-2009

They don't. You configure an amount of memory to the VM. That is what the VM "sees" as total memory amount. If it needs more, it will start paging (using the SAN disks). Limits and reservations only determine how memory behaves on an ESX level, ESX can "clip" memory to the VM (the VM thinks it has 2GB of RAM, but it only gets 1GB for example due to the limit of 1GB). Paging then occurs even sooner. Reservation is just so that ESX keeps an amount of memory free for that particular VM in case it needs it.

Yes...the configured amount of RAM for a VM can be viewed as being equivalent to the DRAM chips you put into a physical machine. That's all the OS can use, no matter how much it needs. If the OS needs more memory, it will have to use some sort of virtual memory technology to make it available - this typically means swapping pages of memory out to a pagefile. This pagefile is the pagefile within the guest OS - not the .vswp file that is created for use by the vmkernel.

You might try and create a separate disk on the VM where you put the pagefile. The you are at least able to split the "normal" workload from the paging workload on the SAN.

You could do this, but in most cases it's not worth the administrative overhead that it introduces. It's usually a better idea to simply monitor your VM's memory usage and adjust the allocation so that you can minimize intra-VM paging activity.

You could even consider putting the pagefile disk on local storage (although you will have to give up on HA/DRS/VMotion then).

Placing the .vswp files on local storage doesn't prevent HA/DRS/VMotion from functioning. For HA, it's a non-problem, because in an HA event, the VM dies and the .vswp file gets recreated wherever the new host is configured to put it. For VMotion (and thus, DRS), the .vswp file will be migrated along with the contents of RAM to the new host. This could add to the length of time that it takes to complete a VMotion, but it will not prevent it from happening.

Splitting out the guest OS pagefile to a dedicated .vmdk file and putting that on local storage will prevent HA/VMotion/DRS from being able to move the VM.