VMware Communities
PnwGuy
Enthusiast
Enthusiast

VMware 7.1 Not Releasing "Standby" Memory in Win7 Host

I have documented a variety of performance issues with Workstation 7.1 in this discussion and I have finally identified at least one cause. It is serious enough I think it deserves its own discussion.

The VMware documentation states that when you pause (or "sleep") a VM the host memory resources are supposed to be released. I have identified when you pause a VM, the RAM perviously used by the VM changes from "In Use" to "Standby" in the Win 7 Resource Monitor. That's expected.

The problem is Standby memory is supposed to be old cached data that's marked for deletion. So if another application needs more RAM than is available, Windows 7 is supposed to make the Standby RAM available to the new application. Unfortunately, if that Standby memory is a paused VMware VM, it sometimes is never released by Windows 7 (64 bit). So when you, for example, resume or open a new VM, there are extensive page faults as the host experiences massive swapping because a large block of memory is somehow being reserved by VMware for the previous VM.

I have tried several examples of pausing and resuming various VMs and sometimes the Standby RAM is used by the new VM, and sometime it is not. So far I haven't figured out why. But it's certain when this Standby RAM is not released, despite my host having 8 GB, the performance of everything seriously suffers.

Why is VMware not allowing the previous VM's "Standby" RAM to be made available to the new VM? I can't confirm it, but I strongly suspect this problem started with 7.1.x as I was not having any problems with 7.0.1.

The attached screenshot shows the aftermath of a new VM being resumed after a previous one was paused. You can see a 3GB block of Standby memory that was never released--that's largely from the previous VM. And you can see the massive hard page faults that just ended when a bit of free RAM was finally available (during the high faults there was zero free RAM).

Can someone from VMware who's familiar with Windows 7 memory management please comment on this?

Reply
0 Kudos
10 Replies
Scissor
Virtuoso
Virtuoso

Not sure if it will help, but this utility make give you more insight into RAM usage of your Host:

http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

As for resuming a VM taking a long time, try adding the following line to the Guest's .VMX file after powering it off to see if it makes a difference:

mainMem.useNamedFile = "false"

PnwGuy
Enthusiast
Enthusiast

As for resuming a VM taking a long time, try adding the following line to the Guest's .VMX file after powering it off to see if it makes a difference:

mainMem.useNamedFile = "false"

In researching the above I've discovered at least a half dozen config file parameters that people widely suggest changing regarding memory management to improve performance. Some of the others include:

MemTrimRate=0

sched.mem.pshare.enable = "FALSE"

prefvmx.minVmMemPct = "100"

MemAllowAutoScaleDown = "FALSE"

prefvmx.useRecommendedLockedMemSize = "TRUE"

This forum (and others) are littered with suggestions to change these obscure VMware config settings in the .ini and .vmx files. Yet I can't find any official VMware documentation for these settings--not in the big VMware Workstation User Guide, not in the "Best Practices" guide, and not in the "Performance and Tuning" guide. So what do they really do?

If these parameters are so critical to making VMware workstation perform reasonably well, in my case under rather plain vanilla circumstances, why are they not officially documented or why don't more of them have corresponding settings in the GUI?

I realize VMware is a very complex product, supports a lot of operating systems, etc. But it should either work better "out of the box" (without what amount to massive multi gigabyte memory "leaks" and other severe performance problems) or we the users should at least be given the information we need to configure it correctly for our usage. Some of the third party information on the above settings is conflicting.

I have spent, honestly, at least 10+ hours troubleshooting these VMware problems. I'm a consultant and bill my time out. I could have purchased a couple additional PCs and run my other applications on them rather than under VMware for the time (money) I've lost so far. This blind shotgun approach of "try this" and "try that", like the performance problems themselves, is getting very frustrating.

Reply
0 Kudos
Scissor
Virtuoso
Virtuoso

As for resuming a VM taking a long time, try adding the following line to the Guest's .VMX file after powering it off to see if it makes a difference:

mainMem.useNamedFile = "false"

This option made a big difference when I was running multiple VMs off a (slow) laptop hard drive, especially when shutting them down. It tells VMware to use the Windows Swapfile to back the Guest RAM, rather than creating a large .vmem file in the Guest Directory.

VMware probabaly doesn't set this option by default because it has some other drawback (maybe running a ton of Guests can overcommit the windows Swapfile?) , but I have never had a problem.

In researching the above I've discovered at least a half dozen config file parameters that people widely suggest changing regarding memory management to improve performance.

Welcome to the dark art of performance tuning. I think many people suggest adding .VMX settings rather than GUI settings because it's easier than telling you which GUI menu and tab to click on. Smiley Happy

However, there are some settings that are not exposed in the GUI. Continuum has gathered up a list of undocumented paramters, mostly gleaned from posts here on the Forums by VMware Employees: http://sanbarrow.com/vmx.html

>Some of the others include:

MemTrimRate=0

This corresponds to the "Disable memory page trimming" option in the GUI -> VM Menu -> Settings -> Options -> Advanced . You can read the Workstation Help for more info, but basically VMware Workstation will slowly return "idle" Guest RAM pages back to the Host OS. Workstation defines "idle" RAM pages as RAM that the Guest has not accessed in a while -- it doesn't really know what each RAM page is used for though.

You may have noticed that if you leave a Guest running overnight that in the morning it seems to be a bit "slow" to wake up again, so I disable this option on mostly-idle Guests that I leave running overnight to prevent that from happening.

sched.mem.pshare.enable = "FALSE"

I disagree with this suggestion. Sharing Memory Pages between running guests is good so I leave it enabled (the default). Smiley Happy In previous versions of Workstation, I believe that checking the "Disable memory page trimming" option mentioned above also set this to false, but as of Workstation 7 it does not appear to do so anymore. That may explain why some people bundle the two suggested entries together.

prefvmx.minVmMemPct = "100"

MemAllowAutoScaleDown = "FALSE"

prefvmx.useRecommendedLockedMemSize = "TRUE"

These have something to do changes made in the Workstation GUI -> Edit -> Preferences -> Memory Tab. I have never had to modify these options myself.

This forum (and others) are littered with suggestions to change these obscure VMware config settings in the .ini and .vmx files. Yet I can't find any official VMware documentation for these settings--not in the big VMware Workstation User Guide, not in the "Best Practices" guide, and not into the "Performance and Tuning" guide. So what do they really do?

Performance tuning and Benchmarking Guidelines for VMware Workstation 6: http://www.vmware.com/pdf/WS6_Performance_Tuning_and_Benchmarking.pdf , PDF Pages 5 - 7 basically covers almost all of the options listed above.

If these parameters are so critical to making VMware workstation perform reasonably well, in my case under rather plain vanilla circumstances, why are they not officially documented or why don't they have corresponding settings in the GUI?

I realize VMware is a very complex product, supports a lot of operating systems, etc. But it should either work better "out of the box" (without what amount to massive multi gigabyte memory "leaks") or we the users should at least be given the information we need to configure it correctly for our usage.

I can't remember how much RAM you have allocated to your XP Guest, but if it's more than 768 MB or 1 GB then you are no longer a "vanilla" circumstance. It was only a year or so ago that Desktop computers started arriving with a ton of memory installed.

It might be that the VMware Workstations defaults should be changed, but VMware has to make sure their product works well out of the box on underpowered low RAM desktops as well as monster multi-core multi-GB desktops.

Also, it's a dirty little secret, but running Guests with more than 1 vCPU hardly ever equals better performance. I think everyone goes through the phase where they configure 2 vCPU and 4 vCPU Guests just because they can (I know I did), but then realize over time that 1 vCPU Guests perform better overall.

I have spent, honestly, at least 10+ hours troubleshooting these VMware problems. I'm a consultant and bill my time out. I could have purchased a couple additional PCs and run my other applications on them rather than under VMware for the time (money) I've lost so far.

In the future if you don't have the time to research the answer to a problem, then you can hire your own Consultant by opening a paid Support Incident with VMware Technical Support for $50 and get an answer from the experts.

http://www.vmware.com/support/services/incident.html

Virtulization is a very complicated topic. Your 10 hours are not totally wasted -- you have learned a ton about how VMware Workstation works. It will help in the future.

RParker
Immortal
Immortal

This blind shotgun approach of "try this" and "try that", like the performance problems themselves, is getting very frustrating.

Uh..what?!? You call yourself a consultant? You are a disgrace to Techs. That's EXACTLY what you are doing, try this try that.. That's how all but a VERY few MAKE our money. It's called "thinking outside the box". If everything was documented, easy to figure out, and laid out perfect.. well we wouldn't NEED you now would we?

You are whining because the product isn't working, or you are upset that you actually have to THINK for yourself and figure things out when they AREN'T working? I am confused. Techs, this is what it's all about. If you are frustrated, maybe you should change career fields.

Reply
0 Kudos
PnwGuy
Enthusiast
Enthusiast

This blind shotgun approach of "try this" and "try that", like the performance problems themselves, is getting very frustrating.

Uh..what?!? You call yourself a consultant? You are a disgrace to Techs. That's EXACTLY what you are doing, try this try that.. That's how all but a VERY few MAKE our money. It's called "thinking outside the box". If everything was documented, easy to figure out, and laid out perfect.. well we wouldn't NEED you now would we?

I wouldn't mind trying things if I could find a clear definition of what those things are. But I can only find a few of them officially documented by VMware. So I'm mostly objecting to the "blind" aspect of this. It shouldn't be Rocket Science, or some Black Art to get VMware to work reasonably well under pretty average conditions (at least for modern day PCs).

And, for the record, I develop software for a living. And those who take a shotgun approach to developing software should not be in the business professionally. It works much better when you actually understand what you're doing, have a clear spec to develop to, clear guidelines for software practices, and a clear test plan to test the result against.

Sure, troubleshooting isn't always black and white. But having to go into undocumented settings to get a product like VMware to work even reasonably well in a rather typical application seems wrong to me.

Reply
0 Kudos
PnwGuy
Enthusiast
Enthusiast

Welcome to the dark art of performance tuning. I think many people suggest adding .VMX settings rather than GUI settings because it's easier than telling you which GUI menu and tab to click on. Smiley Happy

However, there are some settings that are not exposed in the GUI. Continuum has gathered up a list of undocumented paramters, mostly gleaned from posts here on the Forums by VMware Employees: http://sanbarrow.com/vmx.html

Thanks for all your helpful comments! Continuum's effort seems quite helpful too. If I was trying to eek that last few percent of performance out of my VMware setup, I'd probably agree that's a "dark art". But, in fact, I'm just trying to get it to even be usable without delays of several minutes just trying to resume a VM, giant memory leaks, etc.

sched.mem.pshare.enable = "FALSE"

I disagree with this suggestion. Sharing Memory Pages between running guests is good so I leave it enabled (the default). Smiley Happy In previous versions of Workstation, I believe that checking the "Disable memory page trimming" option mentioned above also set this to false, but as of Workstation 7 it does not appear to do so anymore. That may explain why some people bundle the two suggested entries together.

Supposedly the shared memory slows things down because VMware is constantly having to compare the memory between the VMs. But this is exactly what I mean about "conflicting information". And it doesn't help some things seem to have changed from 6.5 to 7.0 and even from 7.0 to 7.1. I'd rather devote dedicated memory if it stops all these other problems I've been having.

Performance tuning and Benchmarking Guidelines for VMware Workstation 6: http://www.vmware.com/pdf/WS6_Performance_Tuning_and_Benchmarking.pdf , PDF Pages 5 - 7 basically covers almost all of the options listed above.

I can only find 2 of the 6 config/ini parameters documented on those pages. But yes, there's some good info there even if it is for an obsolete, and significantly different, version of Workstation.

I can't remember how much RAM you have allocated to your XP Guest, but if it's more than 768 MB or 1 GB then you are no longer a "vanilla" circumstance. It was only a year or so ago that Desktop computers started arriving with a ton of memory installed.

This I can't agree with. Even $300 computers have had at least 2GB of RAM for many years now. Vista was nearly useless with anything less and it came out in 2006. And today 4GB is the norm. Short of a bare bones XP install, 512MB just isn't suitable for modern Windows running any sort of serious modern applications. Neither is 1 GB. So there's a problem here. With Vista needing more RAM since 2006 I can't cut VMware any slack here.

If VMware, out of the box with default settings, is going to fall to its knees if you try to use more than 1 GB of guest RAM, that's a serious problem. Because, for example, a Windows 7 Pro guest running any sort of serious applications is going to page itself to death with only 1 GB. And that paging is going to be extra slow because it's taking place on a virtual drive with the limited resources and other overhead of the VM.

Most all of my VM guests are 2 GB. This worked fine under Workstation 7.0.1, but, upgrading to 7.1 caused all sorts of problems.

Also, it's a dirty little secret, but running Guests with more than 1 vCPU hardly ever equals better performance. I think everyone goes through the phase where they configure 2 vCPU and 4 vCPU Guests just because they can (I know I did), but then realize over time that 1 vCPU Guests perform better overall.

Yeah, I read that loud and clear before I ever installed 7.0. That's something else that really should not be a secret, but instead should be clearly documented by VMware. In fact, if you try to set it above a single CPU, the GUI should warn you it may cause reduced performance.

In the future if you don't have the time to research the answer to a problem, then you can hire your own Consultant by opening a paid Support Incident with VMware Technical Support for $50 and get an answer from the experts.

I'm not trying to be negative here, but if I opened an incident and paid the $50 I'd still likely be subjected to a long process of back and forth with "try this" and "try that" with the VMware employee. But it's a good point. As I've said elsewhere, I generally have a lot of respect for VMware as a company.

Reply
0 Kudos
ksc
VMware Employee
VMware Employee

The VMware documentation states that when you pause (or "sleep") a VM the host memory resources are supposed to be released. I have identified when you pause a VM, the RAM perviously used by the VM changes from "In Use" to "Standby" in the Win 7 Resource Monitor. That's expected.

I don't remember the guide exactly, but it's not supposed to say that - "pause" release CPU resources but NOT memory, "suspend" releases both. Are you completely sure of "pause" versus "suspend"? If the description of "pause" says memory is released, I need to go have a chat with TechPubs...

The problem is Standby memory is supposed to be old cached data that's marked for deletion.

From 30,000 feet, this is quite true. From 20 feet up, it's a lot more subtle.

A page can be "pinned" - from an OS perspective this means I/O is in progress on that page, but from the perspective of standby memory this means the memory cannot be released back to the system. Normally this is not much memory (tens of megabytes at most), so Windows doesn't bother treating it specially; the fact that 2% of your standby memory cannot actually leave standby isn't relevant to most users.

Running a virtual machine requires a VERY large amount of pinned memory - we use up to ~80% of the host's RAM, in fact. Now, most of that standby memory cannot leave standby. In reality the memory is active, but Windows doesn't know this (and doesn't have an API to be told either), so it accounts the memory as standby.

And a "paused" VM keeps all its memory pinned, because it expects to be unpaused and a working set for a VM is an order of magnitude larger than it is for a typical process. So if releasing the memory is important, "suspend" is the right option - "suspend" unpins the memory so Windows can write it back to disk and reclaim it.

Reply
0 Kudos
EdP2
Enthusiast
Enthusiast

I'd just like to thank Scissor for his very helpful post. For whatever unknown reasons I found that the move to 7.x coupled with a move to Windows 7 as a host resulted in generally poor performance compared with VMWare 6.x. However after applying all the fixes suggested by Scissor, performance has returned to the old 'snappy' state. I'm sorry I just couldn't afford the time for step-wise exploration, but one or more of the fixes did the trick!

I'd suggest that VMWare performance tuning really deserves a 'sticky' as I was totally unaware of the pdf referenced by Scissor.

Reply
0 Kudos
PnwGuy
Enthusiast
Enthusiast

I don't remember the guide exactly, but it's not supposed to say that - "pause" release CPU resources but NOT memory, "suspend" releases both. Are you completely sure of "pause" versus "suspend"? If the description of "pause" says memory is released, I need to go have a chat with TechPubs...

Sorry for the error in terminology. But on the "Power" toolbar, what looks like a PAUSE button (the two vertical yellow bars) is really the SUSPEND button. So what I am doing, and was referring to, is a suspend. As I understand it, and you confirmed, a SUSPEND is supposed to release (or unpin as you describe it) the memory for that VM. I have, however, clearly documented circumstances where the memory is not released forcing the host OS to swap instead creating drastic performance issues.

I have tried using both the Suspend button, and also doing a sleep (standby) from within the guest OS itself, and it does not seem to make a difference.

The problem is so unpleasant when it happens I've taken to completely shutting down (powering off) my guests rather than suspending them. As long as I do that, I've not had any issues with memory being released. But then what good is suspend? It obviously takes much longer to boot each guest from scratch everytime you need it. And since upgrading to 7.1 I still cannot run 2 VM's at once like I could with 7.0. So I'm now cold booting my VM's several times a day as I have to switch back and forth.

Like EdP, I have also taken something of a shotgun approach and applied Scissor's suggestions, and those from elsewhere, regarding various memory settings, and that has also helped with some of the problems. Although I still have random issues running XP guests, Win7 guests have been relatively trouble free. But, again, I'm rarely using suspend as I simply don't trust VMware not to hang onto the memory.

Reply
0 Kudos
Rikk30
Contributor
Contributor

Being a lurker mostly, I found the posts in this thread very helpful.

Reply
0 Kudos