This is a Mac OS X issue.  OS X will convert inactive memory to free memory when it thinks it's in an OOM condition, but no sooner.  Obviously, there's a performance hit when this occurs.  If you want to control when this conversion occurs, you need the "purge" unix command from the Developer Tools for Mac OS X.  If you already have Xcode installed, typing in "purge" in Terminal should clear this memory for you after appearing to hang your computer for a few seconds.

I believe the issue is with memory mapped files.  Fusion is "dirtying" the files and OS X is saving some disk wear and write delays by not writing the file until it is required to do so (you shut Fusion down, or use the purge command).