With Fusion running a VM, my stats are as follows:
Wired: 373.51MB/Active: 1.06 GB/Inactive:
550.82MB/Used: 1.96GB/Free 39.15MB
After I quit Fusion:
Wired: 282.93MB/Active:
330.64MB/Inactive:257.55MB/Used: 871.13MB/Free 1.15
GB
This means Fusion was using over a 1 GB of ram.
This means that, while Fusion was running, Mac OS found some way to use nearly all the physical memory on your system. Most of this is going to be in the form of cached disk blocks. That's not surprising- even if the VM was idle, the process of booting a VM will cause a lot of disk activity, and that disk activity will be cached by Mac OS. That said, it's a little surprising to me that the cached memory would be reclaimed so quickly after stopping the VM.
It's very hard to measure the amount of memory actually "used" by a process. Typically, on unix-like operating systems, you want to use the number of pages owned by that process that are physically resident in memory. This is RSS on linux or RSIZE on Mac OS's "top". This doesn't account for shared libraries which may be used only by that process, and it doesn't account for data that the OS is caching on that process's behalf.
Things are further complicated by the way VMware manages memory. The VM's memory isn't "allocated" from your host in a fixed block, it is paged in as needed. Any VM memory that's needed by VMware itself will show up in the vmware-vmx process, but the bulk of memory in a virtual machine ends up being needed only by the guest OS and by the VMM.
There are a lot of different ways to measure this, but the most important figure is probably wired memory. Memory that the VM is actively using is "wired" in order to keep Mac OS from reclaiming it unexpectedly. Wired memory is also now completely unavailable to other applications on your system, since Mac OS can't (on its own) swap that memory out to disk.
I can't speak for the behaviour of unreleased products, but on the VMware Fusion private beta, your system's wired memory will be roughly equal to the wired memory when no VMs are running plus, for each VM, an amount roughly equal to the working set of that VM.
In your example above, VMware Fusion was using only about 90MB above the amounts you see vmware-vmx and the UI using. (Also note that there is some overlap between wired memory and the vmware-vmx resident set, so it's misleading to simply add these numbers.)
Also, as HPreg said above, the memory usage will grow up to a point (as the VM starts utilizing more of its "physical" memory) but it should not grow in an unbounded way. Please let us know if you observe such behaviour!
).