I've seen this error on the forums but, I just got this error and expanding the virtual memory didn't fix it. Here's my story:
I have a Windows 2008 Guest that will become a file server in my DMZ. It has a 590 gb data drive. We started to copy over the data from an old server and it get's about 300gb's and stops and can't e written to anymore. The drive is set up as a simple drive, not dynamic or gpt. Virtual Memory is set to 8gb. Anyone have any ideas?
Try changing the swapfile setting. I have had problems although I don't recall what the problem was (Exchange). I needed to set the swapfile to "Store in the Hosts swapfile datastore". (Settings/Options/Swapfile location) It lets Windows manage the swapfile.
Are you storing your Guest disks (.vmdk) on a VMFS partition? I think formatting your VMFS partition using the default block size of 1 MB allows for a maximum single file size of 256 GB, which would limit a Guest Disk (.vmdk file) to that size. Would that explain the behavior you see?
The hole in my theory is that I am not sure if ESX would even allow you to initially create the 590GB virtual disk to attach to your Guest...