you must shrink manually if one of the vmdks of the VM is a physical disk
to do that create a wiperfile manually inside the VM - see http://www.feyrer.de/g4u/#shrinkimg
then power down the VM and use the shrink option of vmware-vdiskmanager against the vmdk you want to shrink
I have been having the same error, and for the sake of posterity thought I would post my experience.
Just to summarise what others have already said, vmdk (virtual hard drives) can be growable or preallocated. Preallocated reserves the space on the host (create a 20GB partition in guest OS, and a 20GB .vmdk file will be generated). Growable will only take up as much space on the host OS as is needed (create the same 20GB partition in the guest, and put 4GB of files on it, and you will have a 4GB .vmdk file on the host system). Once you understand this, it becomes obvious that shrinking a preallocated disk makes no sense.
When you delete a file in the guest OS, all that happens is the space that file was taking up is marked as free space. The file contents still exist on the drive. The guest OS knows this is unused space, but the host system does not, so the space cannot be reclaimed. If you take our 20GB partition above with 4GB space used, and write another 10GB your .vmdk file will grow to 14GB. Delete that 10GB and the .vmdk file will still be 14GB. Write another 10GB to the partition and the .vmdk file will grow to 24GB, even though only 14GB of data is used. This is the danger with growable disks - they can grow larger than the partition sizes in your guest. This is what happened in my situation, and I ran out of space on the host.
To shrink a growable .vmdk file, you boot into the guest OS, and click the shrink tab in Vmtools. This prepares the partition for shrinking by running through the partition and zero-filling the contents of any files that have been deleted. Once you have done this, you power down the guest and run vmware-vdiskmanager -k source.vmdk. It then runs through your .vmdk file, and when it finds a string of 0's it knows that it is space taken by a now-deleted file, and it can be removed. This should bring your .vmdk file size back into line with the amount of space taken up on the partition in the guest OS.
I had the same error as the original poster when I ran VMtools from within the guest. I still don't know why, but the workaround I chose was to download a tool called sdelete. This does the same thing as the preparation step in VMtools. Usage: "sdelete -c -s" from a command prompt. Sdelete works by creating large new files, then writing their contents with 0's, then deleting them. I was almost out of space on the host, so when the large new files were created I completely ran out of space on the host, and the guest OS crashed and was no longer bootable (no room for the .vmem file). I had to go to a backup to get the guest booting again, and in the end I increased capacity on the host and was able to run sdelete (in my case I ran it on a 30GB partition with 13GB taken, which was a 60GB .vmdk file. After sdelete ran the .vmdk file ballooned to 90GB, then dropped to 16GB after the shrink completed). Sdelete took about an hour for me, and the shrink took about 25 mins.
My preference is to move to preallocated virtual disks, which will guarantee this problem doesn't happen again. You can do this by powering down the guest OS, and running the .vmk through the free converter provided by Vmware. Choose virtual to virtual, and point it to the .vmk file (I think you can also point it to a running guest, but I haven't tried this). It will read the .vmk file, and give you options which include being able to change the virtual disks to preallocated, and you can also choose to change their size. I have tried this previously, but it failed, probably due to other inconsistencies in my VM that are a separate issue.
I know this is a pretty dead discussion by now, but I found it helpful. I was not able to resolve my issue with this same error until removing all of my prior snapshots for this VM. Since the error message did not even mention this as a possibility of what might interefere, I thought I'd note it back to this conversation for posterity.