Old topic, I know.
Installed ESXi 3.5u5 on a dual Xeon 2.2GHz system with only 1GB of RAM and ran into this same issue.
I set the vim memory reservation down to 64MB as that was what the system seemed to be using when I had one VM running. Horrible idea.
System ran okay still with that one VM, and actually somewhat okay with a second one, but the third really killed things.. even after turning off #2.
VIClient had all sorts of weird issues, unable to connect half the time, other half the time I could get through a few pages but run into problems after clicking on a couple screens.
Ended up bumping the vim reservation up to 256MB, and all is doing okay so far. VIclient is FAR more responsive..about as good as it is on my dual Xeon 2.8GHz w/ 8GB. I'll probably end up adding some more RAM to it, but I'll have to pick up a pair of 512/1GB sticks. All I've got besides the 2x512 that are in it are some 2GB sticks, but I'd rather use those on my pair of dual Opteron 275 machines.
