I went through the KB you attached, and verified the RAM setting in the vmx file.
for a resolution of 1024 x 1280, I would need a svga.vramsize = 1024 x 1280 x 4 = 5242880.
If I set the Display memory to Auto-Detect video settings, it puts svga.vramsize = 16777216 in the vmx file.
If I enter total video RAM to 48MB, it puts svga.vramsize = 50331648.
These vramsize settings should be plenty for the resolution I'm looking for.
I tried 2 different ESXi clients. One is running xp and one is running Win7. Both had the same results.