Dear community Members,
you shouldn't be running it from within a VM but use a bootable CD and run it outside of ESX. That way you will test all memory areas instead of just a subset.
Duncan (VCDX)
Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive
No, as Duncan said, you should boot the ESX server from the linux ISO and run memtest natively. Also, you need to run the test for 24-48 hours to really have accurate results.
some server has memtest built into the BIOS. so you may not need the hassle to get a bootable CD. (useful if you're far physically from the server).