I have this same question for ESXi 5.1, except for a different reason.
I can use the installer and the host video in the default mode, so this isn't urgent, it's just annoying.
I'd like to reduce the text layout geometry of the display from 48x128 to something like 25x80. I think I should be able to do this by reducing the amount of video memory from the host system's default of 16Meg to something like 2Meg.
Is there a boot option that can define the video memory size that the host uses ?