I have an ESXi 6.5 installed on an HP dl360-GEN8 server with a DELL KVM console attached to it and other two servers DELL.
If I switch the KVM on HP server the monitor returns me the "Out of range" error.
On the other two servers DELL (on which runs the same version of esxi 6.5) there are any problem.
In order to avoid any compatibility with hardware, I started the system with pendrive with a Linux SO, and after changing the monitor resolution to "800 x 600", the problem solved. This clarify that the problem is ESXi.
I tried to change the "xorg.conf" file to change video resolution but it reply I have no permission to modify the file, even if I login with administrator password.