I have the exact same problem with an El Capitan guest under Fusion 11.5. Apparently, the VMware Tools 11.0 doesn't install a working graphics driver for El Cap guest, and so you are stuck with only 1024x768 resolution. I tried reinstalling VMware Tools several times.
After I reverted to Fusion 11.1.1, and reinstalled VMware Tools 10.3.10 on the guest, everything was back to normal for the display resolutions.
FYI, here is a direct link to the VMware Tools package from Fusion 11.1.0
Wil| Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
| Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
| More info at https://www.vimalin.com
| Twitter @wilva
| VMware Wiki at http://www.vi-toolkit.com
I gave up too :
- Went to time machine to restore Fusion v11.1
- Restored a snapshot taken just before upgrade of VM to v11.5 tools
…and I'm good to go, it seems.
I just lost 2 hours with the v11.5 tools install+reinstall+uninstal+reinstal and many, many VM restarts, Fusion quit+open, Mac restarts, etc.
I have not tried to fix it yet as I am not prepared to upgrade anytime really soon to Catalina anyway – must be the worst (or most worthless/costly) macOS new system for a very long time. But I have clean macOS environments already available for when I want to reinstall the older VMware Fusion, but was hoping that maybe VMware would fix some of the problems soon so I did not need to go back ... . Maybe in version 12, but they seem more focussed on other things. (It is odd that good functionality is taken away, like when they took away the obvious automatic attachment of USB-connected devices that many relied on without notice.)
VMWare's aware of the issue and working on it...no eta.