I tried to use VMware 15.5.6 for gaming, host ubuntu20, guest win8.1.
added mks.gl.allowBlacklistedDrivers = "TRUE" and 3d work good.
some game says about videocard incompatibility. In other applications did not work antialiasing.
Question: are possible, make videocard (driver) more compatible? maybe registry editing, or I don't know about aby better driver.
HW: radeon rx 480, amdgpu driver.
Version 16 supports DX11 in Windows guests but it requires Nvidia on a Linux host.
Anyway, working with what you have, version 15.5.x should support DX10.1 with MSAA with 4 samples.
You can see from edge://gpu in Edge browser or chrome://gpu in Chrome browser, the SVGA 3D driver version and MSAA sample count. I don't have an AMD graphics card. But on an Ubuntu 18.04 host with Nvidia graphics card with Workstation 15.5.x, it shows Max. MSAA samples as 4 and dxdiag shows 10_1 in the feature levels in a Windows 10 VM.
Make sure the virtual hardware version is version 16 (should be 16 if the VM was created in version 15.x) and that the latest SVGA 3D driver is installed.