I suspect Workstation is/was using the Intel P630 (assuming it is enabled on motherboard UEFI) of the E2174G and not using the Nvidia P1000 when the nouveau driver was installed. Workstation 16 can use Vulkan for Intel Kaby Lake or newer iGPU. Even without a monitor connected to the Intel iGPU, Workstation can still use Intel iGPU as Vulkan render device. Most motherboards let you disable the Intel iGPU at the BIOS/UEFI.
The mksSandbox.log could look like this for getting Vulkan ready. And it will use the first Vulkan device it finds (assuming no prior changes to vmx configuration files).
<timestamp> In(05) mks Vulkan Renderer: VKRDevice_Create trying DISPLAY=:1
<timestamp> In(05) mks Vulkan Renderer: Available Vulkan devices:
<timestamp> In(05) mks Vulkan Renderer: Vulkan Device 0:
<timestamp> In(05) mks Vulkan Renderer: Vendor= 'Intel'
<timestamp> In(05) mks Vulkan Renderer: ID= 0x3e92
<timestamp> In(05) mks Vulkan Renderer: Type= 0x1
<timestamp> In(05) mks Vulkan Renderer: Name= 'Intel(R) UHD Graphics 630 (CFL GT2)'
<timestamp> In(05) mks Vulkan Renderer: Vulkan Device 1:
<timestamp> In(05) mks Vulkan Renderer: Vendor= 'Mesa'
...
<timestamp> In(05) mks Vulkan Renderer: Vulkan Device 2:
<timestamp> In(05) mks Vulkan Renderer: Vendor: 'NVIDIA'
...
<timestamp> In(05) mks Vulkan Renderer: Using device 0 of 3
It shows Using device 0 meaning the Intel iGPU in this example as it is Vulkan Device 0. If the Nvidia is used, it should show instead
<timestamp> In(05) mks Vulkan Renderer: Using device 2 of 3
You could shut down the VM and add the following lines to the vmx configuration file
mks.forceDiscreteGPU = "TRUE"
mks.vk.allowUnsupportedDevices = "TRUE"
The mks.forceDiscreteGPU is so that the dGPU like Nvidia is used. The mks.vk.allowUnsupportedDevices line is to let Workstation use Vulkan with Nvidia P1000 (since it is Pascal) with Nvidia proprietary driver; I think nouveau driver will also require this line even if the GPU is Turing or newer.
Try this first see if it resolves the problem.
I don't know why you are getting the mksSandbox timeout. The default behaviour is Workstation will try Vulkan first and fallback to OpenGL if Vulkan fails (such as having a Pascal GPU without the vmx configuration change). There are other community posts that has this problem and turning off Vulkan rendering makes the problem go away.
mks.enableVulkanRenderer = "FALSE"
mks.enableGLRenderer = "TRUE"
Aside from looking through mksSandbox.log, you can also use nvidia-smi to confirm if the Nvidia GPU is used. You should see mksSandbox process(es) for every running VM with 3D enabled. For Vulkan, the type is C+G, for OpenGL the type is G.
