Have you considered installing the Horizon agent directly onto the physical PC and brokering the connection directly to that rather than using VDI as a proxy/bastion host to reach the desktop? This would make for a much better experience.

Using Horizon 7 to Access Physical Windows Machines | VMware