I think you are on the right path with a VDI type solution.
The VDI will provide you with a controlled access platform, somewhere that can be controlled, services can be monitored, you can ensure compliance, AV, AntiMalware, IPS and IDS, syslog solutions could be installed.. The solution could even be made non-persistent, in that the VDIs could be potentially deleted after a call out (although that might introduce a risk)... Or suspended and access removed after a call out had been completed, so forensics could be run perhaps?
If you had a secure VPN type solution with a second factor of authentication and permitted connections only based upon passing certain criteria (such as AV install on the non-trusted device etc...). You could plan for and think of VDI solution as taking your non-trusted device and providing it a trusted connection. You could go further perhaps and make the non-trusted device a thin client even?
Something similar to passing through airport security perhaps. What controls you place around this is entirely dependant on the risk profile. However, I would suggest providing a more secure administration point is preferable to not trusting any devices (e.g. laptop at home, even desktop in office for daily use) What's the alternative? Each administrator in front of a KVM in the data centre?
Best of Luck