We also have vGPU machines but have never had an issue with the driver when installing an application.
I have no idea what the implications would be if you leave the agent there.
When starting with Appvolumes we never removed the agent on the packaging machine and haven't had issues with applications.
