Riachard, Welcome to the community..!!
Did you check the network latency between you vMA VM-network and the target network .? Or please provide the vma-support bundle to troubleshoot more.
~dGeorgey
From my experience since using VIMA (the vMA predecessor) this has always been "normal" behavior and has nothing to do with bandwidth/latency etc. Every command you run against a host with vifastpass initiates a new authentication session and that seems to take quite a bit of time.
I usually gets immediate response. ~dGeorgey.