WHenever I switch to my guest, the computer will stall for around 2/3s and the CPU jumps to around 60%. 

I am Linux and the guest is Windows 10 with guest tools installed. 

Once I am over this "spike" performance inside the VM is perfectly natural and in no way slow.

Here you see me entering the guest and then doing some things, then leaving after maybe 10 seconds.