If I remember correctly, VMware stated in the tech preview that shared folders might be a bit sluggish on Apple Silicon as the "high bandwidth" communications channel was not optimized for arm64. I'm suspecting that this didn't get addressed for Fusion 13.

What happens if you try Windows file share configured on the Mac and see if the situation improves.

The other idea is to use something like an sftp client...