I am having USB crashes with my CDMA card and with using a WD external HDD. I wonder if this is related. I posted my question and bluescreen in another thread and have yet to hear back.
Fundamentally, there will be a bit more latency when running in a virtual machine as opposed to running natively, since there's another layer to go through. One thing that can help a bit is to plug USB devices directly into the Mac instead of going through a hub.
In case there's something else going on, though, please enable USB debugging (see Information Gathering for VMware Fusion if you need help) and attach the log.