It is not problem to dis/connect it and use with other VM. But it is not possible to use it with more VMs at the same time.
USB does not support multi-hosting, so any usb-device may be at particular time attached only to single usb-host, be it physical or VM. The only way how to circumwent this limitation (for usb-dongles/sticks/disks) is to get usb-to-ethernet converter (or usb-to-sata & sata-to-ethernet) to create network-storage.