Answer: when Windows 11 pops up the notification, select the three-dot menu, and choose the option that turns off notifications for this USB device. Do this for all usb removeables that are freed up to Windows when the VM is shutdown. You can then shutdown the Guest OS without Windows nagging you to do something with Guest-connected devices.
Windows 10 you have to go to settings/bluetooth&devices/usb/notifications - off.