Greetings!
I am using ESXi 6.7.0 Update 3 (Build 15160138) on an Intel Nuc to host my VMs wit the free ESXi hypervisor license.
The Nuc has an internal M2 SSD. As storage was running low, I've added another external SSD via a JMicron USB2SATA adapter.
Basically I was following this guide: https://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/
So, disabling usbarbitrator, adding SSD as storage device, etc.
Everything works as expected.
However, now I cannot anymore passthrough USB devices into a VM as the arbitrator service is no longer running:
[root@esxi:~] esxcli hardware usb passthrough device list
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
--- --- -------- --------- ------- --------------------------------- -------------------------------------------------------------------
1 2 76b 3021 true no (usbarbitrator is not running) OmniKey AG CardMan 3121
1 3 8087 a2a true no (usbarbitrator is not running) Intel Corp.
1 4 152d 1561 false no (usbarbitrator is not running) JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U
Specificially I would like to make the OmniKey CardMan available as an USB device in a VM. The JMicron device at the bottom is the adapter that connects my SSD via USB3.0 to the host.
Now, lets start the usbarbitrator again:
[root@esxi:~] /etc/init.d/usbarbitrator start
UsbUtil: Attempting to reserve for VMkernel USB adapter(s):
Errors:
Rescan complete, however some dead paths were not removed because they were in use by the system. Please use the 'storage core device world liommand to see the VMkernel worlds still using these paths.
usbarbitrator started
Hostd must be restarted to enable USB passthrough functionality on any VM. After hostd is restarted, to enable USB passthrough functionality oowered on VM, the VM must be powered off and powered on again.
[root@esxi:~] esxcli hardware usb passthrough device list
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
--- --- -------- --------- ------- ----------------- -----------------------------------------------------------------------------------
1 2 76b 3021 true yes OmniKey AG CardMan 3121
1 3 8087 a2a true yes Intel Corp.
1 4 152d 1561 true yes JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U
This looks promising, as I am now apparently able to "mount" the Omnikey into a VM.
However, I can no longer access the UI anymore once the usbarbitrator has been restarted. I presume this is because the SSD/storage device is now kind of blocked.
As soon as I stop the arbitrator service again, the UI is responsive again.
So I assume I need to kind of mask/blacklist the JMicron adapter from the USB arbitrator so he allows the mounting of all the connected USB devices but NOT of the JMicron. The JMicron should not be arbitrated but be available exclusively to the host.
How can I achieve this? Which commands do I need to issue?
As the message on the screen states I also tried to restart the hostd service, however the ESXi was then no longer accessible at all, giving me an error when I opened the webpage:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x0000001987322480] _serverNamespace = / action = Allow _port = 8309)
After a reboot he UI was available again, but the arbitrator was again disabled.
I found these commands but am not sure whether this is the correct way to go: Re: Disable USB support on ESXi OS or BIOS, both?
You could disable the usb-storage module by using
esxcli system module set -m usb-storage -e false
( followed by
vmkload_mod -u usb-storage
or a reboot to make the change effective )
This would prevent the usage of any USB storage devices without affecting KVM.
I am not sure whether I can issue this command without doing any harm.
Kindly advise how to proceed.
Thank you!
By analyzing the bash script /etc/init.d/usbarbitrator and looking and evaluating all the commands I managed to achieve my goal.
Not sure whether this is the intended way but....
First: No changes to the config of usbarbitrator -> stays disabled as in the HowTo Guide for adding a SSD via USB adapter.
Either enter directly in the bash on the ESXi or write a small bash script for the following commands
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
USBARBITRATOR=/usr/lib/vmware/bin/vmware-usbarbitrator
USBARBITRATOR_TAG=usbarbitrator
USBARBITRATOR_RP="usbArbitrator"
MAX_RETRIES=5
PASSTHROUGH_DEV=/vmfs/devices/char/vmkdriver/usbpassthrough
# Disable passthrough for JMicron
localcli hardware usb passthrough device disable --device 1:4:152d:1561
# Enable passthrough for JMicron
localcli hardware usb passthrough device enable --device 1:2:76b:3021
/sbin/watchdog.sh ++memreliable,group=${USBARBITRATOR_RP} -d -s "${USBARBITRATOR_TAG}" -t "${MAX_RETRIES}" "${USBARBITRATOR}" -t "${MAXIMUM_SUPPORTED_CLIENTS_PARAM}"
localcli hardware usb passthrough device list
Outputs now
UsbUtil: VmkuserCompat_PClose in UpdateDeviceNamesfailed, reason: Bad file descriptor
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
-----------------------------------------------------------------------
1 2 76b 3021 true yes OmniKey AG CardMan 3121
1 3 8087 a2a true yes Intel Corp.
1 4 152d 1561 false no (passthrough disabled) JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U
No hostd restart is necessary!
As you can see the OmniKey CarMan is now available for mounting, the JMicron is not - as desired.
I can now select the OmniKey as USB device, however when booting the VM I get another error, that the device cannot be mounted.
The answer is in here:
https://communities.vmware.com/thread/560091
The following line must be added to the VM Config
usb.generic.allowCCID = "TRUE"
I've also had the other lines like "usb.autoConnect.device0" that are mentioned in the post but I did NOT need to remove these.
No, when booting the VM I can up the USB device and it is shown successfully when I issue "lsusb" on the VM
Case closed 😄
By analyzing the bash script /etc/init.d/usbarbitrator and looking and evaluating all the commands I managed to achieve my goal.
Not sure whether this is the intended way but....
First: No changes to the config of usbarbitrator -> stays disabled as in the HowTo Guide for adding a SSD via USB adapter.
Either enter directly in the bash on the ESXi or write a small bash script for the following commands
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
USBARBITRATOR=/usr/lib/vmware/bin/vmware-usbarbitrator
USBARBITRATOR_TAG=usbarbitrator
USBARBITRATOR_RP="usbArbitrator"
MAX_RETRIES=5
PASSTHROUGH_DEV=/vmfs/devices/char/vmkdriver/usbpassthrough
# Disable passthrough for JMicron
localcli hardware usb passthrough device disable --device 1:4:152d:1561
# Enable passthrough for JMicron
localcli hardware usb passthrough device enable --device 1:2:76b:3021
/sbin/watchdog.sh ++memreliable,group=${USBARBITRATOR_RP} -d -s "${USBARBITRATOR_TAG}" -t "${MAX_RETRIES}" "${USBARBITRATOR}" -t "${MAXIMUM_SUPPORTED_CLIENTS_PARAM}"
localcli hardware usb passthrough device list
Outputs now
UsbUtil: VmkuserCompat_PClose in UpdateDeviceNamesfailed, reason: Bad file descriptor
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
-----------------------------------------------------------------------
1 2 76b 3021 true yes OmniKey AG CardMan 3121
1 3 8087 a2a true yes Intel Corp.
1 4 152d 1561 false no (passthrough disabled) JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U
No hostd restart is necessary!
As you can see the OmniKey CarMan is now available for mounting, the JMicron is not - as desired.
I can now select the OmniKey as USB device, however when booting the VM I get another error, that the device cannot be mounted.
The answer is in here:
https://communities.vmware.com/thread/560091
The following line must be added to the VM Config
usb.generic.allowCCID = "TRUE"
I've also had the other lines like "usb.autoConnect.device0" that are mentioned in the post but I did NOT need to remove these.
No, when booting the VM I can up the USB device and it is shown successfully when I issue "lsusb" on the VM
Case closed 😄
Wow - thank you for this guide!
I followed it all the way through and it worked as described!
I thought I had maxed out my motherboard using a USB Datastore for an OmniOS/Napp-it VM and passed through the SATA controller. It seamed such a waste to lose all the USB ports, but now I can pass through an external USB drive to my backup VM for backup copies.
I noticed that If a USB device is removed and reconnected, the device is not automatically re-enabled for passthrough and not recognised by the VM. To reconnect the device I have to rerun the command:
localcli hardware usb passthrough device enable --device [Bus:Dev:VendorID:ProductID]
and then in ESXi, remove (if device is still listed) and re-add the device
The device then re-appears in the VM, without restart.
Is there a way to allow the device to be automatically re-enabled for passthrough and reconnect to the VM?
Thanks again!
To be honest - no idea. I am just a private user of the free ESXi and far from being a pro for this product. Happy that everything runs as it should now. Hope you can find a solution for your problem as well. Best of luck!
I also noted that after a reboot of the ESXi the following commands need to be re-executed in order to allow the pass-through of USB devices again.
Not 100% sure if it might work if I just re-enable the usbarbitrator service to start automatically (-> need to check):
[root@esxi:~] export PATH=/bin:/sbin:/usr/bin:/usr/sbin
[root@esxi:~] USBARBITRATOR=/usr/lib/vmware/bin/vmware-usbarbitrator
[root@esxi:~]
[root@esxi:~] USBARBITRATOR_TAG=usbarbitrator
[root@esxi:~]
[root@esxi:~] USBARBITRATOR_RP="usbArbitrator"
[root@esxi:~]
[root@esxi:~] MAX_RETRIES=5
[root@esxi:~]
[root@esxi:~] PASSTHROUGH_DEV=/vmfs/devices/char/vmkdriver/usbpassthrough
[root@esxi:~] /sbin/watchdog.sh ++memreliable,group=${USBARBITRATOR_RP} -d -s "${USBARBITRATOR_TAG}" -t "${MAX_RETRIES}"
"${USBARBITRATOR}" -t "${MAXIMUM_SUPPORTED_CLIENTS_PARAM}"
[root@esxi:~] localcli hardware usb passthrough device list
UsbUtil: VmkuserCompat_PClose in UpdateDeviceNamesfailed, reason: Bad file descriptor
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
-----------------------------------------------------------------------
1 2 76b 3021 true yes OmniKey AG CardMan 3121
1 3 8087 a2a true yes Intel Corp.
1 4 152d 1561 false no (passthrough disabled) JMicron Technology Corp. / JMicron USA Technology
Corp. JMS561U
[root@esxi:~]
I might investigate further at some point in the future, but for now I'm happy.
I had tried restarting the host and simply re-enabling the USB arbitrator service
/etc/init.d/usbarbitrator start
Output:
UsbUtil: Attempting to reserve for VMkernel USB adapter(s): vmhba33
Errors:
Rescan complete, however some dead paths were not removed because they were in use by the system. Please use the 'storage core device world list' command to see the VMkernel worlds still using these paths.
usbarbitrator started
Hostd must be restarted to enable USB passthrough functionality on any VM. After hostd is restarted, to enable USB passthrough functionality on a powered on VM, the VM must be powered off and powered on again.
and it broke the USB Datastores, I restarted the host and the USB Datastores were back up.
I also tried to enable the USB Arbitrator after entering your commands, it then outputs:
usbarbitrator already running
no change
On each reboot of my host, the USB device's Dev number changes, so I would need to check on reboot and adjust the enable/disable commands accordingly.
I would like to write a batch script that runs on ESXi host startup, to pass through all USB devices except for specific devices used for USB Datastores. I have no idea where to start with this though.