Issues with vmkusb on 7.0U1c with SD Card boot

[MillardJK]

Ever since upgrading my hosts to 7.0.1 b17325020, I've had issues with my boot-from-SD Card devices. In a nutshell, they disappear from the host at the adapter level (vmhba32 is no longer present in the storage device list), and only a reboot brings it back. In one host, I've replaced the media with new cards (time consuming but easy when the hardware is using RAID1 for a pair of them) on the off-chance that it was the age of the media that was causing it. This is a lab system, so I can't bring Support into the mix to figure it out, but I think something is misbehaving with the interaction between the latest vmkusb driver and the SD Card module (the hosts are Dell R620 with latest firmware/iDRAC/LCM), which shows as online & nominal in hardware management.

This all came to light when trying to upgrade to b17325551, and discovering that the bootbank was missing, so the update(s) couldn't be applied. Rebooting and then doing the update "mostly" worked: for some crazy reason, some updates--like getting VMware Tools written to the device--weren't succeeding.

I'm wondering if anyone else is seeing that sort of thing. It may be that the slow performance of the device is causing timeouts (the media is all C10/U1 or C10/U3, so the write performance is high for the class of media) that weren't "fatal" in older versions of the driver.

[MillardJK]

More details: I've discovered that cycling the USB Arbitrator service brings the adapter back...

[root@esx3:~] esxcli storage core adapter list
HBA Name  Driver          Link State  UID                                     Capabilities         Description
--------  --------------  ----------  --------------------------------------  -------------------  -----------
vmhba0    vmw_ahci        link-n/a    sata.vmhba0                                                  (0000:00:1f.2) Intel Corporation Patsburg 6 Port SATA AHCI Controller
vmhba1    lsi_mr3         link-n/a    sas.5848f690e685cd00                                         (0000:02:00.0) Broadcom PERC H710P Mini (for monolithics)
vmhba2    intel-nvme-vmd  link-n/a    pscsi.vmhba2                                                 (0000:41:00.0) Intel Corporation NVM Express PCIe SSD DC P3600 AIC
vmhba3    intel-nvme-vmd  link-n/a    pscsi.vmhba3                                                 (0000:42:00.0) Intel Corporation NVM Express PCIe SSD DC P3600 AIC
vmhba64   qfle3i          online      iscsi.vmhba64                           Second Level Lun ID  QLogic 57800 10 Gigabit Ethernet Adapter
vmhba65   qfle3f          link-down   fcoe.200044a8420a3ced:200144a8420a3ced  Second Level Lun ID  () QLogic Inc. FCoE Adapter
vmhba66   qfle3i          online      iscsi.vmhba66                           Second Level Lun ID  QLogic 57800 10 Gigabit Ethernet Adapter
vmhba67   qfle3f          link-down   fcoe.200044a8420a3cef:200144a8420a3cef  Second Level Lun ID  () QLogic Inc. FCoE Adapter
[root@esx3:~] /etc/init.d/usbarbitrator stop
watchdog-usbarbitrator: Terminating watchdog process with PID 2102587
stopping usbarbitrator...
usbarbitrator stopped
[root@esx3:~] /etc/init.d/usbarbitrator start
usbarbitrator started
[root@esx3:~] esxcli storage core adapter list
HBA Name  Driver          Link State  UID                                     Capabilities         Description
--------  --------------  ----------  --------------------------------------  -------------------  -----------
vmhba0    vmw_ahci        link-n/a    sata.vmhba0                                                  (0000:00:1f.2) Intel Corporation Patsburg 6 Port SATA AHCI Controller
vmhba1    lsi_mr3         link-n/a    sas.5848f690e685cd00                                         (0000:02:00.0) Broadcom PERC H710P Mini (for monolithics)
vmhba2    intel-nvme-vmd  link-n/a    pscsi.vmhba2                                                 (0000:41:00.0) Intel Corporation NVM Express PCIe SSD DC P3600 AIC
vmhba3    intel-nvme-vmd  link-n/a    pscsi.vmhba3                                                 (0000:42:00.0) Intel Corporation NVM Express PCIe SSD DC P3600 AIC
vmhba32   vmkusb          link-n/a    usb.vmhba32                                                  () USB
vmhba64   qfle3i          online      iscsi.vmhba64                           Second Level Lun ID  QLogic 57800 10 Gigabit Ethernet Adapter
vmhba65   qfle3f          link-down   fcoe.200044a8420a3ced:200144a8420a3ced  Second Level Lun ID  () QLogic Inc. FCoE Adapter
vmhba66   qfle3i          online      iscsi.vmhba66                           Second Level Lun ID  QLogic 57800 10 Gigabit Ethernet Adapter
vmhba67   qfle3f          link-down   fcoe.200044a8420a3cef:200144a8420a3cef  Second Level Lun ID  () QLogic Inc. FCoE Adapter
[root@esx3:~]

Note how vmhba32 returns through this sequence. I've also noted that the BOOTBANK file links are once again responsive, and failed dependencies (like running a configuration backup) are now succeeding. It's not perfect, because the adapter is still disappearing again after some amount of time... I'm going to stop the arbitrator on one of them to see if that helps any.

[MillardJK]

More data: I've discovered that stopping the USB Arbitrator daemon is sufficient to bring the card reader back online (not a stop/start). I have three hosts, so I have also seen where stopping the Arbitrator daemon seems to keep the device from going offline, while stop/start isn't any better than "just leaving it on" like a normal boot.

Starting to think that the issue may be more related to the Arbitrator daemon than the vmkusb driver. At least, I know I can live without the USB arbitrator--not trying to give any VMs access to host USB--while living without the SD card reader is more problematic.

[MillardJK]

Over 24h of running with the arbitrator disabled, and zero issues with the USB adapter.