Solved: Re: ESXi 6.5 USB Devices attached to VMs Issues? - Page 2

srwsol · ‎11-24-2016

Has anyone else experienced any issues with ESXi 6.5 and host USB 3 devices attached to VMs?

I'm seeing messages like this from an Arch Linux VM that has a number of USB 3 hard disks attached to it:

Nov 25 02:07:40 archnfs kernel: usb 1-4.1: reset high-speed USB device number 8 using xhci_hcd

There are quite a number of other error messages associated with it to, and I'm betting that data is being lost. I'm running a big test copy right now, and when it finishes I'm going to verify the md5sum versus the source to find out for sure.

jaytee75 · ‎12-06-2016

On 6.0U2 no problem, after update erros as in last posts.

I try again an update to 6.5 on weekend.

dmc72 · ‎12-08-2016

Hoping this gets fixed.think I'm having same/similar issue.

SuperMicro X9SCM motherboard. I am running UNRAID in a vm.

Started getting the "reset high-speed USB device number" when UNRAID moved to linux kernel 4.0.x and up.(tested on ESXI 5.5 / 6.0.u2)

discussion on UNRAID forum: http://lime-technology.com/forum/index.php?topic=40605.0‌

upgraded to ESXI 6.5 and am seeing these errors:

2016-12-08T10:27:39.333Z cpu4:65947)WARNING: usb_non_isoc_callback:1019: transfer error [22]: endpt = 0x81, nframes = 1, timeout = 5000, interval = 0, max_packet_size = 512, max_frame_size = 512: Broken pipe

and

2016-12-08T10:18:31.631Z cpu0:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:18:38.235Z cpu3:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:18:45.351Z cpu3:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:18:45.351Z cpu3:65948)uhub_explore_handle_re_enumerate:543: Failed to reset device usb0105, error=USB_ERR_TIMEOUT

2016-12-08T10:18:45.967Z cpu0:84037)WARNING: udev_reset_device:2214: Failed to set config index to 0: Bad parameter

2016-12-08T10:18:45.967Z cpu0:84037)WARNING: FreeBSDCharIoctl:1778: ioctl failed: dev = usb0105, cmd = 0x5514, err = -22: Failure

2016-12-08T10:18:48.059Z cpu3:84028)WARNING: udev_submit_urb:1555: usbd_do_request_flag error USB_ERR_TIMEOUT: Failure

2016-12-08T10:18:54.768Z cpu1:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:01.878Z cpu1:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:08.480Z cpu1:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:15.590Z cpu7:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:15.590Z cpu7:65948)uhub_explore_handle_re_enumerate:543: Failed to reset device usb0105, error=USB_ERR_TIMEOUT

2016-12-08T10:19:17.691Z cpu3:84028)WARNING: udev_submit_urb:1555: usbd_do_request_flag error USB_ERR_TIMEOUT: Failure

2016-12-08T10:19:24.360Z cpu7:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:31.468Z cpu7:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:38.074Z cpu7:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:45.183Z cpu7:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

2016-12-08T10:19:45.183Z cpu7:65948)uhub_explore_handle_re_enumerate:543: Failed to reset device usb0105, error=USB_ERR_TIMEOUT

2016-12-08T10:19:47.451Z cpu0:84028)WARNING: udev_submit_urb:1555: usbd_do_request_flag error USB_ERR_TIMEOUT: Failure

2016-12-08T10:19:54.149Z cpu4:65948)usbd_setup_device_desc:1987: getting device descriptor at addr 5 failed, USB_ERR_TIMEOUT

and

2016-12-08T07:47:34.119Z cpu0:81845)WARNING: FreeBSDCharIoctl:1778: ioctl failed: dev = usb0105, cmd = 0x8004550f, err = -2: Failure

2016-12-08T07:47:34.119Z cpu0:81845)WARNING: FreeBSDCharIoctl:1778: ioctl failed: dev = usb0105, cmd = 0x8004550f, err = -22: Failure

2016-12-08T07:47:37.251Z cpu6:81845)WARNING: usb_read:2654: Failed to get config descriptor 0 at bus 0, err 20: I/O error

2016-12-08T07:47:37.251Z cpu6:81845)WARNING: FreeBSDCharRead:1876: Failed to read 9 byte(s) from usb0105 [errno 19]: No such target on adapter

2016-12-08T07:47:37.251Z cpu6:81845)WARNING: FreeBSDCharIoctl:1778: ioctl failed: dev = usb0105, cmd = 0x8004550f, err = -2: Failure

I am only seeing the errors when i have a usb flash drive with the UNRAID O/S on it. all other usb devices working good.

if i delete the "ldlinux.sys" file from the flash drive all the errors stop. i can then attach and read the drive in any vm(plugging it into host +add usb device)

when i attach it to a windows 7 VM, i can boot normal and read from the usb flash drive. As soon as i run "makebootable.bat" this adds the "ldlinux.sys" file and the ESXI errors start up again.

I'm still a novice with ESXI + Linux, but if there is any info I can give you to help just let me know

jaytee75 · ‎12-11-2016

Updated to 6.5 again, Adding device...

then....

and reader disappears from list of hardware of vm-machine...

Also added this to vmx-file:

usb.generic.allowCCID = "TRUE"

lsusb on Host shows device correct:

Bus 001 Device 004: ID 076b:3021 OmniKey AG CardMan 3121

Bus 001 Device 003: ID 076b:3021 OmniKey AG CardMan 3121

dmesg shows:

....

2016-12-11T11:05:27.606Z cpu0:65907)WARNING: usb_non_isoc_callback:1019: transfer error [20]: endpt = 0x82, nframes = 1, timeout = 5000, interval = 0, max_packet_size = 64, max_frame_size = 64: Failure

2016-12-11T11:05:27.648Z cpu1:65907)WARNING: usb_non_isoc_callback:1019: transfer error [22]: endpt = 0x82, nframes = 1, timeout = 5000, interval = 0, max_packet_size = 64, max_frame_size = 64: Broken pipe

2016-12-11T11:05:28.973Z cpu1:65907)WARNING: usb_non_isoc_callback:1019: transfer error [20]: endpt = 0x81, nframes = 1, timeout = 5000, interval = 10, max_packet_size = 6, max_frame_size = 6: Failure

2016-12-11T11:05:29.570Z cpu0:65907)WARNING: usb_non_isoc_callback:1019: transfer error [20]: endpt = 0x81, nframes = 1, timeout = 5000, interval = 1, max_packet_size = 16, max_frame_size = 16: Failure

...

I add an other reader:

lsusb on Host:

Bus 002 Device 003: ID 104f:0004 WB Electronics Infinity Unlimited

Bus 002 Device 002: ID 076b:3021 OmniKey AG CardMan 3121

lsusb on client:

Bus 001 Device 003: ID 104f:0004 WB Electronics Infinity Unlimited

only one showing. OMNI 3121 didn't work. I tired to server. (HP N40L & Micorserver Gen8)

Seems to go back to 6U2.

jaytee75 · ‎12-12-2016

@ lancechou: Which type of omnikey you use?

different modell?

lancechou · ‎12-13-2016

Mime is "Omnikey AG CardMan 3121 076b:3021"

I just bought a different one, "Omnikey AG CardMan 3021 076b:3022". I will test it when I'm available.

Thanks,

Lance

lancechou · ‎12-13-2016

Omnikey AG CardMan 3021 doesn't work. I'll be working on it when I'm available later.

Thanks for reporting the issue.

Lance

jaytee75 · ‎12-15-2016

Any news? I am still on 6.5, maybe I can beta test things for you.

lancechou · ‎12-15-2016

Thanks. I've been busy on other issues, plan to start working on this one tomorrow (Fri)

-Lance

jaytee75 · ‎12-20-2016

Any news? May there a solution soon?

Thanx in advanced.

lancechou · ‎12-20-2016

Still debugging. I hope I could root cause it by end of tomorrow.

-Lance

lancechou · ‎12-21-2016

Not the driver issue but an issue in the virtual device layer. The device does not work with WS as well. I'm still looking.

-Lance

lancechou · ‎12-21-2016

Hi jaytee75,

I spent the whole day debugging the smart card issue and here is the story.

There're two versions of Omnikey 3121 Smart card readers. The pid:vid of the old one is 076b:3021 while the new one is 076b:3022. You can use 'lsusb' command to find out which one yours is.

I used a ubuntu VM to test both readers. The old one always works while the new one always doesn't, where 'works' means it can pass pkcs11_inspect in the ubuntu VM.

When I tested the new one with pkcs11_inspect, it complains about 'there are no slots available'. I've confirmed that its not the usb host controller driver issue although you can see 'transfer error [20]...' in vmkernel.log. Those errors don't mean anything but there's no card activities in the past 5 seconds, so the queued transfer requests timed out. Same thing on VMWare Workstation. the old one works while the new one still doesn't. So, it's not caused by the driver change in ESX between 6.0 and 6.5. Otherwise, Workstation would have had the same issue.

I also used a hardware analyzer to debug why the new one did not work and found the virtual machine layer did everything right. It handled all the requests from the upper layer, guest OS in this case. The difference between the new one and the old one is that there's no CCID activities between the new reader and the guest OS. Then I switched to Windows guest for the same test. Windows 10 at first did not recognized the new reader but worked well after upgrading the driver which again convinced me it's might be a guest driver issue. You can check out the post at Bug 1276726 – Omnikey 3121 with PID 0x3022 support. ccid 1.4.18 is required to make the new one work. Ubuntu 14 bundles with ccid 1.4.7 which does not support the new one.

You mentioned it used to work in 6.0 but not in 6.5 which I don't know why. But I think it's better off knowing what exactly the reader you're using. You can use 'lsusb' to find out the vid:pid of your reader. Also, run '/etc/init.d/pcscd status'. If you want to passthru the reader to a guest.

Thanks,

Lance

jaytee75 · ‎12-22-2016

Hi Lance,

thanks for your very good investigation.

I am using a Ubuntu 16.10 Server on Client.

I just had a look in libccid.so and found as version string "Driver version: 1.4.24".

lsusb shows on Host:

Bus 002 Device 002: ID 076b:3021 OmniKey AG CardMan 3121

/etc/init.d/pcscd status on Host shows

pcscd is running

lancechou · ‎12-22-2016

Hi jaytee75,

You're not supposed to passthru smart card reader to a VM while pcscd is running.

Please note that connecting a reader to a VM requires the reader being disconnected from the host. if pcscd (on host) is running at the moment, the host is using it so the guest cannot claim it.

I don't know if pcscd is disabled by default in 6.0 but enabled by default in 6.5. But pcscd is a service on the host to enable you to log in to the dcui with smart card. Due to this reason, smart card passthru on ESXi host is disabled by default.

Thanks,

Lance

jaytee75 · ‎12-22-2016

hey,

in gui there is disabled on ssh shell status of pcscd is running

:smileyshocked:

so killing via bash and see what happen...

jaytee75 · ‎12-22-2016

so. also after reboot pcscd runs again. on gui its shown as stopped.

So I do /etc/init.d/pcscd stop an rm /etc/init.d/pcscd so that pcscd never starts again...

But unfortunaly I can' t connect the omni reader to client. there appears the red banner with "can't find path to reader"...

I also had reconnect and restart host. Any finally hint for me?

lancechou · ‎12-23-2016

Hi Jaytee75,

You shouldn't have deleted /etc/init.d/pcscd.

I don't know where the problem is because it always works for me.

I think there're few things you can do.

1. Log in to dcui (F2) -> Troubleshooting Options -> Enable ESXi Shell

2. Switch to Control 1 (Alt-F1) and log in

3. Open your VM settings, mime is located in /vmfs/volumes/datastore/Ubuntu-14.10/Ubuntu-14.10.vmx <-- Make sure the VM is powered off before you make any change

usb_xhci.autoConnect.deviceX = "path:x/x/x autoclean;x" <--- Removing lines containing autoConnect

<--- "can't find path to reader" probably comes from the line with non-existing path

4. make sure smart card reader patthru is allowed

usb.generic.allowCCID = "TRUE" <-- make sure you have this line

5. save .vmx and power on the VM

If things still don't work, please add one more line to .vmx

usb.analyzer.enable = "TRUE"

and attach vmware.log and /var/log/usb.log

Thanks,

Lance

jaytee75 · ‎12-23-2016

Yo did it!

Thanks! Now everything is working fine!!!!

Merry Christmas!

lancechou · ‎12-27-2016

Glad to hear that! Merry Christmas!

ITA003 · ‎01-04-2017

Hi,

I've the same problem with my VMGuest. I can not see the USB attached to Host.

the lsusb command on guest return:

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

In the vmx conf file there are these lines:

usb.pciSlotNumber	33
usb_xhci.pciSlotNumber	224
usb.generic.allowCCID	TRUE
usb_xhci:4.present	TRUE
usb_xhci:4.deviceType	hid
usb_xhci:4.port	4
usb_xhci:4.parent	-1

Do you have any idea to help me solve this issue?

Thanks
Mino

All

ESXi 6.5 USB Devices attached to VMs Issues?