ESXi 8 and Yubikey

[SchuFire]

Brand new esxi 8. Attempting to connect PIV card (Yubikey). PIV is physically attached to via USB-c to the esxi host computer. The Yubikey is attached to the target guest Windows 10 workstation. The Yubikey manager on the workstation can see the Yubikey and manipulate the OTP and FIDO2 stuff. When clicking on PIV, a red banner with "Failed connecting to the Yubikey. Make sure the application has the required permissions"

I have the following in the vmx file

usb.generic.allowHID = "TRUE"
usb.generic.allowLastHID = "TRUE"
usb.generic.CCID = "TRUE"

What am I missing?

[michelkeus_stwg]

Hi ShuFire,

This question has been asked before: Not able to passthrough a Yubikey 

I've tested the following solution from that topic: 

1. Edit /etc/vmware/config

usb.quirks.device0 = "0x1050:0x0407 allow"

2. Edit /bootbank/boot.cfg and changed the kernelopt line to 

kernelopt=autoPartition=FALSE CONFIG./USB/quirks=0x1050:0x0407::0xffff:UQ_KBD_IGNORE

Both of these settings do require a reboot. So I would change both of them and restart the ESXi server once

---

I've tested this solution on my vSphere 7 host in a vCenter 8 configuration with a freshly installed Windows 10 Pro.

I was able to add the Yubikey as an USB device to my VM:

With the above configuration changes and simply adding the USB device to the VM I was able to use the Yubikey for both FIDO2 (tested with the Azure Portal) and for PIV (tested by using the certificate on the Yubikey to sign into a Fortigate using PKI).

I did no other changes, not even the "usb.generic" ones you listed.

[SchuFire]

Thanks for the response.  

I did see those posts and implemented them as you stated. I can assign  the Yubikey to my Win 10 guest and the FIDO2 and OTP parts work fine.  However, when I try to access the PIV section, I get the "Failed connecting to the Yubikey. Make sure the application has the required permissions." message. I am trying this against an esx 8 host on an Intel NUC.

[SchuFire]

I did notice that   esxcli hardware usb passthrough device list   does not show the Yubikey at all. 

Running   esxcli hardware usb passthrough device enable -d 2:3:1050:407  does not return an error but does not add it to the list either.

[michelkeus_stwg]

Hi @SchuFire,

The changes I made and are working for me also do not show it in the passthrough list. I believe the original post also mentioned that. So it not showing in that specific list on the CLI would be to be expected.

So you implemented the 2 steps in my quote and then rebooted the system too?

Can you try this:

1. Removed the changes you made to the VM including adding the Yubikey already (so the usb.generic.allowHID = "TRUE", usb.generic.allowLastHID = "TRUE" and usb.generic.CCID = "TRUE")
2. Then shutdown the VM, remove it from the inventory and then re-add it to the inventory (this forces the system to re-read the VMX-file).
3. After that try re-adding the Yubikey to the VM via GUI alone. 

Also is your system licensed and if so what kind of license?

[SchuFire]

First off, I really appreciate your feedback and suggestions. 

What I did:

1. Unregistered the Win10 VM

2. Edited the vmx and removed *all* references to usb

3. Verified   '  usb.quirks.device0 = "0x1050:0x0404 allow "  '  (without the single quotes) was in the /etc/vmware/config file of the ESX 8 host.

4. Verified  '  CONFIG./USB/quirks=0x1050:0x0404::0xffff:UQ_KBD_IGNORE  '  (without the single quotes) was in the proper location in the /bootback/boot.cfg file of the ESX 8 host

5. Rebooted the ESX 8 host

6. Registered the WIn10 VM back

7. Added back a USB 2.0 controller to the Win10 VM  (Note: I also followed this same process adding the 3.1 controller - same result)

8. Added the Yubikey USB device (which the ESX interface correctly sees) to the Win10 VM

9. Powered up the Win10 VM

10. The Yubikey does not show in the device mangler or through the Yubikey Manager. 

11. If I add '   usb.generic.allowCCID = "TRUE"  ' (without the single quotes) to the Win10 VM vmx file and reboot the Win10 VM - the Yubikey is visible in the Win10 device mangler and the Yubikey Manager.  I can configure the OTP and FIDO2 stuff but get the same error when trying to configure the PIV stuff in the Yubikey Manager. 

I am running ESX 8 on a non-expiring temp/demo license.

I am at a loss.

On another note, I tried the to add an older HID Omnikey 3121 reader and Crescendo C1150 smart card to the Win10 VM via direct attachment of the reader to the ESX host machine (usb). The Win10 VM recognized the reader and card but I could never get anything to enumerate the card (certutil -scinfo returns the 'resource manager not running' message). 

What did work (but is not optimal for a variety of reasons) is direct connecting the Yubikey to my laptop, then allowing the Yubikey to passthrough my RDP session to the Win10 VM. The Win10 VM correctly sees the Yubikey and I can fully manipulate it. So I am thinking the issues lies with ESX 8.

Thoughts?

[michelkeus_stwg]

I have been seeing statements that using a shared Smart Card Reader requires a license.

The docs do thate that too on Docs / VMware vSphere / vSphere Virtual Machine Administration Add a Shared Smart Card Reader to Vir... :

A license is required for the shared smart card feature. See vCenter Server and Host Management.

What kind of exact license are you running? It may very well be that this feature is not enabled on the "vSphere Hypervisor"-license and requires a higher license. My machine is running with a vSphere Enterprise Plus license.  Not sure if you fully consumed your "trial" period else you may want to test with that to see if it is a license function that is simply missing. From what I gathered from an older version feature matrix is that it is not present in the "Hypervisor" (free) or "Essentials/Essentials Plus" licenses.

As a final thing I found this article: VMware KB: Unable to passthrough a USB smart card reader to a guest operating system in ESXi version... 

It references the "usb.generic.allowCCID" = "TRUE" that you added on your VM to get it working, but it also states that you need to stop the "pcscd" service to prevent the system from claiming it for itself. Not sure if this still applies to ESXi v8 though.

1. Run this command to stop the pcscd process:
   /etc/init.d/pcscd stop
2. Run this command to verify that the pcscd process is not running:
   ps | grep pcscd

---

That the smartcard is working through RDP is completely understandable and a completely viable option too. I am using mine like that for some time. And yes, the issue is definitely with ESXi in this case. It's just figuring out what the problem is in your case. 

[SchuFire]

Spoiler

So I have made *some* progress.

I have an older HID Omnikey 3121 USB smart card reader and Crescendo C1150 smart card. They work just fine. I see the reader in lsusb and in esxcli hardware usb passthrough device list.  I can also utilize them in my Win11 workstation without modification of the vmx file. 

The Yubikey is another story. I see it in lsusb but NOT in esxcli...

I am sure I have the settings right in /etc/vmware/config and /bootbank/boot.cfg 

I am using the vSphere 8 Enterprise Plus licensing so that is not the issue.

I am open to suggestions.