vmxnet3 casuing issues for Nested ESXi 6.7/7.01 running inside QEMU/KVM

[tsi_turbo636]

Hi Nested Community,

I have been using nested ESXi 6.7 under QEMU/KVM on a Ubuntu 18 host for a while now, and it has been working perfectly when the QEMU "e1000" network device is selected. I use the default NAT network that libvirt sets up where is it adds the guests (my ESXi hosts) to a virbr0 device automatically. Nested ESXi hosts are able to talk to each other and reach the outside world.

However, now that e1000 has been removed starting with ESXi 7, I tried using vmxnet3, which comes with Ubuntu 18 already compiled and available as a QEMU network device. The nested ESXi hosts boot just fine with this, and I can login to the ESXi web UI and SSH to them from my Ubuntu 18 host. The weird thing is they fail/timeout at certain tasks, for example:

• Adding a host to vCenter hangs at 80% and never completes with the error of "A general system error occurred: Unable to push signed certificate to host". I am able to add the host that the VCSA runs on, but no other hosts.
• When I go to download a OVA file when deploying a new OVA via vCenter GUI, after I put the URL in it asks me to verify the SSL thumbprint, but then hangs and fails with the error of "Unable to retrieve manifest or certificate file."

On ESXi 6.7, I simply stop the ESXi hosts, switch back to e1000, then everything works as expected. The problem is e1000 is not supported in ESXi 7, so I am out of luck running nested virt with this version.

Has anyone else came across this issue before?

I tried coming up with a few workarounds, but QEMU is limited in what network cards it can emulate. Please let me know if you have any ideas!

Thanks!

[tsi_turbo636]

While this is not a fix to my original question, it is worth noting a solid workaround is to use VMware Workstation Pro instead of KVM, on the base Ubuntu 18 host. The vmxnet3 driver is used for ESXi when you launch it in Workstation Pro, and it works quite nicely at 10 Gbps virtualized speed between the nested ESXi hosts. When you install ESXi in Workstation Pro, vmxnet3 is used by default, so no customization of the physical adapter is needed.

Still leaving my original question as unanswered in case anyone else has more insight, but thought I'd post to help others out there with a workaround. Of course, you will need to obtain a proper license for Workstation Pro, but this solution works good for the time being.

[iriartec]

To add more info, successfully installed ESXi 7.0.1 on top of Proxmox 6.3-3. VMXNET3 is the only provided NIC that is supported out of the box by ESXi, the only issue (deal breaker actually) is that only payloads of 3030 bytes seem to be supported.

Not sure if it's a guest driver issue (ESXi) or the host implementation (KVM/QEMU). VSAN cannot be installed in this scenario, vSphere can still run VMs though.

root@bigiron:~# kvm -version
QEMU emulator version 5.1.0 (pve-qemu-kvm_5.1.0)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers
root@bigiron:~#
root@bigiron:~#
root@bigiron:~# uname -a
Linux bigiron 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 GNU/Linux

[acancro]

+1 to this.  I'm trying to build ESXi images for MaaS, which requires a Linux build host.