VMware Cloud Community
iehrenwaldHBG
Contributor
Contributor

Nested ESXi 6.5, Auto Deploy, and VMXNET3

I am playing with a test lab and have run into a weird problem and was hoping someone could point me in the right direction.

I have a physical ESXi 6.5U1 host that I am deploying nested ESXi 6.5U1 VMs into.  These nested VMs have 4 x VMXNET3 interfaces.  If I boot the nested VMs from the ESXi installation ISO, the installer is happy and I can proceed with a manual installation.  If I PXE boot the nested VMs to pass off to Auto Deploy (which is running on VCSA on the physical ESXi 6.5U1 host), the installer complains that there are no compatible NICs installed in the system.  If I change the NIC type in the nested VMs to E1000, the Auto Deploy proceeds without fatal errors about missing NICs.

So, my question is, why isn't the VMXNET3 driver getting used when I boot/install from network?  I actually see it getting brought down from the Auto Deploy server when watching the ESXi bootup screen.  I've tried various builds of the 6.5 offline bundle in my Auto Deploy Software Depot, from GA to U1, with the same results.  It must be something I am inadvertently doing.

Has anyone else run into this or have ideas?

Thanks.

Reply
0 Kudos
21 Replies
daphnissov
Immortal
Immortal

Gotta understand that the VMXNET3 adapter is a paravirtualized device. Because of this, it doesn't actually "exist" as a valid device without the assistance of VMware Tools (which now comes embedded in ESXi 6.5). The e1000 series are all virtualized including their boot ROMs, which is why that succeeds because it's independently represented by your host.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

So a PXE/AutoDeploy system does not pull down or start the VMware Tools, but installing from an ISO does?  I don't understand the purpose of that distinction, I think I am misunderstanding you.  As I said, booting a nested hypervisor from the ISO with vmxnet3 type adapters works perfectly fine.

I do see that there is both a 'no-tools' and 'standard' Image Profile in the various Offline Bundles I've been uploading to my Software Depot.  I have tried both with no change in observed behavior.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Yes, when you boot ESXi from an image, part of the boot process is to load ESXi into memory, and when you're running a nested instance that loads the embedded tools as well, which provides VMXNET3 services.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

If it is true that PXE boot/Auto Deploy does not work with nested hypervisors with VMXNET3 NICs due to VMware Tools not being run, how did Testing ESXi Auto Deploy in a nested environment ​work ?

Reply
0 Kudos
daphnissov
Immortal
Immortal

No, duh, you're right. I somehow had a touched moment there and got confused with earlier versions. Yeah, what you have with VMXNET3 should work. What image bundle are you using? What hardware level did you assign to your nested ESXi?

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

No problem about the 'duh' moment.. those are most of my moments Smiley Happy

I uploaded to my VCSA both 'ESXi650-201704001.zip' (6.5.0d) and 'update-from-esxi6.5-6.5_update01.zip' (6.5U1).  The former contains 'ESXi-6.5.0-20170404001-no-tools' and 'ESXi-6.5.0-20170404001-standard'.  The latter contains 'ESXi-6.5.0-20170701001s-no-tools', 'ESXi-6.5.0-20170701001s-standard', 'ESXi-6.5.0-20170702001-no-tools', and 'ESXi-6.5.0-20170702001-standard'.  Both of those Offline Bundles and the Image Profiles contained within them have the vmxnet3 and nvmxnet3 Software Packages.

The Image Profile currently specified in my Deploy Rule is 'ESXi-6.5.0-20170702001-standard'.  The hardware level for all VMs created in this lab is 13.

The installation ISO I am using for comparison is 'VMware-VMvisor-Installer-6.5.0.update01-5969303.x86_64.iso' seems to correspond with the same build number as the '20170702001-standard' Image Profile.

Reply
0 Kudos
TronAr
Commander
Commander

Only thing that comes to mind is that somehow your VM is marginally small (e.g. in memory) and loading the vmxnet3 driver fails.

Or that it does not have VT-x emulation enabled, and some "intelligent" choice disables vmxnet.

Reply
0 Kudos
TronAr
Commander
Commander

Nope, just tested without VT-x and works. Also bare 4GB memory nested vm loads fine, using build 4553939.

Maybe some issue with your selected profile ? If your profile is based on a physical host with e1000s, that may be the issue.

Try installing one "by hand" and using that as a profile seed ?

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

Went down that path too, unfortunately.  All of the nested VMs have 16GB.  Just for the heck of it I increased one of the auto-deploy installed VMs to 24GB but that didn't make a difference.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

Host profile is based on one of the manually installed nested ESXi hosts which use VMXNET3 no problem. 

I'm tempted to just tear everything down and start over, it's just a lab, but am achingly curious as to why this isn't working.

Reply
0 Kudos
TronAr
Commander
Commander

Interesting... for me it just worked.

Using a profile that was provided, I'm attacjing it just in case. If this works, then we have something to research Smiley Happy

Reply
0 Kudos
TronAr
Commander
Commander

Well, to be fair, I had an issue with ipxe not being able to get entropy. That's a whole other story. But no issues with the network card.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

I tried your host profile and maddeningly got the same results.  ESXi PXE boots, loads everything from the auto deploy cache, host profile settings are applied, and then "No compatible network adapter found" at the yellow/gray F2/F12 screen.

I also tried with no host profile selected in the auto deploy rule, same end result.  Is there a "banging head against desk" emoticon? Smiley Happy

Reply
0 Kudos
TronAr
Commander
Commander

I'm out of ideas Smiley Sad

Reply
0 Kudos
daphnissov
Immortal
Immortal

Just for fun, have you tried William Lam's pre-built nested ESXi appliance (as OVA)? I'm just curious to know what would happen there.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

I really appreciate your troubleshooting.  I think I'm going to tear it all down and start over, see if some magic bit gets flipped somewhere.  Will update the thread either way.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

Funny, found that a few minutes ago via web search.  I'll give it a try right now before throwing my hands up.

Reply
0 Kudos
iehrenwaldHBG
Contributor
Contributor

Mr Lam's OVA works out of the box pre-installed as expected.  I added more NICs to it (4 x VMXNET3) and it boots from the virtual hard drive.

If I change the boot order in the VMs BIOS to use network adapters before local hard drives, the VMXNET3 network boot gets an IP address/netmask/gateway and then the system sits there doing nothing.

If I change the firmware type from BIOS to EFI and boot from the first network adapter, ESXi boots successfully from the network, then finally 'No compatible network adapter found'.

At this point I wonder if the underlying hardware has anything at all to do with it.  This is on a TinkerTry/SYS-5028D-TN4T with 64GB of RAM, 2 x 1Gb via igbn driver, 2 x 10Gb via ixgbe driver.  But that doesn't explain why booting a VM from the ISO makes the installer happy.

Reply
0 Kudos
TronAr
Commander
Commander

Sorry about that.

I have successfully demoed (well, actually implemented course labs) with nested^2 VMs and auto-deploy.

I've used BIOS always though, and the no-tools image profile.

Not much more I can add.

Reply
0 Kudos