Nested ESXi 6u2 Host on VMware Workstation v12 for Linux Network Problems

[PBensonNet]

I Hello everyone,

I have a virtual lab environment running in VMware Workstation v12 for Linux, and my ESXi hosts are not working properly on one of the virtual networks. All of my troubleshooting suggests an issue with the nested ESXi hosts that may be a bug of some sort, but I want to make sure that I have done everything correctly first. That is why I'm posting here with the hope that if I did make a mistake that perhaps someone else can point it out to me.

Physical System

8 core Intel Xeon 2.1GHz CPU

128 GB of RAM

OS - Linux Mint 17.3 64-bit w/3.19.0-32-generic Linux kernel (fully updated as of this posting)

VMware Workstation 12 Pro - 12.1.1. build-3770994 (fully updated as of this posting)

Nested ESXi Host VM

ESXi v6.0.0 (Build 3825889, fully update as of this posting)

4 "physical" NICs (only 3 being used for now)

    - all use e1000 NIC virtual hardware, but also tried vmxnet3 NICs with no difference

    - vSwitch0 uses vmnic0 & 1 on virtual network vmnet16 in an active/standby pair

    - vSwitch1 uses vmnic2 on virtual network vmnet18

    - vmk0 used for management on vSwitch0, and vmk1 to be used for iSCSI on vSwitch1

    - promiscuous mode and forged transmits enabled on all vSwitch Port Groups (enabling or disabling these features makes no difference)

Testing Done So Far

I have verified that all IP adresses and netmasks being used are correct.

Using vmkping I have pinged other nodes on the vmnet16 network successfully.

Using vmkping I have attempted to ping other nodes on the vmnet18 network, but that has been unsuccessful.

I have depolyed other non-ESXi VMs onto the vmnet18 network, and they are able to ping each other but are unable to ping or be pinged by the ESXI host.

I have tried various virtual hardware NICs as mentioned before, but with no changes in results.

I have tried using LAN segments instead of the host-only network vmnet18 with no changes in results.

When I view the state of the ESXi host's NICs via vCenter or the embedded host client vmnic0 & 1 both show network information, but vmnic2 shows no networks. Yet I know that there is a network with various VMs communicating on it. Furthermore I was able to get all of this working on a Windows system running Workstation 10 (this is the laptop that my employer provides me with).

Having built working nested ESXi labs on various platforms as well as physical environments in the past I'm very confused as to why I can't get this particular setup to work. At this point my gut tells me that it is probably a bug of some sort with the nested ESXi hosts themselves. Since I can get everything working on vmnet16 including the ESXi host management and the VCSA that I am using I am certain that my vSwitch configuration is correct (other than IP space and vmnics the configurations are bascially identical). Since I can get other VMs to communicate over the vmnet18 network I do not see how this can be a VMware Workstation of Linux physical host issue. Is there something obvious that I am missing here? I have read about nested ESXi hosts running on VMware Workstation having known issues and bugs with networking. Has anyone else encountered this?

Thanks for any help that others can provide!

Regards,

Patrick

[covelli]

I have forwarded your question to our networking team,

[PBensonNet]

Thank you covell!

This morning I tried the following:

1) Turned on DHCP for the vmnet18 network.

2) Changed the vmk1 VMkernel adapter on the nested host esxi-0a to use DHCP instead of a static config.

3) Changed a Linux VMs NICs to use DHCP instead of a static config.

The results were that the ESXi VM did not obtain an IP address, but the Linux VM did.

I read about problems with multiple NICs on ESXI hosts running on Workstation for WIndows. I then tested my active/stand-by NIC config for vSwitch0 which is the management network:

1) In Workstation I disconnected vmnic0, and connected vmnic1. Both were on vmnet16. Pings to the static IP address failed.

2) I then reversed the setup. So vmnic0 was connected, and vmnic1 was disconnected. Pings to the static IP address succeeded.

So it seems that only the first NIC actually works on my nested ESXi hosts. I'm going to some more changes, and if I come across a working config I'll post it here.

[PBensonNet]

I tried editing the ESXi nested VM's vmx file to change the connection type from "pvn" to "custom" based on a suggestion from another forum. Still no change.

Then I tried putting the management NIC vmnic0 on the vmnet18 network, and then with a LAN segment. Pings work just fine in both cases.

At this point I cannot think of anything else that I can change and test. From all of this testing I am guessing that this is a bug with ESXi being virtualized when using VMware Workstation Pro v12 for Linux:

• Other non-ESXi VMs with mulitiple NICs on various virtual networks and LAN segments work as expected.
• The same configurations used for ESXi VMs running on VMware Workstation v10 for Windows works as expected.
• As long as I use vmnic0 on the ESXI VM it does not matter which virtual network or LAN segment is used.

If I have time I might rebuild the system with Windows 10 Pro and install Workstation Pro v12 and import the same VMs to see if that makes any difference. Anyone else have any other suggestions?

[covelli]

> - promiscuous mode and forged transmits enabled on all vSwitch Port Groups (enabling or disabling these features makes no difference)

This actually has to be done on the physical NIC - in your case you will have to configure Linux to use promiscuous mode.  Have you tried this?

[PBensonNet]

covelli - I'm not quite sure which physical NIC I would need to configure promiscuous mode on. The virtual network that is experiencing the problem is a host-only network isolated to the host itself. I have tried it with and without a host virtual adapter, but since the vmnet18 virtual network is only being used for VM to VM traffic that never leaves the host.

Now vmnet16 is a NAT virtual network that the VMs use for Internet access, and that works fine without any additional changes to the physical networking. Plus if I put vmnic0 on any virtual network they all work for any of the ESXi hosts.

The issue seems to be that any NIC on a nested ESXi host other than vmnic0 fails to work.

I will figure out how to configure all of the physical NICs on the host as well as the wireless to use promiscuous mode, and let you know the results. Yet I don't understand how that will resolve the issue for VMs being unable to communicate with other VMs on an isolated host-only virtual network. Would you please explain why that is needed when I am running Workstation Pro? I want to make sure that I am not misunderstanding how host-only networks operate. Thanks!

[covelli]

Sorry.  What I meant to ask is :

Is promiscuous mode allowed in Linux host for the vnic(vmnic2)?

Is forged mac allowed in linux host for the vnic(vmnic2)?

[PBensonNet]

covelli

Yes, all of the standard vSwitches are set to accept promiscuous mode and forged transmits. I just verified this for vSwitch1 which has vmnic2 as the active adapter. The vSwitches are also configured to accept MAC address changes.

[PBensonNet]

Just to rule out a Linux permissions issue I also tried launching VMware as the root user. There were no changes.

At least it is one more thing to mark off of the list of possible causes.

[PBensonNet]

This problem is still unresolved. Here are some updates to what I have done since my last post:

• Migrated VMs off of my Linux Mint 17.3 PC running VMware Workstation Pro v12 to a WIndows 7 laptop also running VMware Workstation Pro v12. The issue does not occur when running the software on Windows.
• Upgraded Linux Mint from 17.3 to 18. The issue still occurs.
• Completely rebuilt the VM environment on Linux Mint 18. Used the latest iso files for ESXi 6 Update 2 and the VCSA. The issue still occurs.

Again, this issue only appears with nested ESXi 6 VMs. The first vnic seems to work just fine, but any additional vnics do not appear to work. VMs that are not ESXi do not have this issue appear, and ESXi VMs on a Windows host do not have the issue appear.

I'm at a loss as to what to do next in order to resolve this issue. Any help would be greatly appreciated at this point. I want to avoid using Windows, so that I can keep my costs down by nto having to purchase expensive third party software for what I am working on. Has anyone had the same issue occur with their nested ESXi VMS?

[PBensonNet · Accepted solution]

Okay, the issue is now resolved. The problem was related to what covelli was trying to explain, but that I did not understand because physical host NICs were being mentioned.

The issue was on the Linux host, but it was a permissions issue with the virtual Ethernet adapters. The following article had the fix:
Using Virtual Ethernet Adapters in Promiscuous Mode on a Linux Host (287) | VMware KB

Even though I launched Workstation as the root user I still encountered this issue, and that still does not make sense to me, but changing the permissions does resolve the issue. I was not getting an error message on my Linux Mint and Ubuntu systems, but when I tried on a second Linux Mint box that I just happened to have the error was generated, and the error message had the link above within it.

So here is what I learned:

1) The issue has nothing to do with a VMware Workstation setting that you can configure.

2) The issue has nothing to do with a physical NIC.
3) The issue has nothing to do with any vmnics on the Nested ESXi host.

4) Virtual Ethernet Adapters apparently are not given the correct permissions when created.

All you have to do to correct this problem is:

1) Open a terminal.

2) Run the command "sudo chmod a+rw /dev/vmnet*" (be sure to run this every time you create a new virtual network).

You can also create a group as mentioned in the link above, and just give that group the correct permissions. Personally I think it is easier to just give everyone rw permission.

I appreciate the help that others have offered. I did not understand what was being asked for in the previous responses because of the terms that were used.

[iemem15]

Worked for me thanks