VXLAN appears missing in ESXi 6.0

budbeacham · ‎05-01-2015

I have been trying to automate the testing of our NIC cards with VXLAN, and it has been nothing but a nightmare. I am using the 'net-dvs' commands per VMware NIC certification guides, and the documentation is rife with errors. Anyway, I finally was able to configure two ESXi5.5 hosts and send encapsulated traffic between two virtual machines; one on each host. However, there is still a VXLAN issue. It is erratic in operation. By erratic I mean a test case would work, at the most, two times in a row, and then it would fail. In this case, failure is defined as bi-directional traffic does not pass between two virtual machines, one on each host. Run it again, and it would mayber pass or fail. I finally decided this is an ESXi 5.5 issue since it occurs with two different vendors NICs, and different drivers.

I upgraded my ESXi hosts to 6.0 hoping that would resolve the issue. Wrong.

After loading 6.0, my first thought was that VXLAN is now correctly working. I was able to run my test case 10 times, and it passed each time. However, after looking into it, I realized it is passing because the traffic is not encapsulated VXLAN traffic. It is regular traffic going between two VMs. Apparently, VXLAN is missing from ESXi 6.0

6.0 Issues:

[root@dut1088:~] vmware -v

VMware ESXi 6.0.0 build-2494585

[root@dut1088:~]

This is what I have found with ESXi 6.0.

First, the VXLAN module ‘vdl2’ is missing.

[11:29:00] (AddVxlan) COMMAND vmkload_mod vdl2

[11:29:06] (AddVxlan) RESPONSE: ['VMKMod_ComputeModPath(vdl2) failed: Not found', 'vmkload_mod: Can not load module vdl2: not found']

Is there another module that replaces ‘vdl2’? If so then what is its name? If ‘vdl2’ is still used then where do I find it for 6.0? It was included in the default ESXi5.5.

However, the other commands for configuring VXLAN appear to work, except the traffic is not encapsulated.

[12:10:37] (AddVxlan) COMMAND net-dvs -s "com.vmware.netoverlay.layer0=vxlan" -p globalPropList TEST_DVS

[12:10:42] (AddVxlan) RESPONSE: []

[12:10:42] (AddVxlan)

[12:10:43] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.vmknic=1" -p pgvmk_vxl_1 TEST_DVS

[12:10:48] (AddVxlan) RESPONSE: []

[12:10:48] (AddVxlan)

[12:10:48] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.id=16777216" -p VXLAN TEST_DVS

[12:10:54] (AddVxlan) RESPONSE: []

[12:10:54] (AddVxlan)

[12:10:54] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.mcastip=16777455" -p VXLAN TEST_DVS

[12:10:59] (AddVxlan) RESPONSE: []

Second, the ‘esxcli’ command for verifying VXLAN traffic statistics has been removed from ESXi 6.0. The ‘vxlan’ namespace appears to have been removed from 6.0. However, it might be possible the ‘vxlan’ namespace gets installed with the VXLAN 'vdl2' module.

[root@dut1088:~] esxcli network vswitch dvs vmware vxlan stats list --vds-name TEST_DVS

Error: Unknown command or namespace network vswitch dvs vmware vxlan stats list

So, my second question here is what command has been added to verify VXLAN statistics if the ‘vxlan’ namespace has been removed?

thakala2 · ‎05-02-2015

Have you deployed VXLAN properly with vCloud Networking and Security (vShield Manager) or NSX? Either of those is a requirement for VXLAN on ESXi.

I have had no issues whatsoever with VXLAN on ESXi 5.5 using vCloud Networking and Security 5.5.3. I have not used VXLAN with NSX on ESXi 6.0 much yet but it deploys OK, vxlan namespace is also available on ESXi 6.0 with NSX.

[root@esxi3:~] uname -a

VMkernel esxi3 6.0.0 #1 SMP Release build-2615704 Mar 25 2015 18:04:52 x86_64 x86_64 x86_64 ESXi

[root@esxi3:~] esxcli network vswitch dvs vmware vxlan stats list --vds-name=dvSwitch

Name Value

---------------------- -----

tx.passThrough 0

tx.vxlanTotal 0

tx.clone 0

tx.tso 0

tx.csum 0

tx.drop.invalidFrame 0

tx.drop.guestTag 0

tx.drop.noResource 0

tx.drop.invalidState 0

rx.passThrough 0

rx.vxlanTotal 0

rx.clone 0

rx.drop.invalidFrame 0

rx.drop.notExist 0

rx.drop.noResource 0

forward.pass 0

forward.reject 0

forward.rpf 0

arpProxy.reply.total 0

arpProxy.reply.fail 0

arpProxy.request.total 0

arpProxy.request.fail 0

mcastProxy.tx.total 0

mcastProxy.tx.fail 0

mcastProxy.rx.total 0

mcastProxy.rx.fail 0

[root@esxi3:~] esxcli network vswitch dvs vmware vxlan

Usage: esxcli network vswitch dvs vmware vxlan {cmd} [cmd options]

Available Namespaces:

config A set of commands for VXLAN config related operations

network A set of commands for VXLAN network related operations

vmknic A set of commands for VXLAN vmknic related operations

stats A set of commands for VXLAN switch statistics related operations

Available Commands:

get Get VXLAN global states on the system.

list Get VXLAN switches info on the system.

[root@esxi3:~]

aqwelea · ‎12-15-2015

Hello,

I am currently starting with NSX with vSphere 6.0 update 1 and NSX 6.2.

VMkernel esx3 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 x86_64 x86_64 ESXi

I had a similar issue, the vxlan namespace was unknown although vsip and vxlan vib were correctly installed.

A simple reboot helped me :smileyconfused: (as though vib were not really running...)

I tried to reproduce the issue (to check if vxlan process was really running immediatly after install) by installing a second ESXi in a new cluster with a new VDS, preparing it and configuring vxlan => Everything was ok withtout any reboot. I've been able to get the vxlan namespace....

Did you finally find out what was going wrong or did you simply reinstall the proper vibs (or the host) ?

evil242 · ‎04-28-2016

I have the same issue, reboot, and still nothing:

vmware -v

VMware ESXi 6.0.0 build-3620759

esxcli software vib list |grep esx-v

esx-vsip 6.0.0-0.0.3521449 VMware VMwareCertified 2016-04-28

esx-vxlan 6.0.0-0.0.3521449 VMware VMwareCertified 2016-04-28

ps |grep vxlan

35739 0 tq:vxlan

35741 0 vxlan_CPWorld

esxcli network vswitch dvs vmware vxlan

Error: Unknown command or namespace network vswitch dvs vmware vxlan

esxcli network vswitch dvs vmware

Usage: esxcli network vswitch dvs vmware {cmd} [cmd options]

Available Namespaces:

lacp A set of commands for LACP related operations

Available Commands:

list List the VMware vSphere Distributed Switch currently configured on the ESXi host.

vdl2 appears to be loaded

2016-04-28T08:31:41.755Z cpu6:35738)Loading module vdl2 ...

2016-04-28T08:31:41.756Z cpu6:35738)Elf: 1865: module vdl2 has license VMware

2016-04-28T08:31:41.769Z cpu6:35738)vxlan: VDL2Init:573: Using netstack 'vxlan'

2016-04-28T08:31:41.769Z cpu6:35738)vxlan: VDL2_UDPPortChange:166: VXLAN UDP port(s) set to [8472, 0]

2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.instancelist successful

2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.conncheck.receive successful

2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.debug successful

2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.debug.string successful

2016-04-28T08:31:41.770Z cpu6:35738)vxlan: VDL2_CPActivate:4926: VDL2 control plane is activated

2016-04-28T08:31:41.770Z cpu6:35738)vxlan: VDL2Init:683: VDL2 initialization succeeded. Module ID:120

2016-04-28T08:31:41.770Z cpu6:35738)Mod: 4943: Initialization of vdl2 succeeded with module ID 120.

2016-04-28T08:31:41.770Z cpu6:35738)vdl2 loaded successfully.

2016-04-28T08:31:41.770Z cpu5:35741)vxlan: VDL2CPWorldFunc:315: Control plane world starts

net-vdl2 -l

VXLAN Global States:

Control plane Out-Of-Sync: No

UDP port: 8472

VXLAN VDS: vDSwitchNSX_Lab

VDS ID: a7 4c 0b 50 9f c4 1e 8d-28 0c 1c 97 a3 d5 5c b3

MTU: 1600

Segment ID: 192.168.8.0

Gateway IP: 192.168.8.1

Gateway MAC: 38:22:d6:61:48:0f

Vmknic count: 1

VXLAN vmknic: vmk2

VDS port ID: 37

Switch port ID: 50331656

Endpoint ID: 0

VLAN ID: 0

IP: 192.168.8.211

Netmask: 255.255.255.0

Segment ID: 192.168.8.0

IP acquire timeout: 0

Multicast group count: 0

Network count: 3

VXLAN network: 5001

Multicast IP: N/A (headend replication)

Control plane: Enabled (multicast proxy,ARP proxy)

Controller: 192.168.8.196 (up)

MAC entry count: 0

ARP entry count: 0

Port count: 2

VXLAN network: 5000

Multicast IP: N/A (headend replication)

Control plane: Enabled (multicast proxy,ARP proxy)

Controller: 192.168.8.195 (up)

MAC entry count: 0

ARP entry count: 0

Port count: 2

VXLAN network: 5002

Multicast IP: N/A (headend replication)

Control plane: Enabled (multicast proxy,ARP proxy)

Controller: 192.168.8.195 (up)

MAC entry count: 0

ARP entry count: 0

Port count: 1

Damion Terrell  .   +  (He/Him)  +  . *  .  +   @   + .    *  .    +      .                    
Core IT Service Specialist  *     .    +    *  .   +   .     +   .    +     *   +             
UNM – IT Platforms – VIS  + .  . .     .    .  .    .     .             .        
          .       +    .      +       *        .          +  *   .
      *     .  .      +      .        .   .   .      +   .    +     *   +        .
“You learn the job of the person above you,     *     +   .    +     *    @       
and you teach your job to the person below you..”              .    *         +

crooky1106 · ‎05-03-2016

Hey evil242,

Looks like a bug... can you try the same command replacing the "esxcli" part with "localcli"? This worked for me.

Correcting the behaviour is to reboot the host or restart the 'hostd' process by logging in to the host with root over SSH and running the command:

/etc/init.d/hostd restart

Note: This may cause short disruption to the host from vCenter on management level, but it will not affect the VM data traffic

Cheers,

James

payeljayesh · ‎06-10-2016

Hi,

please restart the services.sh and vxlan name space will appear.

Thanks,

Jayesh

All

VXLAN appears missing in ESXi 6.0