I have been trying to automate the testing of our NIC cards with VXLAN, and it has been nothing but a nightmare. I am using the 'net-dvs' commands per VMware NIC certification guides, and the documentation is rife with errors. Anyway, I finally was able to configure two ESXi5.5 hosts and send encapsulated traffic between two virtual machines; one on each host. However, there is still a VXLAN issue. It is erratic in operation. By erratic I mean a test case would work, at the most, two times in a row, and then it would fail. In this case, failure is defined as bi-directional traffic does not pass between two virtual machines, one on each host. Run it again, and it would mayber pass or fail. I finally decided this is an ESXi 5.5 issue since it occurs with two different vendors NICs, and different drivers.
I upgraded my ESXi hosts to 6.0 hoping that would resolve the issue. Wrong.
After loading 6.0, my first thought was that VXLAN is now correctly working. I was able to run my test case 10 times, and it passed each time. However, after looking into it, I realized it is passing because the traffic is not encapsulated VXLAN traffic. It is regular traffic going between two VMs. Apparently, VXLAN is missing from ESXi 6.0
6.0 Issues:
[root@dut1088:~] vmware -v
VMware ESXi 6.0.0 build-2494585
[root@dut1088:~]
This is what I have found with ESXi 6.0.
First, the VXLAN module ‘vdl2’ is missing.
[11:29:00] (AddVxlan) COMMAND vmkload_mod vdl2
[11:29:06] (AddVxlan) RESPONSE: ['VMKMod_ComputeModPath(vdl2) failed: Not found', 'vmkload_mod: Can not load module vdl2: not found']
Is there another module that replaces ‘vdl2’? If so then what is its name? If ‘vdl2’ is still used then where do I find it for 6.0? It was included in the default ESXi5.5.
However, the other commands for configuring VXLAN appear to work, except the traffic is not encapsulated.
[12:10:37] (AddVxlan) COMMAND net-dvs -s "com.vmware.netoverlay.layer0=vxlan" -p globalPropList TEST_DVS
[12:10:42] (AddVxlan) RESPONSE: []
[12:10:42] (AddVxlan)
[12:10:43] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.vmknic=1" -p pgvmk_vxl_1 TEST_DVS
[12:10:48] (AddVxlan) RESPONSE: []
[12:10:48] (AddVxlan)
[12:10:48] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.id=16777216" -p VXLAN TEST_DVS
[12:10:54] (AddVxlan) RESPONSE: []
[12:10:54] (AddVxlan)
[12:10:54] (AddVxlan) COMMAND net-dvs -s "com.vmware.net.vxlan.mcastip=16777455" -p VXLAN TEST_DVS
[12:10:59] (AddVxlan) RESPONSE: []
Second, the ‘esxcli’ command for verifying VXLAN traffic statistics has been removed from ESXi 6.0. The ‘vxlan’ namespace appears to have been removed from 6.0. However, it might be possible the ‘vxlan’ namespace gets installed with the VXLAN 'vdl2' module.
[root@dut1088:~] esxcli network vswitch dvs vmware vxlan stats list --vds-name TEST_DVS
Error: Unknown command or namespace network vswitch dvs vmware vxlan stats list
So, my second question here is what command has been added to verify VXLAN statistics if the ‘vxlan’ namespace has been removed?
Have you deployed VXLAN properly with vCloud Networking and Security (vShield Manager) or NSX? Either of those is a requirement for VXLAN on ESXi.
I have had no issues whatsoever with VXLAN on ESXi 5.5 using vCloud Networking and Security 5.5.3. I have not used VXLAN with NSX on ESXi 6.0 much yet but it deploys OK, vxlan namespace is also available on ESXi 6.0 with NSX.
[root@esxi3:~] uname -a
VMkernel esxi3 6.0.0 #1 SMP Release build-2615704 Mar 25 2015 18:04:52 x86_64 x86_64 x86_64 ESXi
[root@esxi3:~] esxcli network vswitch dvs vmware vxlan stats list --vds-name=dvSwitch
Name Value
---------------------- -----
tx.passThrough 0
tx.vxlanTotal 0
tx.clone 0
tx.tso 0
tx.csum 0
tx.drop.invalidFrame 0
tx.drop.guestTag 0
tx.drop.noResource 0
tx.drop.invalidState 0
rx.passThrough 0
rx.vxlanTotal 0
rx.clone 0
rx.drop.invalidFrame 0
rx.drop.notExist 0
rx.drop.noResource 0
forward.pass 0
forward.reject 0
forward.rpf 0
arpProxy.reply.total 0
arpProxy.reply.fail 0
arpProxy.request.total 0
arpProxy.request.fail 0
mcastProxy.tx.total 0
mcastProxy.tx.fail 0
mcastProxy.rx.total 0
mcastProxy.rx.fail 0
[root@esxi3:~] esxcli network vswitch dvs vmware vxlan
Usage: esxcli network vswitch dvs vmware vxlan {cmd} [cmd options]
Available Namespaces:
config A set of commands for VXLAN config related operations
network A set of commands for VXLAN network related operations
vmknic A set of commands for VXLAN vmknic related operations
stats A set of commands for VXLAN switch statistics related operations
Available Commands:
get Get VXLAN global states on the system.
list Get VXLAN switches info on the system.
[root@esxi3:~]
Hello,
I am currently starting with NSX with vSphere 6.0 update 1 and NSX 6.2.
VMkernel esx3 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 x86_64 x86_64 ESXi
I had a similar issue, the vxlan namespace was unknown although vsip and vxlan vib were correctly installed.
A simple reboot helped me :smileyconfused: (as though vib were not really running...)
I tried to reproduce the issue (to check if vxlan process was really running immediatly after install) by installing a second ESXi in a new cluster with a new VDS, preparing it and configuring vxlan => Everything was ok withtout any reboot. I've been able to get the vxlan namespace....
Did you finally find out what was going wrong or did you simply reinstall the proper vibs (or the host) ?
I have the same issue, reboot, and still nothing:
vmware -v
VMware ESXi 6.0.0 build-3620759
esxcli software vib list |grep esx-v
esx-vsip 6.0.0-0.0.3521449 VMware VMwareCertified 2016-04-28
esx-vxlan 6.0.0-0.0.3521449 VMware VMwareCertified 2016-04-28
ps |grep vxlan
35739 0 tq:vxlan
35741 0 vxlan_CPWorld
esxcli network vswitch dvs vmware vxlan
Error: Unknown command or namespace network vswitch dvs vmware vxlan
esxcli network vswitch dvs vmware
Usage: esxcli network vswitch dvs vmware {cmd} [cmd options]
Available Namespaces:
lacp A set of commands for LACP related operations
Available Commands:
list List the VMware vSphere Distributed Switch currently configured on the ESXi host.
vdl2 appears to be loaded
2016-04-28T08:31:41.755Z cpu6:35738)Loading module vdl2 ...
2016-04-28T08:31:41.756Z cpu6:35738)Elf: 1865: module vdl2 has license VMware
2016-04-28T08:31:41.769Z cpu6:35738)vxlan: VDL2Init:573: Using netstack 'vxlan'
2016-04-28T08:31:41.769Z cpu6:35738)vxlan: VDL2_UDPPortChange:166: VXLAN UDP port(s) set to [8472, 0]
2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.instancelist successful
2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.conncheck.receive successful
2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.debug successful
2016-04-28T08:31:41.769Z cpu6:35738)NetDVS: 1498: register client for command com.vmware.net.vxlan.debug.string successful
2016-04-28T08:31:41.770Z cpu6:35738)vxlan: VDL2_CPActivate:4926: VDL2 control plane is activated
2016-04-28T08:31:41.770Z cpu6:35738)vxlan: VDL2Init:683: VDL2 initialization succeeded. Module ID:120
2016-04-28T08:31:41.770Z cpu6:35738)Mod: 4943: Initialization of vdl2 succeeded with module ID 120.
2016-04-28T08:31:41.770Z cpu6:35738)vdl2 loaded successfully.
2016-04-28T08:31:41.770Z cpu5:35741)vxlan: VDL2CPWorldFunc:315: Control plane world starts
net-vdl2 -l
VXLAN Global States:
Control plane Out-Of-Sync: No
UDP port: 8472
VXLAN VDS: vDSwitchNSX_Lab
VDS ID: a7 4c 0b 50 9f c4 1e 8d-28 0c 1c 97 a3 d5 5c b3
MTU: 1600
Segment ID: 192.168.8.0
Gateway IP: 192.168.8.1
Gateway MAC: 38:22:d6:61:48:0f
Vmknic count: 1
VXLAN vmknic: vmk2
VDS port ID: 37
Switch port ID: 50331656
Endpoint ID: 0
VLAN ID: 0
IP: 192.168.8.211
Netmask: 255.255.255.0
Segment ID: 192.168.8.0
IP acquire timeout: 0
Multicast group count: 0
Network count: 3
VXLAN network: 5001
Multicast IP: N/A (headend replication)
Control plane: Enabled (multicast proxy,ARP proxy)
Controller: 192.168.8.196 (up)
MAC entry count: 0
ARP entry count: 0
Port count: 2
VXLAN network: 5000
Multicast IP: N/A (headend replication)
Control plane: Enabled (multicast proxy,ARP proxy)
Controller: 192.168.8.195 (up)
MAC entry count: 0
ARP entry count: 0
Port count: 2
VXLAN network: 5002
Multicast IP: N/A (headend replication)
Control plane: Enabled (multicast proxy,ARP proxy)
Controller: 192.168.8.195 (up)
MAC entry count: 0
ARP entry count: 0
Port count: 1
Damion Terrell . + (He/Him) + . * . + @ + . * . + .
Core IT Service Specialist * . + * . + . + . + * +
UNM – IT Platforms – VIS + . . . . . . . . .
. + . + * . + * .
* . . + . . . . + . + * + .
“You learn the job of the person above you, * + . + * @
and you teach your job to the person below you..” . * +
Hey evil242,
Looks like a bug... can you try the same command replacing the "esxcli" part with "localcli"? This worked for me.
Correcting the behaviour is to reboot the host or restart the 'hostd' process by logging in to the host with root over SSH and running the command:
/etc/init.d/hostd restart
Note: This may cause short disruption to the host from vCenter on management level, but it will not affect the VM data traffic
Cheers,
James
Hi,
please restart the services.sh and vxlan name space will appear.
Thanks,
Jayesh