Different routes for VXLAN interfaces?

[bobbyccie]

Hi guys,

According to the NSX Design Guide v2.1:

The need for static routing is due to the fact that ESXi support only two TCP/IP stacks:

- VXLAN: this is dedicated to traffic sourced from the VMkernel VTEP interface. A dedicated default-route 0.0.0.0/0 can then be configured on this stack for each ESXi pointing to the gateway deployed on the local ToR, and this allows communication with all the remote VTEPs deployed in different Transport subnets.

- Default: this stack is used for all the other traffic types (vMotion, Management, Storage). It is typical to leverage a default route for management purposes (since connectivity to the vmk0 management interface could be originated from many remote IP subnets). This implies that static routing configuration is required to support inter-subnet communication for the other types of traffic.

Sounds good but is this special VXLAN stack added by default? Or do I need to create and configure it? I have configured a host with separate management and VXLAN VMkernel interfaces. I can see my VXLAN traffic getting pushed via the default gateway over the management network.

I am using static IP pools on both management and VXLAN interfaces. Does anyone know how to add this dedicated default route under the VXLAN stack?

Outputs from one of my compute hosts:

vmk0 = Management

vmk1 = VXLAN

~ # esxcfg-route -l

VMkernel Routes:

Network          Netmask          Gateway          Interface

172.16.100.0    255.255.255.128  Local Subnet     vmk0

default          0.0.0.0          172.16.100.1    vmk0

~ # esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type       

vmk0       0                   IPv4      172.16.100.100                         255.255.255.128 172.16.100.127 00:0c:29:ab:31:c9 1500    65535     true    STATIC     vmk0       0                   IPv6      fe80::20c:29ff:feab:31c9                64                              00:0c:29:ab:31:c9 1500    65535     true    STATIC, 

PREFERRED  

vmk1       2                   IPv4      172.16.200.100                         255.255.255.128 172.16.200.127 00:50:56:64:0e:f3 1600    65535     true    STATIC     vmk1       2                   IPv6      fe80::250:56ff:fe64:ef3                 64                              00:50:56:64:0e:f3 1600    65535     true    STATIC, 

PREFERRED  

Regards,

Bobby

[ddesmidt · Accepted solution]

The VTEP interface is created by NSX during the host preparation.

You can see that step in the NSX Getting Started Guide - "Step 3: Prepare ESXi hosts for NSX" (Getting Started Guide for NSX vSphere).

Once you have your VXLAN stack in ESXi, you can see its stack via vCenter UI (like usual).

    

Or if you love CLI:

. route:

root@Lab1_ESXi1:~#  esxcli network ip route ipv4 list -N vxlan

Network       Netmask        Gateway       Interface  Source

------------  -------------  ------------  ---------  ------

default       0.0.0.0        192.168.20.1  vmk1       MANUAL

192.168.20.0  255.255.255.0  0.0.0.0       vmk1       MANUAL

. ping

root@Lab1_ESXi1:~# ping ++netstack=vxlan 192.168.20.22

PING 192.168.20.22 (192.168.20.22): 56 data bytes

64 bytes from 192.168.20.22: icmp_seq=0 ttl=64 time=0.616 ms

Dimitri

[bobbyccie]

Thanks Dimitri,

So the VXLAN stack is enabled by default. That makes sense.

I can see the other gateway and have confirmed VXLAN is routing via vmk1.

~ # esxcli network ip route ipv4 list

Network        Netmask          Gateway        Interface  Source

-------------  ---------------  -------------  ---------  ------

default        0.0.0.0          172.16.100.1   vmk0       MANUAL

172.16.100.0   255.255.255.128  0.0.0.0        vmk0       MANUAL

~ # esxcli network ip route ipv4 list -N vxlan

Network        Netmask          Gateway        Interface  Source

-------------  ---------------  -------------  ---------  ------

default        0.0.0.0          172.16.200.1   vmk1       MANUAL

172.16.200.0   255.255.255.128  0.0.0.0        vmk1       MANUAL

Many thanks,

P.S. The NSX Getting Started Guide is great !