VXLAN Unicast over L3

[JFSynovatec]

Hello!

I'm trying different scenarios in a lab and I have some difficulties with VXLAN bridging in Unicast mode.

I have 2 clusters with one Host in each.

I'd like each VMK to be in a different subnet and routed together with VyOS VM (With interfaces at 1600 MTU)

I have added all the routes needed and each VMK can reach each other successfully.

I have created all needed infra (Logical switch etc,)

I was wondering if it's possible to "route' the VXLAN VMK to another one on a different subnet or do you absolutely need multicasting from the moment it has to through routing?

Thanks!

[lhoffer]

Yes, you can have you your VMKs on separate subnets and if your control plane mode is already unicast, then multicast isn't required at all as described on page 38 of the VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0 .

[JFSynovatec]

I feel there's something missing.

I've checked the controller and can see that all MACs are registered under the good VTEP, ESXi Hosts, etc.

But, when VTEPs are on different VLANs (subnet) they can't reach each other.

On ESXi hosts, the routing table is not set si that the vxlan stack will use a router IP to reach other VTEPs, is that whatMs missing?

My router can effectively ping both VTEPs. But ESXi hosts cannot ping the other VTEP.

Any ideas?

[lhoffer]

If you're using IP pools to assign the VXLAN VMK ports, the VXLAN stack will use the default gateway you configured in the IP pool to reach the other cluster's subnet.  If you're using DHCP, it'll use the default gateway assigned there.

Either way, I'd first try running "esxcli network ip route ipv4 list -N vxlan" on the affected hosts to make sure that the default gateway for the VXLAN stack is what you think it should be as well as using the "esxcli network ip neighbor list -N vxlan" to make sure that it has the MAC of your default gateway box in its ARP table.  You can also verify that you can ping the default gateway from the VXLAN VMK port using "ping ++netstack=vxlan -I vmk1 x.x.x.x".

[JFSynovatec]

Found out that I can effectively ping the other gateway but not the other VTEP vmknic IP.

Frewall issue?

[jamib]

Multicast is not a requirement for VTEP reachability when on different subnets. Just IP reachability and MTU is. Also, VYOS does not support multicast routing so you definitely need Unicast control plane if using VYOS as your gateway between VLAN backed dvPG's and VMK's.

Can you validate the MTU for VYOS is working by doing a ping on just a couple of standard VM's on each host attached to a VLAN backed dvpg? You would need to set the VM's MTU to 1600, and set the ping to use 1600 MTU and set DF-Bit to ensure VXLAN will route through the VYOS router in its current configuration state to ensure its working.

For anything else, it would be helpful to see the output of a "show configuration" from your VYOS router, and screenshots of your VMK settings as shown in vCenter under the host, configuration, VMK so we can see the IP/SM/GW information.

[JFSynovatec]

My setup is simple as this:

ESX1 192.168.10.100   ---->   192.168.10.1 -Switch Dell N2000 - 192.168.11.1 <--------  ESX2 192.168.11.100

From one host or the other I can ping both gateways, just not the other VTEP IP.

The VMK stack configs are all good.

Something is blocking the other host trafic from reaching the other one and it'snot the backend routing.

[JFSynovatec]

I tried using a N2000 switch instead, and I get the same result.

I can ping all gateways put not the VMkernel IP of the other hosts.

I can ping the other gateway from any hosts (Host 192.168.10.100 can ping 192.168.11.1)

Here's some config and results that might help:

Switch:

interface vlan 110

ip address 192.168.10.1 255.255.255.0

exit

!

interface vlan 111

ip address 192.168.11.1 255.255.255.0

exit

!

interface Gi1/0/1

switchport mode trunk

switchport trunk allowed vlan 110

exit

!

interface Gi1/0/2

switchport mode trunk

switchport trunk allowed vlan 111

exit

console(config)#show interfaces configuration gigabitethernet 1/0/1

Port      Description                    Duplex Speed   Neg  MTU   Admin

                                                                   State

--------- ------------------------------ ------ ------- ---- ----- -----

Gi1/0/1                                  Full   1000    Auto 9000 Up

console(config)#show interfaces configuration gigabitethernet 1/0/2

Port      Description                    Duplex Speed   Neg  MTU   Admin

                                                                   State

--------- ------------------------------ ------ ------- ---- ----- -----

Gi1/0/2                                  Full   1000    Auto 9000 Up

ARP Table on the switch

IP Address       MAC Address        Interface       Type      Age

---------------  -----------------  --------------  --------  -----------

192.168.10.1     F8B1.565C.0307     Vl110           Local         n/a

192.168.10.100   0050.566D.1B68     Vl110           Dynamic    0h  2m 49s

192.168.11.1     F8B1.565C.0307     Vl111           Local         n/a

ESX1

vmk1       58                                      IPv4      192.168.10.100                          255.255.255.0   192.168.10.255  00:50:56:6d:1b:68 1600    65535     true    STATIC              vxlan

[root@esx1:~]  esxcli network ip neighbor list -N vxlan

Neighbor      Mac Address        Vmknic   Expiry  State  Type

------------  -----------------  ------  -------  -----  -----------

192.168.10.1  f8:b1:56:5c:03:07  vmk1    966 sec         Autorefresh

[root@esx1:~]  esxcli network ip route ipv4 list -N vxlan

Network       Netmask        Gateway       Interface  Source

------------  -------------  ------------  ---------  ------

default       0.0.0.0        192.168.10.1  vmk1       MANUAL

192.168.10.0  255.255.255.0  0.0.0.0       vmk1       MANUAL

ESX2

vmk1       49 IPv4      192.168.11.100                          255.255.255.0   192.168.11.255  00:50:56:6d:92:55 1600    65535     true    STATIC              vxlan

[root@ESX2:~] esxcli network ip neighbor list -N vxlan

Neighbor      Mac Address        Vmknic  Expiry  State  Type

------------  -----------------  ------  ------  -----  -----------

192.168.11.1  f8:b1:56:5c:03:07  vmk1    89 sec         Autorefresh

[root@ESX2:~] esxcli network ip route ipv4 list -N vxlan

Network       Netmask        Gateway       Interface  Source

------------  -------------  ------------  ---------  ------

default       0.0.0.0        192.168.11.1  vmk1       MANUAL

192.168.11.0  255.255.255.0  0.0.0.0       vmk1       MANUAL

[bayupw]

Are you using nested ESXi hosts?

[JFSynovatec]

Hi Brian,

I'm not trying to bridge VXLAN to another VLAN.

I'm trying to make VTEPs communating together even if they are on different VLANs(subnets)

When they use the same subnet it work great but not when they are not.

And I did a few tests, and they can both bing the other's router IP but not the other VTEP IP.

My L3 configs seems alright, it looks like a firewall issue that I can't figure.