Hello!
I'm trying different scenarios in a lab and I have some difficulties with VXLAN bridging in Unicast mode.
I have 2 clusters with one Host in each.
I'd like each VMK to be in a different subnet and routed together with VyOS VM (With interfaces at 1600 MTU)
I have added all the routes needed and each VMK can reach each other successfully.
I have created all needed infra (Logical switch etc,)
I was wondering if it's possible to "route' the VXLAN VMK to another one on a different subnet or do you absolutely need multicasting from the moment it has to through routing?
Thanks!
Yes, you can have you your VMKs on separate subnets and if your control plane mode is already unicast, then multicast isn't required at all as described on page 38 of the VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0 .
I feel there's something missing.
I've checked the controller and can see that all MACs are registered under the good VTEP, ESXi Hosts, etc.
But, when VTEPs are on different VLANs (subnet) they can't reach each other.
On ESXi hosts, the routing table is not set si that the vxlan stack will use a router IP to reach other VTEPs, is that whatMs missing?
My router can effectively ping both VTEPs. But ESXi hosts cannot ping the other VTEP.
Any ideas?
If you're using IP pools to assign the VXLAN VMK ports, the VXLAN stack will use the default gateway you configured in the IP pool to reach the other cluster's subnet. If you're using DHCP, it'll use the default gateway assigned there.
Either way, I'd first try running "esxcli network ip route ipv4 list -N vxlan" on the affected hosts to make sure that the default gateway for the VXLAN stack is what you think it should be as well as using the "esxcli network ip neighbor list -N vxlan" to make sure that it has the MAC of your default gateway box in its ARP table. You can also verify that you can ping the default gateway from the VXLAN VMK port using "ping ++netstack=vxlan -I vmk1 x.x.x.x".
Found out that I can effectively ping the other gateway but not the other VTEP vmknic IP.
Frewall issue?
Multicast is not a requirement for VTEP reachability when on different subnets. Just IP reachability and MTU is. Also, VYOS does not support multicast routing so you definitely need Unicast control plane if using VYOS as your gateway between VLAN backed dvPG's and VMK's.
Can you validate the MTU for VYOS is working by doing a ping on just a couple of standard VM's on each host attached to a VLAN backed dvpg? You would need to set the VM's MTU to 1600, and set the ping to use 1600 MTU and set DF-Bit to ensure VXLAN will route through the VYOS router in its current configuration state to ensure its working.
For anything else, it would be helpful to see the output of a "show configuration" from your VYOS router, and screenshots of your VMK settings as shown in vCenter under the host, configuration, VMK so we can see the IP/SM/GW information.
My setup is simple as this:
ESX1 192.168.10.100 ----> 192.168.10.1 -Switch Dell N2000 - 192.168.11.1 <-------- ESX2 192.168.11.100
From one host or the other I can ping both gateways, just not the other VTEP IP.
The VMK stack configs are all good.
Something is blocking the other host trafic from reaching the other one and it'snot the backend routing.
I tried using a N2000 switch instead, and I get the same result.
I can ping all gateways put not the VMkernel IP of the other hosts.
I can ping the other gateway from any hosts (Host 192.168.10.100 can ping 192.168.11.1)
Here's some config and results that might help:
Switch:
interface vlan 110
ip address 192.168.10.1 255.255.255.0
exit
!
interface vlan 111
ip address 192.168.11.1 255.255.255.0
exit
!
interface Gi1/0/1
switchport mode trunk
switchport trunk allowed vlan 110
exit
!
interface Gi1/0/2
switchport mode trunk
switchport trunk allowed vlan 111
exit
console(config)#show interfaces configuration gigabitethernet 1/0/1
Port Description Duplex Speed Neg MTU Admin
State
--------- ------------------------------ ------ ------- ---- ----- -----
Gi1/0/1 Full 1000 Auto 9000 Up
console(config)#show interfaces configuration gigabitethernet 1/0/2
Port Description Duplex Speed Neg MTU Admin
State
--------- ------------------------------ ------ ------- ---- ----- -----
Gi1/0/2 Full 1000 Auto 9000 Up
ARP Table on the switch
IP Address MAC Address Interface Type Age
--------------- ----------------- -------------- -------- -----------
192.168.10.1 F8B1.565C.0307 Vl110 Local n/a
192.168.10.100 0050.566D.1B68 Vl110 Dynamic 0h 2m 49s
192.168.11.1 F8B1.565C.0307 Vl111 Local n/a
ESX1
vmk1 58 IPv4 192.168.10.100 255.255.255.0 192.168.10.255 00:50:56:6d:1b:68 1600 65535 true STATIC vxlan
[root@esx1:~] esxcli network ip neighbor list -N vxlan
Neighbor Mac Address Vmknic Expiry State Type
------------ ----------------- ------ ------- ----- -----------
192.168.10.1 f8:b1:56:5c:03:07 vmk1 966 sec Autorefresh
[root@esx1:~] esxcli network ip route ipv4 list -N vxlan
Network Netmask Gateway Interface Source
------------ ------------- ------------ --------- ------
default 0.0.0.0 192.168.10.1 vmk1 MANUAL
192.168.10.0 255.255.255.0 0.0.0.0 vmk1 MANUAL
ESX2
vmk1 49 IPv4 192.168.11.100 255.255.255.0 192.168.11.255 00:50:56:6d:92:55 1600 65535 true STATIC vxlan
[root@ESX2:~] esxcli network ip neighbor list -N vxlan
Neighbor Mac Address Vmknic Expiry State Type
------------ ----------------- ------ ------ ----- -----------
192.168.11.1 f8:b1:56:5c:03:07 vmk1 89 sec Autorefresh
[root@ESX2:~] esxcli network ip route ipv4 list -N vxlan
Network Netmask Gateway Interface Source
------------ ------------- ------------ --------- ------
default 0.0.0.0 192.168.11.1 vmk1 MANUAL
192.168.11.0 255.255.255.0 0.0.0.0 vmk1 MANUAL
Are you using nested ESXi hosts?
Hi Brian,
I'm not trying to bridge VXLAN to another VLAN.
I'm trying to make VTEPs communating together even if they are on different VLANs(subnets)
When they use the same subnet it work great but not when they are not.
And I did a few tests, and they can both bing the other's router IP but not the other VTEP IP.
My L3 configs seems alright, it looks like a firewall issue that I can't figure.