9 Replies Latest reply on Jul 21, 2018 10:10 AM by JFSynovatec

    VXLAN Unicast over L3

    JFSynovatec Lurker

      Hello!

       

      I'm trying different scenarios in a lab and I have some difficulties with VXLAN bridging in Unicast mode.

       

      I have 2 clusters with one Host in each.

       

      I'd like each VMK to be in a different subnet and routed together with VyOS VM (With interfaces at 1600 MTU)

      I have added all the routes needed and each VMK can reach each other successfully.

      I have created all needed infra (Logical switch etc,)

       

      I was wondering if it's possible to "route' the VXLAN VMK to another one on a different subnet or do you absolutely need multicasting from the moment it has to through routing?

       

      Thanks!

        • 1. Re: VXLAN Unicast over L3
          lhoffer Hot Shot
          vExpertVMware Employees

          Yes, you can have you your VMKs on separate subnets and if your control plane mode is already unicast, then multicast isn't required at all as described on page 38 of the VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0 .

          • 2. Re: VXLAN Unicast over L3
            JFSynovatec Lurker

            I feel there's something missing.

             

            I've checked the controller and can see that all MACs are registered under the good VTEP, ESXi Hosts, etc.

             

            But, when VTEPs are on different VLANs (subnet) they can't reach each other.

             

            On ESXi hosts, the routing table is not set si that the vxlan stack will use a router IP to reach other VTEPs, is that whatMs missing?

             

            My router can effectively ping both VTEPs. But ESXi hosts cannot ping the other VTEP.

             

            Any ideas?

            • 3. Re: VXLAN Unicast over L3
              lhoffer Hot Shot
              VMware EmployeesvExpert

              If you're using IP pools to assign the VXLAN VMK ports, the VXLAN stack will use the default gateway you configured in the IP pool to reach the other cluster's subnet.  If you're using DHCP, it'll use the default gateway assigned there.

               

              Either way, I'd first try running "esxcli network ip route ipv4 list -N vxlan" on the affected hosts to make sure that the default gateway for the VXLAN stack is what you think it should be as well as using the "esxcli network ip neighbor list -N vxlan" to make sure that it has the MAC of your default gateway box in its ARP table.  You can also verify that you can ping the default gateway from the VXLAN VMK port using "ping ++netstack=vxlan -I vmk1 x.x.x.x".

              • 4. Re: VXLAN Unicast over L3
                JFSynovatec Lurker

                Found out that I can effectively ping the other gateway but not the other VTEP vmknic IP.

                 

                Frewall issue?

                • 5. Re: VXLAN Unicast over L3
                  jamib Novice
                  VMware Employees

                  Multicast is not a requirement for VTEP reachability when on different subnets. Just IP reachability and MTU is. Also, VYOS does not support multicast routing so you definitely need Unicast control plane if using VYOS as your gateway between VLAN backed dvPG's and VMK's.

                   

                  Can you validate the MTU for VYOS is working by doing a ping on just a couple of standard VM's on each host attached to a VLAN backed dvpg? You would need to set the VM's MTU to 1600, and set the ping to use 1600 MTU and set DF-Bit to ensure VXLAN will route through the VYOS router in its current configuration state to ensure its working.

                   

                  For anything else, it would be helpful to see the output of a "show configuration" from your VYOS router, and screenshots of your VMK settings as shown in vCenter under the host, configuration, VMK so we can see the IP/SM/GW information.

                  • 6. Re: VXLAN Unicast over L3
                    JFSynovatec Lurker

                    My setup is simple as this:

                     

                    ESX1 192.168.10.100   ---->   192.168.10.1 -Switch Dell N2000 - 192.168.11.1 <--------  ESX2 192.168.11.100

                     

                    From one host or the other I can ping both gateways, just not the other VTEP IP.

                    The VMK stack configs are all good.

                     

                    Something is blocking the other host trafic from reaching the other one and it'snot the backend routing.

                    • 7. Re: VXLAN Unicast over L3
                      JFSynovatec Lurker

                      I tried using a N2000 switch instead, and I get the same result.

                      I can ping all gateways put not the VMkernel IP of the other hosts.

                      I can ping the other gateway from any hosts (Host 192.168.10.100 can ping 192.168.11.1)

                       

                      Here's some config and results that might help:

                       

                       

                       

                      Switch:

                      interface vlan 110

                      ip address 192.168.10.1 255.255.255.0

                      exit

                      !

                      interface vlan 111

                      ip address 192.168.11.1 255.255.255.0

                      exit

                      !

                      interface Gi1/0/1

                      switchport mode trunk

                      switchport trunk allowed vlan 110

                      exit

                      !

                      interface Gi1/0/2

                      switchport mode trunk

                      switchport trunk allowed vlan 111

                      exit

                       

                       

                       

                      console(config)#show interfaces configuration gigabitethernet 1/0/1

                       

                       

                      Port      Description                    Duplex Speed   Neg  MTU   Admin

                                                                                         State

                      --------- ------------------------------ ------ ------- ---- ----- -----

                      Gi1/0/1                                  Full   1000    Auto 9000 Up

                       

                       

                      console(config)#show interfaces configuration gigabitethernet 1/0/2

                       

                       

                      Port      Description                    Duplex Speed   Neg  MTU   Admin

                                                                                         State

                      --------- ------------------------------ ------ ------- ---- ----- -----

                      Gi1/0/2                                  Full   1000    Auto 9000 Up

                       

                       

                       

                      ARP Table on the switch

                      IP Address       MAC Address        Interface       Type      Age

                      ---------------  -----------------  --------------  --------  -----------

                      192.168.10.1     F8B1.565C.0307     Vl110           Local         n/a

                      192.168.10.100   0050.566D.1B68     Vl110           Dynamic    0h  2m 49s

                      192.168.11.1     F8B1.565C.0307     Vl111           Local         n/a

                       

                       

                       

                       

                       

                       

                       

                       

                       

                       

                      ESX1

                       

                       

                      vmk1       58                                      IPv4      192.168.10.100                          255.255.255.0   192.168.10.255  00:50:56:6d:1b:68 1600    65535     true    STATIC              vxlan

                       

                       

                       

                      [root@esx1:~]  esxcli network ip neighbor list -N vxlan

                      Neighbor      Mac Address        Vmknic   Expiry  State  Type

                      ------------  -----------------  ------  -------  -----  -----------

                      192.168.10.1  f8:b1:56:5c:03:07  vmk1    966 sec         Autorefresh

                       

                       

                       

                       

                      [root@esx1:~]  esxcli network ip route ipv4 list -N vxlan

                      Network       Netmask        Gateway       Interface  Source

                      ------------  -------------  ------------  ---------  ------

                      default       0.0.0.0        192.168.10.1  vmk1       MANUAL

                      192.168.10.0  255.255.255.0  0.0.0.0       vmk1       MANUAL

                       

                       

                       

                      ESX2

                       

                       

                      vmk1       49 IPv4      192.168.11.100                          255.255.255.0   192.168.11.255  00:50:56:6d:92:55 1600    65535     true    STATIC              vxlan

                       

                       

                      [root@ESX2:~] esxcli network ip neighbor list -N vxlan

                      Neighbor      Mac Address        Vmknic  Expiry  State  Type

                      ------------  -----------------  ------  ------  -----  -----------

                      192.168.11.1  f8:b1:56:5c:03:07  vmk1    89 sec         Autorefresh

                       

                       

                      [root@ESX2:~] esxcli network ip route ipv4 list -N vxlan

                      Network       Netmask        Gateway       Interface  Source

                      ------------  -------------  ------------  ---------  ------

                      default       0.0.0.0        192.168.11.1  vmk1       MANUAL

                      192.168.11.0  255.255.255.0  0.0.0.0       vmk1       MANUAL

                      • 8. Re: VXLAN Unicast over L3
                        Bayu Wibowo Master
                        Community WarriorsUser Moderators

                        Are you using nested ESXi hosts?

                        Bayu Wibowo | vExpert NSX, VCIX6-DCV/NV, Cisco Champion, AWS-SAA
                        Author of VMware NSX Cookbook http://bit.ly/NSXCookbook
                        https://nz.linkedin.com/in/bayupw | twitter @bayupw
                        • 9. Re: VXLAN Unicast over L3
                          JFSynovatec Lurker

                          Hi Brian,

                           

                          I'm not trying to bridge VXLAN to another VLAN.

                           

                          I'm trying to make VTEPs communating together even if they are on different VLANs(subnets)

                          When they use the same subnet it work great but not when they are not.

                          And I did a few tests, and they can both bing the other's router IP but not the other VTEP IP.

                           

                          My L3 configs seems alright, it looks like a firewall issue that I can't figure.