MAC address of DLR gateway is different when running DHCP Relay

[treadmic]

I have a DLR with a gateway MAC address of 02:50:56:56:44:52 and I setup L2 firewall rules to only allow traffic to/from this MAC on the logical switch that the DLR is a gateway for. My intention is to only allow VMs on this network to only talk to the gateway and not to each other. When using the DHCP Relay feature on the DLR to relay DHCP to an Edge, the incoming packets from the DLR gateway are blocked by the L2 firewall rule. I've performed a packet trace on the host's switch port and found that the incoming DHCP response messages have a different MAC address than that of the DLR gateway, yet the IP address in the packet lists the gateway IP address. The MAC address list in the DHCP responses is 00:50:56:fa:90:62.

I've looked everywhere and cannot seem to figure out what this MAC is, where it came from, and why it's different than the DLR gateway MAC. I suspect it has something to do with the way NSX is handling broadcast messages. When the DHCP response is sent to the broadcast address, NSX is doing something with it and it ends up looking different to the VM.

Anyone have any ideas or troubleshooting steps?

[mhampto]

This may be related to the issue in DHCP relay agents do not function in NSX (2147322)​, if not, what version of NSX are you running? Have you had a chance to review similar forum posts?

[treadmic]

Thanks for the reference. I have seen this behavior as well and it's good to know it's been resolved. My specific issue is something different. I'm currently running on NSX 6.2.

After digging deeper I did find the mystery MAC address. When my ESG sends the DHCP OFFER message, the destination MAC and IP are my DLR's gateway address (172.16.0.1). The traffic goes from the ESG VM to the ESG VM's host. The DLR running on that host receives the DHCP OFFER message and attempts to relay the DHCP OFFER to my VM on the 172.16.0.0 network. When it creates the DHCP OFFER message,  it's using the MAC address of the active VTEP uplink interface on that host instead of the DLR MAC address. I believe this is a problem with the DHCP relay implementation. I don't think any packet within the vxlan network should have a MAC address of the vlan based uplink interface.

As shown above the encapsulated packet has 00:50:56:fa:90:62 as the source MAC address.

That MAC address is the address of the active Uplink interface for the VTEP.

I've raised this issue with VMWare support so we'll see where that goes.

[Mparayil]

Hi

Just want to let you know that the UPlink MAC what you are referring to is not VTEP's MAC or VMNIC's MAC it is a DLR PMAC DLR  has 2 Mac address concepts one called as V-MAC and P-MAC if you see the below OUTPUT from my LAB you will find the difference.

[root@esx-01a:~] net-vdr -C -l default+edge-e9648c5f-0f85-4eb2-9a66-0fc60c81b9cc >>>>>>>>>>>>>>>>DLR instance

Host locale Id:             42009e92-b1a7-d729-d5bf-5fa6838a6c1f >>>>>>>>>>>>>>>>>>>>>>>>>>Locale ID

Connection Information:

-----------------------

DvsName           VdrPort           NumLifs  VdrVmac

-------           -------           -------  -------

Mgmt_Edge_VDS     vdrPort           0        02:50:56:56:44:52 >>>>>>>>>>>>>>>>>This MAC remains the same in all the Host this is your Internal interface Gateway Mac address

    Vdr Switch Port: 50331656

    Teaming Policy: Default Teaming

    Uplink   : Uplink 1(50331650): 00:50:56:f6:ec:8e(Team member) >>>>>>>>>>>>>>This MAC remains Unique in all the Host this is an INNER Mac address technically like VM-MAC

   Stats : Pkt Dropped      Pkt Replaced     Pkt Skipped

   Input : 0                0                58525652

  Output : 0                0                18949223

The PMAC you will not see it on VTEP vmkernel or VMNIC's associated with VTEP interface.

[root@esx-01a:~] esxcfg-vmknic -l

Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack

vmk0       1                                       IPv4      192.168.110.51                          255.255.255.0   192.168.110.255 00:50:56:03:29:00 1500    65535     true    STATIC              defaultTcpipStack

vmk1       8                                       IPv4      10.10.20.21                             255.255.255.0   10.10.20.255    00:50:56:60:66:5c 1500    65535     true    STATIC              defaultTcpipStack

vmk2       16                                      IPv4      10.10.30.51                             255.255.255.0   10.10.30.255    00:50:56:62:46:2d 1500    65535     true    STATIC              defaultTcpipStack

vmk3       67                                      IPv4      192.168.150.52                          255.255.255.0   192.168.150.255 00:50:56:6f:35:29 1600    65535     true    STATIC              vxlan "VTEP MAC"

[root@esx-01a:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed     Duplex MAC Address       MTU    Description

vmnic0  0000:02:00.0 e1000       Up   1000Mbps  Full   00:50:56:03:29:00 1600   Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)

vmnic1  0000:02:01.0 e1000       Up   1000Mbps  Full   00:50:56:03:00:56 1500   Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)

Hope this might help you to further narrow down the issue and resolve it.