NSX vxlan-VTEP and controller communication

[Fanboyuuegl]

HI,

I have network background and have experience with cisco vxlan+evpn solution. Now as business required, I began to touch NSX.

I have to admit NSX makes things easier comparing with the complex Evpn solution.

While there are something confurses me and I hope someone can help to clearify:

  —————————————————————

  |               |                 |                     |                  |

vtep1      vtep2           vtep3            vtep4        controller

  |              |                 |                     |          

VM1        VM2           VM3               VM4

above is the simplified topology. when VMs are online, Vteps will send all relevent details(mac,ip,vtep segment id) to controller, so controller will form a large VIB tables for vxlan data plane.

1. Will controller sync these VIB tables(mac table and arp table) with all VTEP1-4?

2.if yes what is the mechniasm? since VMs behind VTEPs cannot be online and offline, when it happens VIBs in controller will change.So how will controller update VIB in time，periodic update or triggered update?

3.if NO. when VM1 behind VTEP1 wants to speak to VM2 VTEP2. VTEP1 needs to  ask controller where is VM2 and controller reply it is behind VTEP2. if this is done everytime among VTEPS, controller will have unnecessary burdens.

EVPN provides the controll plane making every VTEPs has a synced mac table and arp table, I just do not understand how NSX deai with it.

Regards

Michael

[Sreec]

I think you are confused with VTEP learning procedure. In a nutshell -

NSX-v controller cluster is used to populate the VTEP, MAC, and ARP tables . Basically each host will send the reports (VTEP,MAC,ARP) to controllers and controllers  eventually populates its local VTEP table and with this information ,it will send a VTEP report(VTEP -VNI mapping) message to all ESXi hypervisors hosting VMs actively(This is important concept to remember, if there are no active workloads , no reports be will be shared)connected to that same VXLAN segment.

I would request you to go through below design guide( page 33) and start reading from logical switching till VTEP learning methods. That would certainly clear your basic doubts.

https://communities.vmware.com/servlet/JiveServlet/previewBody/27683-102-10-41631/NSX%20Reference%20...

[Fanboyuuegl]

Hi Sreec,

thanks for your reply and it is very helpful, I found the report mechanism on section “populating the controller table”.

However, I stucked on page 44 figure 35. arp suppression:

IF VM1 wants to speak to vm2, it will send an arp to controller (according to the guide)

However, since controller will send report to all vteps, all vteps will eventualy have the same mac, arp tables with controller and therefore they should take the responsiblity of arp suppression.

if so,why is this arp request still sent to controller?

why not just send BUM straightaway?

—————————————————————

  |               |                 |                     |                  |

vtep1      vtep2           vtep3            vtep4        controller

  |              |                 |                     |        

VM1        VM2           VM3               VM4

for evpn +vxlan solution since there is no controller so vtep takes all the jobs.

While NSX has this controller, so I just want to understand how this controller acts and how important it is.

SO please bear with my question.

Regards

Michael

[Sreec]

For MAC learning, ESXI Host is only aware of locally connected VM MAC address . Controller stores routing,mac,arp and VTEP table.

[Fanboyuuegl]

Hi Sreec,

so actually controller keeps a full mac arp table and vteps will only keep their local mac arp tables.

when hosts in vtep1 wants to hosts in other vteps, vtep will also communicate to control first and control will reply with the host detail.

you mentioned controller will eventually populate the arp and mac table back to vteps via report, is this the way how controller reports back to vtep?

also, for instance, vm1 in vtep1 wants to speak to vm2 in vtep2 and controller has already replied vtep for vm2's detail.

I believe at this stage vtep1 will save vm2's mac and arp details locally.

（save vm2 locally after querying with controller）

  |

vtep1-----------------vtep2--------controller

|                              |

vm1                        vm2

how long will vtep1 keeps this detail in mac and arp table??

say after hours if vm1 wants to speak to vm2 again, willl vtep1 query controller again ??

or if vm2 is offline, how will vtep1 knows about it ??

Appreciate your patience and your precious time!

Regards

Michael

[Sreec · Accepted solution]

Yes, Controllers don't share arp table reports to ESXI host.  You should also listen to VMworld 2017 - NET1775BU - Advanced VMware NSX: Demystifying the VTEP, MAC, and ARP Tables - YouTube...give you same insight with proper packet flow.  However just to add few more points, If DLR is involved, they have ARP table(Stores locally on each host) with a time out of 600 if I'm not wrong ,NSX edges also have ARP table( unsure about the timeout values of edges and controllers) .

[Fanboyuuegl]

watched video

outstanding elaboration!

thanks again Sreec.

Cheers

Michael

[Fanboyuuegl]

Thanks for pointing out that when DLR is involved, it will have has its arp table.

I believe the involvment of DLR will help for arp suppression. With pure Dvs, when vmA wants to speak to another vmB on another vtep, vmA will refer to controller for vmB‘s detail and in this way, BUM will not be necessary.

While if DLR is introduced:

1. Will controller sync the whole arp, mac table to each DLR on each ESXI hosts? in this way, DLR will replace controller for arp suppression.

     VTEP1(DLR)----------------------------VTEP2(DLR)------------------controller(has vtep,mac and arp table;sync to all DLR on all vteps)

             |                                                    |

           vmA                                              vmB

2. Or DLR only keeps the mac and arp entry for VMs in other VTEPs for 600s. when there is no active traffic, dlr will still keep entries for 600s after which entries will remove.

    until next time, when vmA wishes to speak to vmB, it will ask controller and controller will reply->DLR get the reply and put the entry into arp and mac table again.

     (i have arp mac details for vmB,because vmA asked controller before.but if there is no active traffic to vmB，these details are gone in 600s)

                     |

     VTEP1(DLR)----------------------------VTEP2(DLR)------------------controller

             |                                                    |

           vmA                                              vmB

Appreciate your time.

Regards

Michael

[Sreec]

While if DLR is introduced:

1. Will controller sync the whole arp, mac table to each DLR on each ESXI hosts? in this way, DLR will replace controller for arp suppression.

     VTEP1(DLR)----------------------------VTEP2(DLR)------------------controller(has vtep,mac and arp table;sync to all DLR on all vteps)

             |                                                    |

           vmA                                              vmB

Controllers play no role in DLR ARP process(They don't distribute/share the ARP table with DLR) - I hope this answers your second query as well.

Refer this document for detailed flow ->DLR ARP Resolution Process