I have read the CDO whitepapaper and watched the VMworld session on VTEP/MAC/ARP tables and I still find that some details are lacking:
I understand the idea behind a CDO switch, which all ESXi hosts join and use to send BUM traffic, when controller communication is lost. What I do not understand is what happens behind the curtains.
When we talk about NSX, BUM traffic is really just ARP requests that source VMs send to find destination VM MACs. Actual unicast VM communication has to happen over non-CDO logical switches, am I right? If that's the case, all hosts need to update their VTEP tables before that happens. How is that achieved in CDO mode?
Let's imagine a simple scenario:
1. Controller communication is lost
2. New VM1 is created and connected to VXLAN 5000 on ESX1, which was not part of VNI 5000 before controller communication was lost.
3. VM1 tries to communicate to VM2 connected to VXLAN 5000, running on ESX2, which joined VNI 5000 before controller communication was lost.
4. ESX1 sends an ARP request to find VM2 MAC via CDO switch VNI 4999
What next? My wild guess:
5. When ESX1 sent the ARP request via CDO switch, it marked the ARP packet with VNI 5000 and ESX2 adds ESXi to its VTEP table for VNI 5000
6. ESX2 sends ARP reply via CDO switch (or maybe even via the actual non-CDO logical switch?), marking packet with VNI 5000 and ESX1 adds ESX2 to its VTEP table for VNI 5000
7. VMs continue to communicate as normal via logical switch for VNI 5000
I might be completely wrong here.
I think I have to attract bayupw attention if I really want to get an answer to this.
What I found out testing was that during CDO operation, the MAC table gets built in the logical switch's VNI, but the vtep table is still empty. Since the MAC table has the destination inner MAC and the VTEPs IP and MAC, it can send the packet.