NSX Edge Trunks

[JonathanShz]

Hi,

I would like to understand how Trunk interfaces work on NSX Edges but don't find any detailed enough documentation  to satisfy my curiosity

I want to implement a Trunk interface on the vnic0 of my production Edge. This Trunk will only carry subinterfaces connected to logical switchs (VXLAN only).

I understand that I have to create a Distributed Port Group on my Edge compute cluster VDS and connect the Trunk interface to it.

This is the most confusing point for me. what is the role of this port group in the packet walk ? Because the logical switchs I connect to subinterfaces already have a dvpg backing device connected to that same VDS (dynamically created when the logical switch is created).

Should the dvpg I manually create for the trunk interface have a unique VLAN ? all VLANS ? being untagued ?

If I configure this dvpg in let's say VLAN 1000, do I have to tag this VLAN on the physical switches connected to the Edge compute cluster ?

Finally, If I want to implement Trunk interfaces on several NSX Edges do I have to create one dvpg for each Edge or can I share the same dvpg for all edges ?

More confusing to me is the capture on the Trunk interface vnic (on the ESXi with: pktcap-uw --switchport <switchport where the trunk vnic is> --capture VnicTx):

13:53:23.442935 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 148: 0.0.0.0.0 > 0.0.0.0.8472: VXLAN, VNI 30000, Flags [I]

   00:50:56:aa:aa:aa > 00:50:56:bb:bb:bb, ethertype IPv4 (0x0800), length 98: 10.1.1.1 > 10.1.1.10: ICMP echo request, id 10918, seq 17, length 64

The Edge appliance seems to encapsulate the ICMP echo request payload packet  in VXLAN with old UDP port 8472  and the VNI of my logical switch. MACs and IPs on the VXLAN layer being 0...  No trace of the tunnel Id nor the trunk dvpg vlan id in the packets however.

Thanks for your lights.

[DominicFoley]

The uplink interface of your ESG will belong to a VLAN-backed port group and this uplink is used for dynamic/static routing into your VXLAN networks below the ESG.

Now, an ESG can have 10 interfaces, and this is where the trunk interface comes in.  Let's say you want 20 logical switches off this ESG, then create a trunk interface, and then the logical switches will be part of that trunk.  So single uplink (VLAN), multiple downlink (VXLANs on the trunk).  You can then connect other edges or dLR's to the logical switches or just Vm's.

You do however need to create an 'trunking' port group on your dVS.  Essentially all sub-interfaces will be encapsulated within that VLAN port group.  The vlan only needs to exist on the edge cluster, as compute hosts do not require it (whether that VLAN spans your ToR architecture is entirely dependent on youyr physical design).  It also does not require any L3 configuration, just a bulk standard L2 vlan created on your physical switches, because at one end of your trunked logical switch will be your ESG, with a LIF configured, and the other will either be a dLR or ESG interface or just VM's bound to it, with the LIF on the trunking ESG as the default gateway.

Please mark correct if this has helped.

Thanks.

[JonathanShz]

Thank's Dominic for your answer. Do you know what's the best practice concerning these DVPG ?

Do I have to create one unique DVPG per Edge or can I reuse the same one on my DVS for all the Edges I create without loosing isolation ?

If not possible can I at least reuse the same VLAN ID on every  DVPG to avoid touching the physical switches too often (that's one of the main interest in using VXLAN) ?

[DominicFoley]

That's a good question, let me get back to you on it.  I can't see why you couldn't use the same dVPG per edge, whether that is the recommended way I'm not sure.  I'll get back to you.

[errevi]

Hi,

nice question, I was wondering the same too..
My conclusion is that in your specific case (inter-vxlan routing only, performed by the means of sub-interfaces on the same trunk vnic) worrying about which dPG is to be used is useless because, as you've already pointed out, logical switches routed by the ESG LIFs have already their own dPG.
In other words yes, you're required to enter a dPG in the "connected to" field of the trunk vnic properties, but as long as you don't need to bridge any of the sub-interfaces to an external Vlan (indeed that dPG that you pick is vlan backed) the dPG choice is not relevant.

On the contrary, supposing that you need to have 1 ESG sub-interface connected to an external network (physical vlan) then in that case choosing the correct dPG is critical.