shank89's Accepted Solutions

First curl -sku admin:enterpassword -X PATCH 'https://nsxfqdn/policy/api/v1/infra/sites/default/napp/deployment/platform' -H "Content-Type: application/json" -d '{"deployment_action":{"action": "FORCE_UNDEPLOY"}}'

Technically with VDS7 and NSX3.X you shouldn't deploy with NVDS, but use the VDS option. Each host supports a single overlay transport zone on a single host switch (VDS for simplicity), but as many vlan transport zones as you want.  If you want additional overlay transport zones, they will need to be on a separate host switch with their own dedicated pnics.  So what you described is possible, as long as you meet these requirements. I believe this answers both your queries.  

If you have stateful services configured the traffic flow between VMs would be: VM1 > T1DR > T1SR > T0DR > T1 #2SR > T1 #2 DR > VM2. This is assuming that every flow from VM1 must hit the stateful service on the way out and must also traverse the stateful service on the second Tier-1 gateway.  Remove the SR's where not required. On the way out  VM1 > T1DR > T1SR > T0DR > T0SR > out FYI, a stateful service is LB, NAT etc, just attaching an Edge cluster to a T1 is not exactly a stateful service, even though it makes the gateway active-standby.  It's best to not attach the edge cluster to a T1 when there are no services being configured. Also the packet does not hit the same gateway twice within the one flow (like you have depicted in your first post). Hope this helps.

VDR = Virtual Distributed Router, which is instantiated on the transport nodes.  Which is essentially the logical routers you create. The packet captures in the example here should provide you with enough detail regarding VDR ports https://itdeepdive.com/2020/11/nsx-t-packet-capture-east-west-traffic-on-overlay-segment/.

Active-Active means both Edge nodes are active for data forwarding and routing operations.  Each additional 'Active' Edge node also adds an extra next hop for the  T1/T0's to balance across, for example.   The above addresses 169.254.0.2/3  exist on each Edge node (T0SR) on an Active/Active Tier-0 gateway.  The forwarding table in the image is that of the Tier-0 DR on a host transport node.  If the Tier-0 were in Active-Standby, there would only be one next hop / default route. ECMP (Equal cost multipathing) means multiple paths to a destination prefix can be installed in the routing table and forwarding table.  The load / packets can be balanced using a 5 tuple hashing algorithm with the available paths. For example, the below is the routing and forwarding table with ECMP enabled and a default route advertised from both upstream peers.   ECMP disabled   The other thing to consider is, if you have a Tier-0 gateway configured with Active-Active and ECMP disabled, you could inadvertently discard traffic if uRPF is enabled, as traffic may be received on an interface the T0SR did not expect it on. The behaviour of ECMP with the various protocols is the same, so I won't demonstrate between each protocol. Let me know if this clears it up for you. PS. I cover this topic in my book Thanks

In order for an edge to provide North South routing for an overlay segment and hosts within that segment, the transport zone the segment is attached to, should be attached to both the host transport node and edge transport node. The edge Can only belong to one overlay transport zone. Why are you creating multiple overlay transport zones? If it's for security you should use dfw, deploy more edges and attach them to the other tz's 

Hi,   The following two links should help you. https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-406AF9C3-E8F7-447A-8E3D-92AFB9D5E973.html https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-CC18C0E3-D076-41AA-8B8C-133650FDC2E7.html

"I'm not 100% sure which scenario you're describing, but I think you're saying that the edge has little awareness of what is happening upstream of its own virtual nics (i.e., the fastpath interfaces). Therefore, in my scenario where the external traffic is pinned to vmnic#1 and TEP traffic to vmnic#2, if vmnic#2 dies, the Edge doesn't know and blackholes the traffic because his fastpath interfaces are still up. Is that correct?" For the failure scenario, that is correct, especially if you have the portgroups attached to the vnic of the Edge in Active/unused (no redundancy). "How then, do VM Edges know that they are in a down state?" There are various mechanisms used to determine a failure condition, without going too deep into the architecture of the Edge node, it requires management connectivity, and at least one TEP tunnel to be considered up.  There are smarts to determine various states of the Edge and strategic active SR failover. Keep in mind, within the NSX-T fabric there are GENEVE tunnels between each endpoint and BFD sessions to determine tunnel and endpoint availability.  This is the 'heartbeat' functionality that you are referring to. And no probs, happy to help!  

Hi, I've gone through your questions, and have a few observations. You want to be careful of limiting vmnic failover of the uplink port groups on the Edges, it is good to have them in active/standby.  Remember your TEP interfaces are also plumbed through there, and if they do not failover, you will end up with traffic that is blackholed.  If the vmnic fails, the status goes to failback, and the TEP interface on the Edge is not aware of it and is still online.  As long as the Active SR is on the Edge and traffic is still being forwarded to it from the host transport nodes, you will have packet loss.  Hopefully this point makes sense. Traffic steering with A/S works similarly to A/A, use named teaming policies, dictating VLANs, and attach said policy to uplink segments that are subsequently attached to your T0 interfaces. I have not seen ARP issues relating to the problem you described.  That's not to say there haven't ever been, but they were related to bugs. You could let TEP traffic be balanced with A/A uplink portgroups, but that would make the traffic egress less deterministic. Hopefully this has helped in someway!  

Hi PhoenixVM, It's definitely something to consider,  but sometimes overthought.  To be honest, the only time I have seen customers go with A/S, is for a specific reason, eg.  their upstream devices only support A/S etc. A couple of pointers A/A is VVD compliant: https://docs.vmware.com/en/VMware-Validated-Design/6.2/sddc-architecture-and-design-for-a-virtual-infrastructure-workload-domain/GUID-8E524188-65F6-4FF2-B8A3-2D9B0C86FE2A.html A/A aside from ECMP, also allows higher throughput Rather than enabling a stateful service on the Tier-0 which would pin all tenancies / T1's to a specific Edge for ingress and egress.  The better option is to configure them on a Tier-1, ensuring only traffic that needs to route through a specific edge does Depending on your edge cluster design, from the many deployments of NSX-T, the impact of traffic ingressing the Edge not active for the stateful service is non-existent.  That is, the impact of traffic ingressing this edge unnoticeable.  It is common for this to be over thought and over-engineered.  Again, this comment is dependent on appropriate edge cluster design (A/A SRs local and not spanning sites). At the end of the day, the deployment model is up to the customer and their specific requirements.  But if you are talking about a single site, SRs not spanning across WAN links etc, then it shouldn't really be a concern. Cheers

First 5 dot points look good, what is the network you are stretching? 2 Manager Node in Site-A and 1 Node in Site-B --- keep in mind while you only have 1 manager up and running, NSX-T will be in read-only mode.  You need at least 2 up for write access, but best to get all nodes up and running ASAP. For the segments to exist at both sites, yes same TZ for easiest DR process. All Edges will be in the same TZ so they can route traffic for the segments in those TZ's. 2 Edges can be in each site, make sure you understand the traffic flow from hosts to edges, it will be balanced using all paths available to the T0SR;s (I linked you to this earlier). Correct, route maps, ,prepends, peering, BFD as required. A/S is your choice for T1, what is your reason for this? Segment in Overlay TZ, this TZ will be linked to hosts and edges T0DR, T1DR and Segments works in active site, because Host and Edge are in the same TZ -- not exactly, see my note above about datapaths and ECMP.

The option has been removed, you need to leverage firewall policies to create the type of behaviour that you would like.   IE a catch all deny or permit, after filtering applications / workloads above.

It sounds like the edges are wired incorrectly. Edge vms vnic in vcenter should be trunking portgroups. The first vnic is your management portgroup, this doesn't have to be a trunk. Within nsx create some vlan backed segments, tag them with your uplink vlans, then on the t0 uplink interfaces use these segments as the connected to segments when assigning the uplink interface ip  addresses to the edges.   Retest the ping after this.

Can you also post the Edge's N-VDS config.  Are those portgroups trunked portgroups or tagging a specific VLAN? I think this is going to come down to how you have configured the uplink interfaces on the edge.

Spoke with Nagesh, ended up being a two-fold issue.  One needing to allow the routes in (which seems to be a new thing) and two there was an assymetric routing issue in the environment which was not helping. After resolving both of those, it all worked.