ESG and 9k connect using UCS servers

[smcanuff]

I have NSX 6.2.3 install on 6 UCS rack servers that are in a cluster, each servers have two NIC that is connected to two Cisco 9k, one nice on a server going to 9K A and next NIC going to 9k B. I have iBGP running between the ESG and the 9K. I found a problem that ping and BGP peering not working on some of the ESG, until I remove one half of the connections going to any of the 9K and just have the Exi host connect to one 9K.

Can someone tell me what I am not doing? And how can I have all EXi host connect at the same time to the 9K and have iBGP working on all the uplinks. 9k ports are config as trunk ports.

[chuckbell]

Do you have vPC setup? Depending on your hardware it may not be supported.

Supported Topologies for Routing over Virtual Port Channel on Nexus Platforms - Cisco​

From NSX 3.0 design guide:

[rajeevsrikant]

Below is the blog from Brad Hedlund explaining why there should be no dynamic routing over vPC peer links. The same applies in the NSX environment as well.

Routing over Nexus 7000 vPC peer-link? Yes and No.

[VCDX159]

Make sure you are not using a vPC link to peer the ESGs with the upstream 9Ks as Cisco does NOT support this topology on the Nexus 9Ks.

Here is a link to Cisco's latest vPC design guide: http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices...

Paul A. Mancuso VCDX | VCI | CCNP Data Center | CISSP

VMware NSBU; Technical Product Manager

[lynchc]

This is my preferred method of setting this up. A distinct separate point to point port-channel from FI A to N9KA and FI B to N9KB

Each N9K has a BGP adjacency with all ESG's over a VLAN dedicated to that Fabric.

See diagrams.

Regards

Colin