First of all, thanks for your knowledge and time to see my question. 
We are about to deploy the NSX TEST environment in house before we deploy the production. Unfortunately, Test environment is pretty old network (Cisco 4500 switch with HSRP) with traditional architecture while production environment has Nexus 9K with vPC. Cisco 4500 switches are working as access/distribute/core as you can see below. However, it gave me an opportunity to better understand the design of TWO edges. Ok, here are my questions.
1. HSRP is a layer 2 technology meaning that it use the VIP mac address for forwarding in layer 2 domain, therefore instead of HSRP, I think I should use layer 3 protocol such as BGP and OSPF for redundancy. I am going to create a separated vlan for each core (vlan 4 for left switch / vlan 5 for right switch). HSRP does not know about both vlan 4 and 5. For example, left core does not have vlan 5 while right core does not have vlan 4. Am I correct so far?
2. Please clarify if these statements are correct (or guide me a best practice)
In terms of the perspective of left side edge (L-Edge), L-Edge makes OSPF neighbors with L-Core and R-Edge by using vlan 4.
In terms of the perspective of right side edge (R-Edge), R-Edge makes OPSF neighbors with R-Core and L-Edge by using vlan 5.
3. What is the best practice here in edge uplinks?
Should L-Edge only connect L-core switch by using vlan 4 while R-Edge only connects R-core switch by using vlan 5? or my below topology is fine? Basically I like to have a good redundancy when one of cores is down (or when one of Edges is down).
4. Are vlan 4 and vlan 5 VXLANs, so I should make MTU size 1600?
I know that I can use HA of Edges, but I like to utilize both uplinks through two cores.
1. HSRP is a layer 2 technology meaning that it use the VIP mac address for forwarding in layer 2 domain, therefore instead of HSRP, I think I should use layer 3 protocol such as BGP and OSPF for redundancy. I am going to create a separated vlan for each core (vlan 4 for left switch / vlan 5 for right switch). HSRP does not know about both vlan 4 and 5. For example, left core does not have vlan 5 while right core does not have vlan 4. Am I correct so far?
You certainly need a routing protocol for Edge peering with Cisco 4503 . Assuming you have DLR use case as well, its better to simply End-End routing either with OSPF or BGP ,but better don't mix and match by having few OSPF , few bgp kind of design .Also you have mentioned VLAN 4 and 5 is not part of HSRP. That is fine and i don't find any challenges with that approach But one of the requirement is Edge and Core redundancy based on failure scenarios. So at max one device is required (1 edge and 1 core) for traffic to flow flawlessly. Below matrix is for OSPF & BGP and be aware of what features are available in HA&ECMP modes. In your case you are looking for Edge-HA .
Please go through the routing section once, since they are much more points that you should consider or be aware of.
2. Please clarify if these statements are correct (or guide me a best practice)
In terms of the perspective of left side edge (L-Edge), L-Edge makes OSPF neighbors with L-Core and R-Edge by using vlan 4.
In terms of the perspective of right side edge (R-Edge), R-Edge makes OPSF neighbors with R-Core and L-Edge by using vlan 5.
I'm bit confused with above statement/requirement . Are you in need of having routes learned by each Edge to be distributed to core and bring them under a single area ? If you are going with OSPF approach you area design matters more than VLAN-ID's
3. What is the best practice here in edge uplinks?
Should L-Edge only connect L-core switch by using vlan 4 while R-Edge only connects R-core switch by using vlan 5? or my below topology is fine? Basically I like to have a good redundancy when one of cores is down (or when one of Edges is down).
So you are in need of sending same routes via both the edges via different upstream VLANS and expecting redundancy ? ECMP is better approach design here and you can limit the ECMP edges to 2 in number . But no state-full services at edges in that case.
4. Are vlan 4 and vlan 5 VXLANs, so I should make MTU size 1600?
This depends upon on the uplinks selected for VTEP configurations. Assuming you have dedicated uplinks for VTEP connectivity , MTU change is required for entire VTEP path across single/multi subnet based on VTEP design approach. The MTU for the VDS uplinks of the ESXi hosts performing VXLAN encapsulation is automatically increased when preparing the host for VXLAN
Thank you very much Sreec, can you review below comments with updated diagram? I believe that my diagram was not clear along with my questions. Above diagrams are a updated version for physical (left) and logical(right).
In terms of physical diagram, as you can see Red physical connection is a trunk (vlan 1,2,3,4 and 6) interface connected left Cisco 4503 while Green physical connection is a trunk (vlan 1,2,3,5 and 6) interface connected to right Cisco 4503.
In terms of logical routing diagram, left Cisco 4503 will have OSPF adjacency with both ESG 1 and 2 through vlan 4. Right Cisco 4503 will have OSPF adjacency with both ESG 1 and 2 through vlan 5. They are all in same area 0.
Does above statement make sense?
Q1: Yes I am planning to use OSPF area 0 for everything (physical to virtual including ESG and DLR). Also you are correct. I am planning to use ECMP from both DLR and ESG1 and 2
Q2: I am sorry for confusion here. What I meant was OSPF neighborship among physical routers and ESGs
Q3: cleared thanks
Q4: I updated my diagram above that I am planning to use vlan 6 for VXLAN. Physical routers have VIP for vlan 6 with MTU 1600.
Hi,
#1
you can take a look at the design guide VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0 section 5.3.7.4 Edge VM Connectivity and Availability Design page 137 and VMworld Session on NSX Reference design here: VMworld 2017 US - NET1535BU - NSX Design—Reference Design for SDDC with NSX and vSphere: Part 1 - Yo... at 46:54
It covers how to design the VLAN to peer with physical routers which is exactly the same as you described, a disjoint VLAN for point to point peering between Edge to Physical Router.
So you are on the right track!
#2
Yes the VLAN is a way to create a point to point link between Edge and Physical Router but the Edge would still have route peering to both physical router and use the dynamic routing for failover as opposed to NIC teaming. The design guide and VMworld session would be helpful for you to understand this
#3 yes you are correct, see the design guide/VMworld session I mentioned earlier
Here's a figure taken from the design guide
#4
To physical core will be VLAN so technically MTU 1500 will work (unless you share that link with VXLAN, in that case minimum is MTU 1600).
If you see the NSX performance session, increasing the MTU i.e. to 9000 will actually help to increase the performance VMworld 2017 US - NET1343BU - NSX Performance Deep Dive - YouTube