how to assign different VTEP subnet for each rack from one cluster pool?

[hongjun]

From NSX design guide, to use L3 access design, it's recommended to use same VLAN number per rack but with different subnet

10.66.<rack id>.0

10.77.<rack id>.0

10.88.<rack id>.0

10.99.<rack id>.0

If customer has compute cluster span across racks, then we'll have different subnet for each rack VTEP, something like (assuming vlan 77 is for VTEP0

<rack 1> 10.77.1.1, 10.77.1.2, 10.77.1.3....

<rack 2> 10.77.2.1, 10.77.2.2, 10.77.2.3...

During host preparation, GUI will prompt to assign VTEP from DHCP/IP Pool, looks like most cases we use IP Pool, then for this compute cluster, how can we assign VTEP IP in different subnet from one single pool?

Thanks

Hongjun

[hs77]

Remember one VTEP IP pool can assign addresses to Multiple Compute Clusters if they are in same subnet and in different racks.

If Compute Clusters are in different subnets then you need to create VTEP IP Pools for each Compute Cluster.

So in your case you need to create VTEP IP pool for each Compute cluster in different racks.

[hongjun]

Thanks for the reply. What you advised is my understanding as well.

Then I still see no way to implement what's in Pg 71 of NSX-V design guide.

the requirement for the compute notes in the top picture shows

1) L3 Access design with leaf/spine

2) Each rack has different subnet but same vlan number for VTEP(and others including mgmt,...)

3) Each compute cluster(orange and green) will span across different racks(for HA protection of rack failure).

[hs77]

If i interpret your question you mean  how will different traffic types across layer 3 fabric/Racks will communicate.

On each TOR switch VLANs will be terminated as SVI VLAN and they will also serve as default gateway for there respective traffic.

on each host you have to create static Routing for Storage and vmotion Traffic. For Management traffic VMk0 Default gateway is already created on the host.

Storage network route – esxcli network ip route ipv4 add -n 10.66.0.0/16 -g 10.66.1.1

vSphere vMotion network route – esxcli network ip route ipv4 add -n 10.77.0.0/16 -g 10.77.1.1

see pg 80 or 81 of design guide

[hongjun]

Sorry that my question was not clear.

I understood how to design VLAN and routing for leaf/spine.

My only question is still around how to assign VTEP address in different subnet(because my compute cluster is spanning across racks) from single NSX pool.

For other networks like mgmt/vmotion, I can define manually different subnets for my rack spanning compute cluster but I can't do that with VTEP vmkernal assignment. I have to go with pool/DHCP, that's the reason I questioned how to do pg.71 for design guide in regard to VTEP address assignment from pool.

Any thoughts?

[hs77]

You are mistaken here.

The compute cluster is not spanning racks. Each compute cluster in a rack is in different subnet.

Regarding separation of Orange and Green clusters in rack is related to different NSX Domains.

Each NSX Manager can be associated with one 1 vcenter only.

what they have shown is 2 NSX Domains with 2 NSX Manager and 2 vCenters.Where each NSX Manager is registered with one vCenter only.

They have shown this in the diagram to show how to scale NSX because any maximum limits of vCenter gets applicable on NSX Manager also.

[hongjun]

Good point, hs77. I agree with your point but I think in the picture of p71, they also want to show compute cluster spanning across racks to prevent rack failure.

This is the quote directly under the picture.

"The compute, management and edge clusters are laid out based on the purpose they serve. Clusters can be sized up to the vSphere platform limits and span across racks, so that a switch or rack failure only impacts a percentage of the cluster capacity."

I looked back VMworld 2014 NET 1589 session done by Nimesh and Abhishek. It looks to me that they recommend DHCP but didn't give out any details. I'll need to look into that. You can see in their slide attached, they also show compute cluster spanning across racks.

[hs77]

If compute Clusters are spanning racks than in that case it is not leaf/spine design with L3 from Leaf to spine.

[hs77]

If you see point 2 in your image it clearly states:

If fabric is L3, this implies that separate IP subnets are associated with same VLAN ids defined across racks.

[hongjun]

yes and that also means VTEP IP will be in different subnets for each rack(with same vlan id) and IP Pool assignment will NOT work for VTEP vmknics.

It looks like I have to set up ip helper-address on each rack TOR switch for VTEP vlan 88 and point to my central DHCP server and on central DHCP server, I have to create different subnet zones to match different VTEP subnet for each rack and then handle out DHCP IPs to VTEP vmknics.

[hs77]

You can also create Multiple IP pools with 1 Pool for each Cluster in case it is in different subnet.

[hongjun]

i wish I could. right now GUI only allow ONE IP pool assigned to ONE cluster. I can't multi-select individual hosts inside a cluster to assign the pool in one subnet.

[hs77]

For each cluster you will find configure link under VXLAN column. NSX>Installation>Host Preparation

while configuring VXLAN you select option use IP Pool then select new IP Pool. So for clusters in different subnets select new IP Pool.

[RaymundoEC]

In my humble opinion, if you have this strange not leaf/spine deployment, in day of VCD+VXLAN, refering to VTEP's, you was able to edit VTEP ip address directly from DVS in vSphere Netwroking, so if this is the case I will go there and test, since this is not recommended action at least not see anywhere why not to try.

hote this helps.

+vRay

[rbudavari · Accepted solution]

You have two options:

1) DHCP using the helper address on each TOR as you suggest. This is the recommended approach

2) DHCP then manually editing the VMkernel interface IP address on each ESXi host to match the required subnet - this is not recommended as it isn't automated, but is still supported

[hongjun]

thanks for confirming. Neither of these two options are easy for customers to deploy different segments of VTEPs across racks IMO. Maybe NSX future version can have more flexibility for VTEP IP pool assignment to accommodate this scenario.