Seeking help about NSX design for my whole corporate network

[mhdganji]

Hi every one

we have 3 data centers or better to say server farms in different geo locations. A main center including many servers and the WAN and Internet central point (Traffic from there goes to and comes from Internet). The two other centers also include some servers  serving the local site, other sites and also play the role of backup and disaster recovery servers in case of a disaster in main site. Those two sites also have WAN (Intranet) and wireless connections to the main site.

I'm designing and implementing NSX for the whole infrastructure to the extent possible. Now I have some questions which seek and appreciate the answers and any other consideration you may have

I have a couple of VLAN's which is good If I can extend and have them in different sites. (We have some old Novel apps which work in IPX so it's desirable to have them extended in all sites so they can be in a VLAN/VXLAN L2 domain and interact with each other.

I am not interested in having separate vcenters and nsx managers in different sites. I just may deploy another fail-over vcenter appliance in second site.

1- Should I install Universal DLR or not? If yes, one (or two) for each data center or just one (or two) for the whole infrastructure

2- If universal DLR is not needed and I just should have DLR's, Should I have instances (one or two) for each data center or one (or two) for all data centers? I'm asking this because when I install DLR it askes to choose a data center. Let's say again that I have VM's in different sites which I like to be in the same VXLAN if possible

3- Each site has east-west traffic in its data center, traffic from local VM's to VM's in other sites which the latter normally goes to physical network (core switches and routers ) then over wireless and WAN links to the other site. Do I need edge service gateways to accomplish this or DLR's are enough?

4- In the main site we have connections to Internet so we need one or more DMZ area. Is it a good choice to have two ESG's to separate these?

5- What transport zones I need? one for the whole datacenters including DMZ clusters, one for all datacenters except DMZ and one including DMZ's? or better decisions can be made?

6- Some sources say install controllers in management cluster! Is this right? I have management cluster including my main vcenter server and NSX managers but controllers are installed in operational data center.

Sorry for long post and thanks in advance for any help

[Sreec]

First and foremost please do check Latency supported values . If you are exceeding the values , considering geo-setup  ,Cross-VC-NSX will not be possible .

Can you change MTU over Wan if the requirement is End-End VXLAN ? You could still do a L2 VPN but bandwidth will be limited and may not be applicable for all use cases .

You said , second and third site will be a backup and DR sites.  So what kind of pairing this is going to be from primary Site keeping NSX aside . Shared Recovery Site ? Single Site Protected via two Recovery Sites ? 

1- Should I install Universal DLR or not? If yes, one (or two) for each data center or just one (or two) for the whole infrastructure

Like i mentioned above, UDLR deployment may not be a good approach if you have MTU/Latency limitations.  Ideally Single UDLR serve the purpose , but you can't do a direct bridging if there is a use case.

2- If universal DLR is not needed and I just should have DLR's, Should I have instances (one or two) for each data center or one (or two) for all data centers? I'm asking this because when I install DLR it askes to choose a data center. Let's say again that I have VM's in different sites which I like to be in the same VXLAN if possible

Same VXLAN Network request demands L2 to be stretched across the sites . Since VNI-ID is least significant bit , we could still have Site specific DLR and use Second/Third Site only when DR situation demands.

3- Each site has east-west traffic in its data center, traffic from local VM's to VM's in other sites which the latter normally goes to physical network (core switches and routers ) then over wireless and WAN links to the other site. Do I need edge service gateways to accomplish this or DLR's are enough?

Technically we can peer DLR with upstream routers, But Edges are best candidate since they are perimeter specific virtual routers and you should be aware of all supported/unsupported topologies  -> Section 5.3.7 NSX Edge Deployment Considerations in https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmw-nsx-network-virtu...

4- In the main site we have connections to Internet so we need one or more DMZ area. Is it a good choice to have two ESG's to separate these?

This depends upon the feasibility to stretch VXLAN between DC and DMZ. Ideally i wont do that unless there are some shared services . Dedicated ESG  with unique Transit VLAN for DMZ with dedicated Transport Zone is a good approach .

5- What transport zones I need? one for the whole datacenters including DMZ clusters, one for all datacenters except DMZ and one including DMZ's? or better decisions can be made?

Answered -4

6- Some sources say install controllers in management cluster! Is this right? I have management cluster including my main vcenter server and NSX managers but controllers are installed in operational data center.

Considering your setup , dedicated management cluster is very much needed with minimum 4 Host and controllers can be deployed there.

[mhdganji]

You said , second and third site will be a backup and DR sites.  So what kind of pairing this is going to be from primary Site keeping NSX aside . Shared Recovery Site ? Single Site Protected via two Recovery Sites ?

Second and third sites have their own virtual machines serving users there but also are options for disaster recovery and backup (at least the second site is). Consider it this way (This is very similar to our conditions)

Site 1 has some DB, App and file servers and site 2 has some specific VM's too. Backup of VM's in site 1 is stored in site 2 every night and vice versa. We like to be able to have L2 connection between some VM's if possible. Also it is desired to manage all these infrastructure using same DLR's, Edges, ...

We have just one vCenter and I'm not interested in having more separate ones. So I think Cross-vCenter is not my issue. Just I have two sites and I like to use one nsx manager and configuration for the whole infrastructure.

1- Should I install Universal DLR or not? If yes, one (or two) for each data center or just one (or two) for the whole infrastructure

Like i mentioned above, UDLR deployment may not be a good approach if you have MTU/Latency limitations.  Ideally Single UDLR serve the purpose , but you can't do a direct bridging if there is a use case.

As I said:

"We have just one vCenter and I'm not interested in having more separate ones. So I think Cross-vCenter is not my issue. Just I have two sites and I like to use one nsx manager and configuration for the whole infrastructure."

So do we need Universal DLR in this case?

2- If universal DLR is not needed and I just should have DLR's, Should I have instances (one or two) for each data center or one (or two) for all data centers? I'm asking this because when I install DLR it askes to choose a data center. Let's say again that I have VM's in different sites which I like to be in the same VXLAN if possible

Same VXLAN Network request demands L2 to be stretched across the sites . Since VNI-ID is least significant bit , we could still have Site specific DLR and use Second/Third Site only when DR situation demands.

Sorry but what do you mean same VXLAN's demand L2 to be stretched? Cause I thought that using VXLAN and NSX I can make this possible. I have some VM's in both sites which should be in the same VLAN or VXLAN (L2 domain). Is this possible using a DLR and create logical switches and connect both VM's to that logical switch? Or it needs UDLR? or maybe different DLR's in each site?

3- Each site has east-west traffic in its data center, traffic from local VM's to VM's in other sites which the latter normally goes to physical network (core switches and routers ) then over wireless and WAN links to the other site. Do I need edge service gateways to accomplish this or DLR's are enough?

Technically we can peer DLR with upstream routers, But Edges are best candidate since they are perimeter specific virtual routers and you should be aware of all supported/unsupported topologies  -> Section 5.3.7 NSX Edge Deployment Considerations in https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmw-nsx-network-virtu...

Hmm. Thanks. So you say it's better to connect DLR's to edges and then edges to upstream physical uplinks. Seems good but on the other side there are VM's that need connectivity to upstream without any firewall rule or limitation or .. I was thinking maybe it's a good idea to send them directly to physical uplinks.

4- In the main site we have connections to Internet so we need one or more DMZ area. Is it a good choice to have two ESG's to separate these?

This depends upon the feasibility to stretch VXLAN between DC and DMZ. Ideally i wont do that unless there are some shared services . Dedicated ESG  with unique Transit VLAN for DMZ with dedicated Transport Zone is a good approach .

Would you please explain a little bit more? VXLAN is anyway possible cause there are all VM's and can be connected to each other (from DMZ to DC, using same logical switches or edge uplinks). I have another DMZ zone which is physical and above all these but here I'm talking about the DMZ implemented in NSX edges.

And you mean one DMZ is enough  by your second sentence?

5- What transport zones I need? one for the whole datacenters including DMZ clusters, one for all datacenters except DMZ and one including DMZ's? or better decisions can be made?

Answered -4

What about VM's which are connected to Edge via transit networks? They  shouldn't be  in a shared transport zone?

6- Some sources say install controllers in management cluster! Is this right? I have management cluster including my main vcenter server and NSX managers but controllers are installed in the operational data center.

Considering your setup, dedicated management cluster is very much needed with minimum 4 Host and controllers can be deployed there.

I have 4 hosts in cluster management but as a matter of fact I have used a complete separate cluster and a dedicated vCenter for management cluster so this cluster cannot be seen in production cluster. 4 hosts are grouped in a cluster added to management vCenter and vCenter of production cluster plus nsx manager is there but controllers are placed in production cluster. You think it's better to move controllers to the management cluster vCenter?

Thanks so much for your time in advance

[Sreec · Accepted solution]

As I said:

"We have just one vCenter and I'm not interested in having more separate ones. So I think Cross-vCenter is not my issue. Just I have two sites and I like to use one nsx manager and configuration for the whole infrastructure."

So do we need Universal DLR in this case?

Since you have already scope that setup is a single VC . There is no universal concept anymore, all objects will remain local to the sites(In your case single VC with multiple  compute clusters from different sites) . So it would be Distributed logical router.

Sorry but what do you mean same VXLAN's demand L2 to be stretched? Cause I thought that using VXLAN and NSX I can make this possible. I have some VM's in both sites which should be in the same VLAN or VXLAN (L2 domain). Is this possible using a DLR and create logical switches and connect both VM's to that logical switch? Or it needs UDLR? or maybe different DLR's in each site?

It is certainly possible , I was referring to VNI-ID not actual workload subnets. Sorry for the confusion.  So eventual output would be something like

->VXLAN 6000 Connected to SiteA- VM ,  SiteB-VM connected to a common DLR -  Scenario - A

-> VXLAN 60001 Connected to SiteA- VM ->DLR->Site-A Edge , SiteB- VM connected to VXLAN 6002-->DLR->Site-B Edge - Scenario- B

If above tenants needs routing between them, advertise it from respective Edges to upstream routers.

Hmm. Thanks. So you say it's better to connect DLR's to edges and then edges to upstream physical uplinks. Seems good but on the other side there are VM's that need connectivity to upstream without any firewall rule or limitation or .. I was thinking maybe it's a good idea to send them directly to physical uplinks.

Yes that is the preferred way . You can disable firewall if you don't want or if ECMP is preferred for Edges you have no options other than disabling it.

What about VM's which are connected to Edge via transit networks? They  shouldn't be  in a shared transport zone?

Yes, you can define compute&edge cluster under a single transport zone and that would satisfy the above ask.

I have 4 hosts in cluster management but as a matter of fact I have used a complete separate cluster and a dedicated vCenter for management cluster so this cluster cannot be seen in production cluster. 4 hosts are grouped in a cluster added to management vCenter and vCenter of production cluster plus nsx manager is there but controllers are placed in production cluster. You think it's better to move controllers to the management cluster vCenter?

NSX controllers should be deployed in the same vCenter where vCenter integration with NSX is done. In your case integration is done with VC-A and deployment is done on VC-B ,is that correct ? If that is the case you need to correct it. However if you are planning for micro-segmention for management cluster and not looking for any other feature. You can go with your approach of having no controllers of Management Cluster- VC and have it deployed on other cluster which is registered with another set of NSX&VC for leveraging  routing/switching.

[mhdganji]

Thanks Sreec

I almost got all my answers and will proceed and use your help when needed again.

I just may connect the uplink of DLR directly to upstream network cause I don't need any security feature there. VM's in DMZ or any zone needing security would be reachable via NSX edges connecting to upstream.

I think this way suits us and reduces complexity.

Regards,