NSX design with multiple vCenters

[MattG]

I am looking to have multiple PODs each managed by their own vCenter.  NSX will be used for the VM networking on all hosts (across vCenters).

VXLAN cannot be used between vCenters.   Does NSX have the same boundary?  Could separate VXLANs in separate vCenters leverage the same NSX logical networks?   Understanding that the physical traffic between the same logical network in different vCenters would need to traverse an Edge GW (right?).

I would like to be able to configure multiple PODs (managed by their own vCenter) and migrate VMs between PODs.   Since vCenter does not support vMotion between vCenters,  the plan was to use a swing LUN and cold migrate the VM by removing and adding to inventory.   Want to understand how NSX networking handles the network on the other side?   If NSX allows the logical network to traverse vCenters then I may not need to do anything.   Otherwise I may need to re-ip my VM.

What is the best practice with large cloud designs that force multiple vCenters to be used (due to host/VM maximums) and VM mobility between PODs?

Thanks,

-MattG

[admin]

Hi Matt,

NSX for vSphere currently has 1:1 relationship, one NSX Manager (managing its VXLAN domain) to one vCenter Server.

There is no live VM mobility between VC domains. You should be able to move VMs via cold migration, as you've described. This will also require disconnecting them from their original VXLAN or VLAN dvPg, and connecting to a new one at a different VC/NSX.

In case you would like to move a VM and keep its IP address *and* its gateway, the only way to do it is to use VLAN-backed dvPg for the VM, and make that VLAN available to all your VCs.

Would you be able to expand a little bit on what are you trying to achieve by inter-VC VM mobility?

[MattG]

Greenfield VCAC + NSX environment. Pods support up to 10k VMs. So looking to have one vCenter per Pod. Want be able to migrate VMs between Pods. Since VMs will use NSX created logical networks, need to understand how NSX created logical networks can talk to each other if they are managed separately. Cannot I not have the same NSX network across all pods?

[admin]

None of NSX-v services, including Logical Switches, can extend past one vCenter at this point.

I'm curious about the need for the ability to migrate individual VMs between PODs.

If it's not about individual VMs, but instead about multi-machine applications, this is where you may be able to do something interesting.

You could treat a multi-machine app instance as a "bubble" that:

- has internal L2 networks, which could be Logical Switches, to which VMs are connected

- has an Edge gateway which on one end connects to these internal networks, and on the other - to the rest of the environment

- uses private IP space (that can be reused many times), with Edge performing NAT in/out of the bubble

- can be made "mobile", if you include another edge that can advertise small unique subnet used for NAT via a dynamic routing protocol to the rest of the infrastructure

That way "bubble" is portable between NSX instances - you just:

- execute a set of API calls on a destination VC/NSX to create network services (logical switches, routing, DFW, LB, etc), then

- cold-migrate VMs to that new VC and connect them to the new network (keeping all IP config the same), then

- tear down NSX constructs in the source NSX, which stops advertising your app's "visible" IP space, then

- start advertising it from the NSX Edge running on destination VC.

[MattG]

Since client is viewing DC with multiple PODs as one big pool they would like the ability in the future to divide up the PODs into different tiers.   If 5 PODs existed today then vCAC Blueprints could be mapped to different vCenter via Reservations.   However,  new PODs will only be populated as existing ones fill up.

For example,  if they have enough capacity for 5 PODs (5 vCenters) and they now want POD 1 to be dedicated to Tenant1 or Dev.   How do they migrate all Tenant1 VMs to POD2 and all existing Dev VMs to POD4?

Based on your answer since NSX and its logical networks are only consumable by a single instance of vCenter,  while VMs could be manually migrated to another vCenter/NSX instance,  it would require RE-IP of the VM and the traffic would need to traverse the Edge.  Though if 5 PODs exist and they need to communicate with each other,  they would need to traverse an Edge anyway, right?

Thanks,

-MattG

[admin]

Though if 5 PODs exist and they need to communicate with each other,  they would need to traverse an Edge anyway, right?

I don't know vCAC well enough, but my guess is that it would depend on your blueprints - single-machine blueprints may be using an External Network Profile (thus not having an Edge between the VC reservation and the "rest of the world"), while multi-machine blueprints could use Routed, NATed, and Private network profiles, which (first two, at least) would use a Routed Gateway (which would be your Edge, separate one for each VC Reservation, if I'm not mistaken).

[MattG]

With a vSphere NSX Cluster for NSX components being managed by a Management vCenter,  can NSX be applied to hosts/VMs in another vCenter (Resource) that contains the resource hosts and VMs...since NSX maps 1:1 to vCenter?

Thanks,

-MattG

[admin]

You can run NSX Manager in a cluster not managed by the vCenter server that this NSX Manager is connected to.

So if you have a VC for management cluster and another VC for workloads, you can run NSX Manager in the management cluster, but connect it to the VC in the workload one.

The only thing to keep in mind is NSX Manager will use the VC it's connected to when deploying its components, such as Controllers, DLR Control VMs, Edge Services Gateways, and Service VMs. This means that you won't be able to deploy these to the hosts in your management cluster, and they will all have to go to the workload cluster instead.

[MattG]

Dmitri,

Thanks for getting back to me on this!  

So based on your statement "NSX Manager will use the VC it's connected to when deploying its components" you are saying NSX will be mated with a vCenter not by its location,  but during install it will prompt which vCenter to use?

So the only way to isolate all management components to a management cluster and resources only to a resource cluster would be to use a single vCenter?   Based on this fact I am assuming most NSX/vCloud environments use only one vCenter and separate by cluster only?

Thanks,

-MattG

[admin]

NSX will be mated with a vCenter not by its location,  but during install it will prompt which vCenter to use?

NSX Manager is explicitly linked to a VC, using a set of VC admin credentials. This is done after OVF deployment, via NSX Manager's built-in web UI.

Based on this fact I am assuming most NSX/vCloud environments use only one vCenter and separate by cluster only?

Correct, single VC and multiple clusters. Large NSX environment would include separate clusters for Management (VC, NSX Manager, Controllers); Edge (Edge Gateways, DLR Control VMs); and Compute (workloads). In smaller environments these can be collapsed to two or even one.

Here are a couple of public design guides for NSX that cover some of the considerations:

http://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

http://www.vmware.com/files/pdf/products/nsx/vmware-nsx-on-cisco-n7kucs-design-guide.pdf

(These are links from the NSX Resources page: VMware NSX | Network Virtualization | United States)