VMware Cloud Community
munozajj
Contributor
Contributor

VSAN: swapping out old vCenter (Server A) with new vCenter (Server B)

For the sake of argument, "what ifs?" and engineering-level assessments, I'm presenting this question hypothetically

...can it be done...and if so, how?

If you have an operational VSAN implementation, and for whatever reason, you have to swap out the vCenter server VM (vcenter001) that manages the current VSAN cluster, with a separate newly built vCenter Server VM (vcenter002), how would you go about swapping out the vCenter servers while also doing the following:

  • VMs in infrastructure don't experience downtime (unless they absolutely have to)
  • VSAN cluster does not have to be rebuilt (hard requirement)
  • Pre-existing VSAN cluster becomes managed by new vCenter Server (vcenter002)
  • VSAN datastore is kept intact (hard requirement)
  • swap is permanent...meaning that the swap can be left in place permanently without any future roll back needed (ideally...not a hard requirement)
  • A little downtime in the process, if needed, is understandable and not a hard requirement

Extra details regarding VSAN infrastructure:

  • vcenter001 is on same subnet as vcenter002
  • vmkernels and every other requirement is in place and setup properly
  • vCenter server Windows edition is installed on BOTH vcenter001 and vcenter002 using the manual individual method (not quick method)
  • both vCenters as well as the VSAN are functioning 100%
  • vcenter002 is running newer upgraded OS (2012 R2) (on VMware compatibility matrix)
  • both vCenter servers are running vSphere 5.5u1
  • both vCenters are VSAN licensed
  • SSO, web client, SQL Server 2012R2 SP1, vCenter server and inventory service are installed on single server
  • both vCenters are 100% operational...regardless of reason why vCenter needs to be swapped out

Going through the motions, I have tried this situation before with exact same environmental details and continued to get UUID mismatch errors when trying to move the VSAN ESXi hosts from vcenter001 to vcenter002. And to elaborate a little more to avoid people having to ask some of the core expected questions, I tried migrating the ESXi with maintenance mode, without putting into maintenance mode, disconnecting (vcenter001) then connecting to new vCenter (vcenter002), I've tried with old vCenter server still running or with it shutdown.

Attached is a generic diagram that I put together real quick in mspaint to help visualize the infrastructure layout. In advance, I'd like to thank all you who will take the time to try and answer my question.

diagram_vsanvcentermigration.png

Reply
0 Kudos
14 Replies
depping
Leadership
Leadership

I have done this many times in the past simply by blowing the old vCenter Server away, then creating a new Cluster and directly adding the hosts to this cluster. Took about 5 minutes with 4 hosts and no issues whatsoever?

munozajj
Contributor
Contributor

Thanks for you input...

So you never had any type of UUID mismatch when moving a disconnected/unmanaged ESXi host into a new VSAN cluster? Do you setup the VSAN cluster first or do you migrate ESXi hosts to non-VSAN enabled cluster and then enable VSAN after their are all in the cluster?

Also, do you run any cli commands on host, put into maintenance, disconnect from vCenter or anything along those lines so that the ESXi hosts can be put into the new VSAN cluster? When I have issues moving any of the ESXi hosts, the one and only command that ever let me join a new cluster was the [esxcli vsan cluster leave] command....I have yet to see any details of what that command actually does in the background....

Reply
0 Kudos
munozajj
Contributor
Contributor

Also, if you just blew away vCenter, did you at least export dVS configurations from the original vCenter? When you say just "blow away"...do you mean right-click delete, just shutdown, disjoin from domain, remove IP...etc...?

Reply
0 Kudos
depping
Leadership
Leadership

I migrated them in to a non-vsan enabled cluster and then enabled VSAN as far as I recall. But it has been a couple of months ago since I last did it.

Reply
0 Kudos
munozajj
Contributor
Contributor

I'll go through the motions and reply back with results. Thank you again.

Reply
0 Kudos
munozajj
Contributor
Contributor

Alright so I'm all setup to perform a test in one of the virtual environments but I just thought of something and wanted to confirm with you...when you migrate the hosts to the new vCenter cluster, VSAN is initially off....but when you turn on vCenter, do you use automatic disk management mode or manual? Does it make a difference?

Reply
0 Kudos
CHogan
VMware Employee
VMware Employee

Since the hosts are already participating in a VSAN cluster and have already claimed the disks, it should make no difference.

Manual/Automatic is only about claiming new, unused local disks on the host for the VSAN datastore.

http://cormachogan.com
Reply
0 Kudos
munozajj
Contributor
Contributor

Alright I just finished the test and everything went smoothly with no down time....so for those of you in the same position that I was, here are the steps that you would take:

*instructions below are for reference purposes only. I am not/will not be responsible for anyone playing around with a production environment and breaking things. I have tested these in my own environment and I cannot guarantee that your environment is the same and will have the same outcome. USE AT YOUR OWN RISK! YOU HAVE BEEN WARNED!

*VERY IMPORTANT!!!....if you plan on swapping out vCenters or just migrating ESXi hosts to a new vCenter, whatever you do, do not enable the VSAN cluster at the new vCenter before you start these instructions below....REMEMBER, if you have multiple VSAN clusters, you must have IGMP snooping enabled and you must setup the IGMP groupings so that each VSAN cluster has it's own multcast address!!! If you don't follow this rule, your pre-existing VSAN could have conflicts in the multicast network and your VSAN cluster could start to have network connectivity issues. Unless you have setup IGMP snooping and segmented your multicast network, DO NOT have TWO VSAN clusters enabled at the same time....again, you have been warned!


1. Either power down your vCenter server (old) OR disconnected ESXi hosts from vCenter

*if you power down your vCenter server and do not plan on bring it back up, you do not have to perform step #2


2. Now remove the ESXi hosts from your vCenter server

3. The next thing you will do is go to your new vCenter server and add the pre-existing VSAN hosts to your new vCenter. This is the part I could not get past. The trick is to add them to your datacenter and NOT your cluster. If you try to add them to a cluster, you will get errors about the cluster not being a VSAN enabled cluster.


4. Once you have added the ESXi hosts to your datacenter, you will then enable your cluster with VSAN.


5. Now add the ESXi hosts to your VSAN enabled cluster.


At this point, your ESXi hosts will start to communicate with each other and your warnings should start to disappear. At this point, your VSAN cluster should now be managed by the new vCenter server.


Thanks to all of those on this post that provided responses to my questions.

Reply
0 Kudos
lamw
Community Manager
Community Manager

This is actually an operation I've done several times, as mentioned by others it is very straight forward. Looking at your steps, many of them are actually not needed.

1. Create new vSphere Cluster with VSAN enabled on the new vCenter Server (there's no reason you need to enable this afterwards)

2. You could power down the old vCenter Server or disconnect the ESXi hosts, but you could actually skip both of these steps and move to adding the ESXi hosts into the new VSAN Cluster

3. The key here is to add one ESXi host to the new VSAN Cluster. If you try to add all of the hosts at once, say using a script, you will get the UUID mismatch.

4. Once the first ESXi host has been added to the new VSAN Cluster you can then add the remainder (either one by one or through bulk operation)

You'll see a "misconfiguration detected" warning until all ESXi hosts have been moved over from the old vCenter Server to the new one.

I've actually seen this get asked on several occasions internally from customers and figured it would be good to write about the topic, so I even tested this in my lab and blogged about it here http://www.virtuallyghetto.com/2014/09/how-to-move-a-vsan-cluster-from-one-vcenter-server-to-another...

Reply
0 Kudos
jesse_gardner
Enthusiast
Enthusiast

For what its worth, I'll chime in on my recent experience.  Moving a VSAN-enabled 3-host vSphere 6 cluster connected only to a distributed virtual switch for all VM and host traffic, no standard switches.

  1. Use the WebClient to export dVS settings to file.
  2. Import dVS settings into new vCenter.  I chose "Yes" to a question along the lines of maintaining internal identifiers.  Not sure if this cause a step further down...
  3. Create empty cluster.  DRS-enabled, but not yet VSAN or HA.  Did this based on some other feedback in this discussion, but may not be strictly necessary in this manner.
  4. Add one host.  Basically rips it out of the old vCenter and into the new one.
  5. Enable VSAN.  Again, not sure if this could've been done earlier.
  6. Add 2nd and 3rd hosts.
  7. The hosts all have a yellow warning about an invalid proxy dVS switch connection.  To resolve, add the hosts to the dVS, and walk through the wizard performing all the appropriate migrations.  I apparently did everything right, because nothing lost a ping.
  8. Enable HA on the cluster.
  9. I noticed in the vSphere (Fat C#) Client at this point, on each hosts' Summary tab, the Networking pane was empty.  This bugged me.  I resolved by restarting vCenter itself.

In the end, no disruption to VMs, as advertised.

Reply
0 Kudos
ramakrishnak
VMware Employee
VMware Employee

to add to what others have already covered in detail

i do this on almost every major build in my internal test environment Smiley Happy and it holds up fine, only thing is to make sure custom vCenter settings if any are re-added back to newer vCenter

there are few additional things outside of what already been mentioned by folks here

Its more of my observations and best practices rather than anything else...

a. Before migration, collect/record the VM policy settings. and apply them after successful vCenter Swapping

( there should be some scripts to do this today. but not entirely automated/UI driven mechanism exists today...)

I use vsan.recover_spbm rvc cmd

b. also record/save any specific vCenter dependent/specific configurations so you can reapply them on newer vCenter

c. Shutdown the Old vCenter server

d. Always add hosts to root of the datastore on the newer vCenter. Then move them over to newer vsan cluster


e. Check All VSAN Storage providers are registered  and Online. ( Vcenter host -> Manage -> Storage Providers section)

( Currently there is an issue out here that the first moved in host will not have Storage provider registered).

You need to click on the toolbar "Synchronizes all Virtual SAN Storage providers with the current state of the environment".

f. use vsan.recover_spbm rvc command and re-create the vsan policies and apply them to the respective VMs.

g. if there were any VM templates before, you will have to re-add them to the new vCenter

Thanks,

Reply
0 Kudos
terrible_towel
Contributor
Contributor

Lots of great info on this thread..   How about going from a 5.5 vsan cluster (Windows VC and ESXi hosts are 5.5U2) to a new 6.0 Vcenter (VCSA) ?

And then doing the ESXi 5.5 to 6 upgrade after all hosts are on the 6.0 VCSA ?

Thanks,

Paul

Reply
0 Kudos
Paul_Sheard
Enthusiast
Enthusiast

sorry to jump on to the back of this thread, but I'm wondering if some of these steps taken in this fix will fix my problem on a new VSAN 4 host cluster..

as it's a green field site I had to use the boot strapping method used here  : https://www.vmware.com/files/pdf/products/vsan/VMware-TechNote-Bootstrapping-VSAN-without-vCenter.pd...

It's worth noting that when I got to the point of adding all my disks using this document (4 HDD & 1 SSD) I got an error saying I'm adding "too many disks" ?

But anyway once I've bootstrapped the 4 nodes, then built the PSC and VCSA, login to web UI, create DC, Create Cluster, enable VSAN, add 4 hosts.. I have this error on each node..

"host is in a virtual san cluster but does not have virtual san service enabled"

I cannot find any fix on the VMware KB's / forums.. and the health check for the VSAN is all green..

Should I try creating a new Cluster on this same vCenter and move the hosts to this new cluster?

Any idea's much appreciated folks.

Paul Sheard VMware Consultant (Contract) VCP6 DCV NV CMA DTM
Reply
0 Kudos
Paul_Sheard
Enthusiast
Enthusiast

I should have tried this before posting..

So what I basically did was, create a new cluster (enabled VSAN).

Went to the cluster where the hosts were participating and showing the error "host is in a virtual san cluster but does not have virtual san service enabled".

Then disconnected each host, then removed each host.

Then I added each host one by one to the new cluster I created with VSAN enabled..


I now just have an error on each host where I see my system logs are not on persistent storage..  I recollect the VSAN datastore is not a place to put my system logs, so I'll look at putting a syslog server in place.. any suggestions for a good syslog server (free) ?


Thanks

Paul.



Paul Sheard VMware Consultant (Contract) VCP6 DCV NV CMA DTM
Reply
0 Kudos