For the sake of argument, "what ifs?" and engineering-level assessments, I'm presenting this question hypothetically
...can it be done...and if so, how?
If you have an operational VSAN implementation, and for whatever reason, you have to swap out the vCenter server VM (vcenter001) that manages the current VSAN cluster, with a separate newly built vCenter Server VM (vcenter002), how would you go about swapping out the vCenter servers while also doing the following:
Extra details regarding VSAN infrastructure:
Going through the motions, I have tried this situation before with exact same environmental details and continued to get UUID mismatch errors when trying to move the VSAN ESXi hosts from vcenter001 to vcenter002. And to elaborate a little more to avoid people having to ask some of the core expected questions, I tried migrating the ESXi with maintenance mode, without putting into maintenance mode, disconnecting (vcenter001) then connecting to new vCenter (vcenter002), I've tried with old vCenter server still running or with it shutdown.
Attached is a generic diagram that I put together real quick in mspaint to help visualize the infrastructure layout. In advance, I'd like to thank all you who will take the time to try and answer my question.
I have done this many times in the past simply by blowing the old vCenter Server away, then creating a new Cluster and directly adding the hosts to this cluster. Took about 5 minutes with 4 hosts and no issues whatsoever?
Thanks for you input...
So you never had any type of UUID mismatch when moving a disconnected/unmanaged ESXi host into a new VSAN cluster? Do you setup the VSAN cluster first or do you migrate ESXi hosts to non-VSAN enabled cluster and then enable VSAN after their are all in the cluster?
Also, do you run any cli commands on host, put into maintenance, disconnect from vCenter or anything along those lines so that the ESXi hosts can be put into the new VSAN cluster? When I have issues moving any of the ESXi hosts, the one and only command that ever let me join a new cluster was the [esxcli vsan cluster leave] command....I have yet to see any details of what that command actually does in the background....
Also, if you just blew away vCenter, did you at least export dVS configurations from the original vCenter? When you say just "blow away"...do you mean right-click delete, just shutdown, disjoin from domain, remove IP...etc...?
I migrated them in to a non-vsan enabled cluster and then enabled VSAN as far as I recall. But it has been a couple of months ago since I last did it.
I'll go through the motions and reply back with results. Thank you again.
Alright so I'm all setup to perform a test in one of the virtual environments but I just thought of something and wanted to confirm with you...when you migrate the hosts to the new vCenter cluster, VSAN is initially off....but when you turn on vCenter, do you use automatic disk management mode or manual? Does it make a difference?
Since the hosts are already participating in a VSAN cluster and have already claimed the disks, it should make no difference.
Manual/Automatic is only about claiming new, unused local disks on the host for the VSAN datastore.
Alright I just finished the test and everything went smoothly with no down time....so for those of you in the same position that I was, here are the steps that you would take:
*instructions below are for reference purposes only. I am not/will not be responsible for anyone playing around with a production environment and breaking things. I have tested these in my own environment and I cannot guarantee that your environment is the same and will have the same outcome. USE AT YOUR OWN RISK! YOU HAVE BEEN WARNED!
*VERY IMPORTANT!!!....if you plan on swapping out vCenters or just migrating ESXi hosts to a new vCenter, whatever you do, do not enable the VSAN cluster at the new vCenter before you start these instructions below....REMEMBER, if you have multiple VSAN clusters, you must have IGMP snooping enabled and you must setup the IGMP groupings so that each VSAN cluster has it's own multcast address!!! If you don't follow this rule, your pre-existing VSAN could have conflicts in the multicast network and your VSAN cluster could start to have network connectivity issues. Unless you have setup IGMP snooping and segmented your multicast network, DO NOT have TWO VSAN clusters enabled at the same time....again, you have been warned!
1. Either power down your vCenter server (old) OR disconnected ESXi hosts from vCenter
*if you power down your vCenter server and do not plan on bring it back up, you do not have to perform step #2
2. Now remove the ESXi hosts from your vCenter server
3. The next thing you will do is go to your new vCenter server and add the pre-existing VSAN hosts to your new vCenter. This is the part I could not get past. The trick is to add them to your datacenter and NOT your cluster. If you try to add them to a cluster, you will get errors about the cluster not being a VSAN enabled cluster.
4. Once you have added the ESXi hosts to your datacenter, you will then enable your cluster with VSAN.
5. Now add the ESXi hosts to your VSAN enabled cluster.
At this point, your ESXi hosts will start to communicate with each other and your warnings should start to disappear. At this point, your VSAN cluster should now be managed by the new vCenter server.
Thanks to all of those on this post that provided responses to my questions.
This is actually an operation I've done several times, as mentioned by others it is very straight forward. Looking at your steps, many of them are actually not needed.
1. Create new vSphere Cluster with VSAN enabled on the new vCenter Server (there's no reason you need to enable this afterwards)
2. You could power down the old vCenter Server or disconnect the ESXi hosts, but you could actually skip both of these steps and move to adding the ESXi hosts into the new VSAN Cluster
3. The key here is to add one ESXi host to the new VSAN Cluster. If you try to add all of the hosts at once, say using a script, you will get the UUID mismatch.
4. Once the first ESXi host has been added to the new VSAN Cluster you can then add the remainder (either one by one or through bulk operation)
You'll see a "misconfiguration detected" warning until all ESXi hosts have been moved over from the old vCenter Server to the new one.
I've actually seen this get asked on several occasions internally from customers and figured it would be good to write about the topic, so I even tested this in my lab and blogged about it here http://www.virtuallyghetto.com/2014/09/how-to-move-a-vsan-cluster-from-one-vcenter-server-to-another...
For what its worth, I'll chime in on my recent experience. Moving a VSAN-enabled 3-host vSphere 6 cluster connected only to a distributed virtual switch for all VM and host traffic, no standard switches.
In the end, no disruption to VMs, as advertised.
to add to what others have already covered in detail
i do this on almost every major build in my internal test environment and it holds up fine, only thing is to make sure custom vCenter settings if any are re-added back to newer vCenter
there are few additional things outside of what already been mentioned by folks here
Its more of my observations and best practices rather than anything else...
a. Before migration, collect/record the VM policy settings. and apply them after successful vCenter Swapping
( there should be some scripts to do this today. but not entirely automated/UI driven mechanism exists today...)
I use vsan.recover_spbm rvc cmd
b. also record/save any specific vCenter dependent/specific configurations so you can reapply them on newer vCenter
c. Shutdown the Old vCenter server
d. Always add hosts to root of the datastore on the newer vCenter. Then move them over to newer vsan cluster
e. Check All VSAN Storage providers are registered and Online. ( Vcenter host -> Manage -> Storage Providers section)
( Currently there is an issue out here that the first moved in host will not have Storage provider registered).
You need to click on the toolbar "Synchronizes all Virtual SAN Storage providers with the current state of the environment".
f. use vsan.recover_spbm rvc command and re-create the vsan policies and apply them to the respective VMs.
g. if there were any VM templates before, you will have to re-add them to the new vCenter
Lots of great info on this thread.. How about going from a 5.5 vsan cluster (Windows VC and ESXi hosts are 5.5U2) to a new 6.0 Vcenter (VCSA) ?
And then doing the ESXi 5.5 to 6 upgrade after all hosts are on the 6.0 VCSA ?
sorry to jump on to the back of this thread, but I'm wondering if some of these steps taken in this fix will fix my problem on a new VSAN 4 host cluster..
as it's a green field site I had to use the boot strapping method used here : https://www.vmware.com/files/pdf/products/vsan/VMware-TechNote-Bootstrapping-VSAN-without-vCenter.pd...
It's worth noting that when I got to the point of adding all my disks using this document (4 HDD & 1 SSD) I got an error saying I'm adding "too many disks" ?
But anyway once I've bootstrapped the 4 nodes, then built the PSC and VCSA, login to web UI, create DC, Create Cluster, enable VSAN, add 4 hosts.. I have this error on each node..
"host is in a virtual san cluster but does not have virtual san service enabled"
I cannot find any fix on the VMware KB's / forums.. and the health check for the VSAN is all green..
Should I try creating a new Cluster on this same vCenter and move the hosts to this new cluster?
Any idea's much appreciated folks.
I should have tried this before posting..
So what I basically did was, create a new cluster (enabled VSAN).
Went to the cluster where the hosts were participating and showing the error "host is in a virtual san cluster but does not have virtual san service enabled".
Then disconnected each host, then removed each host.
Then I added each host one by one to the new cluster I created with VSAN enabled..
I now just have an error on each host where I see my system logs are not on persistent storage.. I recollect the VSAN datastore is not a place to put my system logs, so I'll look at putting a syslog server in place.. any suggestions for a good syslog server (free) ?