Proven Practice: Splitting a VirtualCenter Server Installation

Version 1

    Metadata

     

    VPP Metadata

    Description

    Title

    Proven Practice: Splitting a VirtualCenter Server Installation

    Version

    Rahul Ravulur and Desmond Chan / VMware 25/August/2008 V1

    Legal

    Copyright (c) 2008 VMware, Inc. All rights reserved

    Tags

    esx_server virtualcenter backup database restore sql_server roll_back

    Author

    VMware (NYSE: VMW) is the global leader in virtualization solutions from the desktop to the data center. Customers of all sizes rely on VMware to reduce capital and operating expenses, ensure business continuity, strengthen security and go green. With 2007 revenues of $1.33 billion, more than 100,000 customers and nearly 14,000 partners, VMware is one of the fastest-growing public software companies. VMware is headquartered in Palo Alto, California, and on the Web at http://vmware.com/.

     

    Rahul Ravulur rravulur@vmware.com

    Desmond Chan dchan@vmware.com

    Context

     

    A centralized model to manage VMware Infrastructure may not be suitable for some VI customers when they start growing into more datacenters. These VI customers may consider splitting the VirtualCenter database into two or more datacenters. This document provides the guidelines on how to accomplish such split.

    Actors

    VMware Infrastructure Administrators

    References

    See document below.

    Outline

    This brief document provides guidelines on how to split up the VirtualCenter database during an upgrade from VC 1 to VC 2.

     

    How to use this proven practice

    This brief document should be read by VMware Infrastructure Administrators who are interested in splitting up the VirtualCenter database into two or more datacenters.

     

    Table of Content

    1 Introduction

    2 Preparing for the split

    2.1 Prepare the server(s) for the new VirtualCenter (VC 2) and new SQL Server installation

    2.2 Split up the inventory in the existing VirtualCenter (VC 1)installation into two or more datacenters

    3 Performing the split

    3.1 Back up the existing VirtualCenter (VC 1) database

    3.2 Stop the VC 1Service and take a backup of the database

    3.3 Restore data into the database used by VC 2

    3.4 Install the second VirtualCenter server

    3.5 Bring up the first VirtualCenter server (VC 1) and delete the moved hosts being managed by VC 2

    3.6 Bring everything back on- line

    4 Risks

    5 Rolling back

     

     

    1 Introduction

    This document describes the process for splitting a single VirtualCenter installation (and database) into two or more separate installations.

    2 Preparing for the split

    The process for splitting involves cloning the database and VirtualCenter installation, and deleting the unwanted entries from each side. This allows the split to take place without incurring any data loss.

     

    To make the task of migration easy, the document assumes the VI administrator has organized the hosts to be migrated in a separate datacenter. Doing so will make the migration a seamless process. We strongly recommend using the datacenter as a unit to be moved.

    2.1 Prepare the server(s) for the new VirtualCenter (VC 2) and new SQL Server installation

    This step can (should) be performed well before actually starting on the split procedure

     

    1. Set up the host(s) that will run the new VirtualCenter and new SQL Server installation.

      Don't install VirtualCenter on this server yet, though.

    2.2 Split up the inventory in the existing VirtualCenter (VC 1)installation into two or more datacenters

     

    1. Create datacenters and organize the hosts and resource pools to match the intended distribution after the split.

    3 Performing the split

    When the big day rolls around:

    3.1 Back up the existing VirtualCenter (VC 1) database

    This set of steps should be performed a short while before you start the split. This involves a short period of downtime (just enough to make a full backup of everything).

     

    If you wish to reduce the size of the database, refer to the KB kb 1000125 at http://kb.vmware.com/kb/1000125 which gives directions to purge historical data that may beno longer required.

     

    1. Stop the VirtualCenter server (VC 1), and back up the VirtualCenter database.We'll call this Backup 1.

      This is a good point to return to if you want to rewind everything back to the way it used to be. Procedures on backing up the SQL Server database are documented at: http://www.vmware.com/files/pdf/vc_microsoft_sql_server.pdf

    3.2 Stop the VC 1Service and take a backup of the database.

     

    1. Disconnect all the hosts in VirtualCenter 1

      Select "Hosts & Clusters" in the left panel, and the "Hosts" tab in the main panel.

      Sort the hosts by the "State" column, and select all of the hosts that are in the Connected state. Right-click and select "Disconnect".

    This step is to prevent multiple VirtualCenter instances from attempting to connect to the same set of hosts.

    1. Shut down the VirtualCenter installation (VC 1).

      Exit all VirtualCenter client(s), and shut down the VirtualCenter service.

     

    1. Back up the original VirtualCenter database again. We'll call this Backup 2

      Ensure that you do not overwrite the backup taken earlier - Backup 1! That one is important for rolling back if things go wrong.

      This backup is that of a modified database, in which all the hosts are marked as disconnected, This is so that when the new environment is brought up, it doesn't attempt to access the hosts immediately.

      This backup is the one you'll be working with, restoring into the other VC database as the "clone".

     

    3.3 Restore data into the database used by VC 2

     

    This is where we load the data into the new database, and set it up for VC 2.

     

    1. Restore the backup into the new database.

      Use the Backup 2 above (the one after disconnecting all the hosts).

    2. Set up the VirtualCenter user and login in this database, so that you can log in and access the VC tables.

      This is the database login used by VirtualCenter to access its tables.

      At this point, the database should be ready for access. To test it out, connect to the database using the VC login, and see if you can access the VPX_VERSION table.

      E.g. for SQL Server, you can run OSQL from a command line on the SQL Server host as follows:

     

    C:\> OSQL -S <SQLservername> -U  -d <

    -P <vcadminpassword>databasename>

    • 1> select * from vpx_version

    • 2> go

     

    SQLservername is whatever name you used to connect to this server in SQL Server Enterprise Manager.

    Vcadminuser and vcadminpassword are the username and password respectively used to connect to this SQL Server instance. This should print out a query result with a single row containing the VirtualCenter database schema version.

     

    If you get errors about unknown table names, or permissions, check the user and login setup.

     

    1. Set up the ODBC data source for this database, using the above user name.

    This is the data source that VirtualCenter should be using.

    3.4 Install the second VirtualCenter server

    You can now install the new VirtualCenter server (VC 2) and point it at this newly created database.

     

    1. Install VirtualCenter on the new host, pointing at the new data source and database.

      Specify "use an existing database", and point it at the data source we created above.

      WARNING: when the installer tells you that this points to an existing VirtualCenter repository, and asks "Would you like to overwrite the data?" it is vitally important to answer No.

      Set up the license server for this VirtualCenter server as needed for the split setup.

    2. Stop the new VirtualCenter server (VC 2) and copy over the SSL certificates from the first VC server (VC 1).

      These certificates are in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL, It is very important to do this step to avoid problems later on when the new VirtualCenter instance (VC 2) tries to connect to all of its hosts.

    3. Restart the new VC server (VC 2), and connect to it using VI client.

      It should show the complete environment from VC1 at this point, with all the hosts disconnected.

    4. Change the VirtualCenter Unique ID.

      In VI client, select Administration/VirtualCenter Management Server Configuration. Select "Runtime Settings" from the left. In the "Unique ID" column, change the value. (Just pick a different value from the one there, which is usually a small integer).

    5. Delete the datacenter(s) that do not need to be managed by this VirtualCenter (VC 2).

      Select the unwanted datacenter in the VC client, and right-click and select "Remove". This step can take a few minutes, and you should see a task in the Tasks pane with a progress bar.

      NOTE: This step can take a long time if you have a lot of data in the database.

      NOTE: While this task is proceeding, you can go on to the next step.

    3.5 Bring up the first VirtualCenter server (VC 1) and delete the moved hosts being managed by VC 2

    Now you can start up the first VC and clean it up in parallel, in exactly the same fashion as the other VC.

    1. Start up the first VirtualCenter server (VC 1) and connect to it using the VI client.

    2. Delete the datacenter(s) now being managed by the second VirtualCenter (VC 2).

      At the end of this step, each VirtualCenter installation should be managing a different set of datacenter(s) and hosts.

    3. Wait for both datacenter remove operations to complete_._

     

    3.6 Bring everything back online

    1. Do a final inspection of both VirtualCenters.

      Just check that the right datacenter has been deleted from each VirtualCenter installation.

    2. Ensure each instance has a unique instance ID.

      Because the second instance is a copy of the first, they will have identical instance IDs.  The instance ID is used to generate unique MAC and UUID addresses for all VMs in the inventory, so each VC instance should have a different instance ID to prevent potential collisions.  This will not affect the MAC addresses of existing VMs in the inventory.

    3. Connect one host in each VirtualCenter.

      Just to check that the SSL certificates have been copied over correctly. Both should reconnect without popping up any dialog boxes, especially a prompt for username and password.

      If you do get a prompt for a username and password, especially from the second (new) VirtualCenter installation, double-check the SSL certificate files, and make sure they are a copy of the files from the first VirtualCenter (VC 1). If they are not, copy them again and restart the second VirtualCenter, and retry the connection.

    4. Finally, reconnect all the remaining hosts in each VirtualCenter.

      The final step of the process. In each VirtualCenter, select "Hosts & Clusters" and then the "Hosts" tab. Sort the rows by "State".

      Shift-select all the disconnected hosts and right-click and select "Connect". This should cause all of them to get connected one by one (you'll see tasks and progress activities). It is OK to select and reconnect all of them in one big batch at this step; VirtualCenter will perform several, but not all, reconnects in parallel, throttling the activity appropriately.

     

    4 Risks

    There are, of course, some risks associated with this process.

     

    1. You must be thoroughly familiar with backing up and restoring SQL server databases.

      This should go without saying. It would be a good idea to practice this on a small test installation, to at least ensure that you can roll back a VirtualCenter installation to the point of the backup.

      This is also vital if you want to just restore everything back to the way it was before the split operation was started. As noted above, the directions to backup and restore SQL Server databases is documented at: http://www.vmware.com/files/pdf/vc_microsoft_sql_server.pdf

    2. Bad database backup/restore during the split.

      Of course, things could still go wrong during the DB backup and restore at step 9 above. If you cannot bring up the second VirtualCenter at this point, and see all of your hosts, the database backup/restore has not been successful.

      At this point, you can stop VC 2 and repeat the backup/restore. Or just go back to the original setup, restore Backup 1, bring up VC 1, and you are back to the original state of affairs.

    3. Hosts may time out during the reconnect.

      If they do, you'll get lots of annoying dialog pop-ups asking you to re-enter the username/password for connections.

      If you know that you have copied the SSL certificates correctly (as in step 3 above), you can just hit "Cancel" on these dialogs, and re-try the reconnect on them later.

    4. SSL certificates not copied over correctly.

      The symptom is that you'll immediately get dialogs popping up asking you for hostname/password for each of the hosts you are reconnecting inVC 2.

      You could either stop VC 2at this point, recopy the SSL certificates, and retry the connect, or you could (of course) always enter the username/password for each host (tedious but safe).

     

    5 Rolling back

    If you do decide to roll back, especially after performing some of the destructive steps above, the best way to do so is as follows:

     

    1. Stop both VirtualCenter servers.

    2. Restore VC 1's database from Backup 1.

    3. Restart VC 1.

     

    At this point, all hosts should be reconnected automatically, since the backup in step 2 was taken before all the hosts were disconnected. Of course, the restore will take correspondingly longer because we are also restoring all the historical statistical data.