DR procedure concerning VM Domain Controllers and Virtual Center

[khughes]

We're gearing up for an off site DR test soon and I've been running the steps through my mind quite a bit the past few days at work. I think I'm getting pretty close to the final procedure I'll go off of during the test but I have a question about VirtualCenter/License server and the domain controllers.

For some background

- Both DCs are Virtual Machines

- Our VirtualCenter / License Server house the license file for our ESX hosts

- We run ESX 3.0.2 / VC 2.0.2

When we did our VMware install / Conversion, we had a server already in the domain that we used for our VirtualCenter box (to use the domain account to do vMotion etc)

Ok for what I think is the fun part. In a DR test, we're going to have to restore all of our VM's from tape, including both DC's. In order to power on any VMs (lets say after I restore DC1/DC2) on ESX 3.0.2 we need to have the ESX hosts licensed which means we need the License server up.

My thought process is (I haven't tested this but plan to in the next couple days):

1. Install the license server on the box that will house VirtualCenter (instead of installing VirtualCenter and License server at the same time) and point to the license file we have.

2. Configure ESX hosts to point to the license server to get a valid license so we can power on VMs

3. Power on the DCs

4. Join the VirtualCenter box to the domain, then install the VirtualCenter server

Then just continue restoring the rest of our VMs...

Are there any serious flaws in my thought process? Does the VC box actually need to be in the domain before installing the VirtualCenter Management software or can it be installed while in a workgroup then added to the domain? Any other thoughts on DR restores or helpful hints?

- Kyle

[ctfoster]

Kyle.

I dont think you are in such a 'Catch22' as you think. Without the VC your ESX servers will continue to function. The only features that will not work are vMotion and DRS. In fact the ESX servers will operate without licenses for up to 14 days before you will be prevented from powering on a VM. Therefore when you reinstall the VC your domain is going to be up and running which solves your workgroup issue.

I'm sure the community to pass you on a few hints - but you left out a key bit of info. Where is the database for the VC stored? I'm guessing it's on the same machine as the VC your are trying to rebuild. Also have you considered running the VC and a VM itself - maybe not under ESX but under Server. That makes a DR recovery to be alot simplier - you just need to install Server on some box and recover your VM files onto it.

[khughes]

Kyle.

I dont think you are in such a 'Catch22' as you think. Without the VC your ESX servers will continue to function. The only features that will not work are vMotion and DRS. In fact the ESX servers will operate without licenses for up to 14 days before you will be prevented from powering on a VM. Therefore when you reinstall the VC your domain is going to be up and running which solves your workgroup issue.

The thing is in my test lab, I could not power on a VM without a valid license. I've run into the same problem 2 times now, including today. So I'm not getting this magical 14 day phase (I wish I would it would make everything 100x easier). This isn't a trial copy of 3.0.2 and no you can not download or get a trial license or ISO file of 3.0.2 from VMware (at least not that I can tell and I tried for 2 weeks). So it comes down to I can't power on any VM without at least the license server I'm assuming.

I'm sure the community to pass you on a few hints - but you left out a key bit of info. Where is the database for the VC stored? I'm guessing it's on the same machine as the VC your are trying to rebuild. Also have you considered running the VC and a VM itself - maybe not under ESX but under Server. That makes a DR recovery to be alot simplier - you just need to install Server on some box and recover your VM files onto it.

Yes it is a local SQL 2000 Server that runs on the VC box. I have read many people doing this and the idea is intriguing to me, but I'm not sure if I can pass it off to my bosses. I'll defiantly have to do some testing with it. Thanks for reminding me of that option because that would make it a lot easier to restore and still have the physical separation from the ESX/SAN environment ... even though if your SAN goes down you're SOL with the rest of your ESX environment

- Kyle

[ctfoster]

This doesn't sound right. After all running the VC as a VM under ESX is a supported configuration - so how would that work? What error are you getting when to start up?

[habibalby]

Hi,

I would add some echo to this thread, interms or IP's and networking.

You have a Primary Site running ESX Host on different IP Schema, and you have your DR Site without the ESX Hosts installed! Correct me if i'm wrong. How your clients are connected in the Primary Site? the DC's are hosted on the ESX Servers where the clients are in the same rage of IP Schema. Does the Primary Site can connect to any Client, Server or host in the DR Site? Okay, suppose you don't have the connectivity between the DR and the Primary Site, what is the use of Restoring your DC's Images to the ESX Hosts. Yes, If you have the connectivity where the Clients at the Primary Site and the DR Site can reach each other, then just restore the VMs and the Clients will be automatically connected. But make sure that your Clients are pointed to the Correct DNS Server at the DR Site.... It means that if you are at the DR, your Clients at the Primary Site they will reach the DNS Server which is hosted in the DR Site. Not on the Primary Site. In this case, you will required to configure the Additional DNS Server at the Client Side Manually or you must have the DHCP Server whcih will broadcast your new DNS Configuration.

Your current, VC since it's hosted Physically, not as VM on the ESX Host, then if you have the ability to convert that to VM, Restore at DR Site and Power it on. Also, you can build a new VC along with the Licensing Services and add your DR Host to it.

But the catch up here is what is your RTO's? The answer here is your budget and how much your company can invest.

Have your ESX Host at the Primary Site connected to a SAN Storage where all your VMs reside, and at the DR Site have the same Setup. You must have a Live connectivity between your Primary Site and your DR Site. Then, synch your Primary SAN Storage to the DR Site SAN Storage, and your can do a VMotion to your DR Esx Host's, then your DHCP will will provide the DR IP Schema to your Clients. DC and DNS Reachability will be there.

But, does your Clients can reach your DR Incase of a Disaster, or only your ESX Hosts at the primary Site can reach your DR Hosts?

Attached is a proposal to my company i'm currentlt working on it. Hope it helps.

[khughes]

Something along the lines of "unable to power on this virtual machine without a valid license" I can get a screen shot and post it tomorrow.

- Kyle

[ctfoster]

Hold on though. I assume you are building these macines from scratch and therefore they've never seen the existing licence server and therefore have no cached file. Also without a valid eval licence loaded up locally I guess that is what you would see.

[khughes]

Without going into extensive details on our DR agreement with this company, they provide us with a pre-agranged set of hardware which would replace our own and a location to setup shop at. So basically it would be like a mobile building? I don't know all the details they have had this contract for a long time (before I got on board and before us moving to VMware). I believe we're going to look at different options when this contract runs out but until then we need to make due with what we have.

This DR test will be in a sandbox setup with no internet connection. Thanks for the attachment its nice to gather information like that to work on when we do test the waters for a different DR solution.

- Kyle

[khughes]

Hold on though. I assume you are building these macines from scratch and therefore they've never seen the existing licence server and therefore have no cached file. Also without a valid eval licence loaded up locally I guess that is what you would see.

Bingo. We will have blank servers in a sandbox setup where we'll bring up an exact mirror image of our network from our last bulk backup on tape. There will be no internet connection so no talking between DR site and production site. Yes this is going to be fresh installs for everything.

- Kyle

[Dave_Mishchenko]

Just wondering how you're doing your backups for your DCs as snapshot technology isn't supported by MS for that and can create problems in the dbs on the DCs get out of sync.

[habibalby]

Okay, no DR Connectivity between the Primary Site and other Branches? This is not a functional DR Site, this is a DR Store:) The question is, Are you planning to have a DR Site? or You already have a DR Site and you wanted to Simulate the DR between the Primary Site and DR Site?

[khughes]

Just wondering how you're doing your backups for your DCs as snapshot technology isn't supported by MS for that and can create problems in the dbs on the DCs get out of sync.

a) we're using esXpress to do bulk backups which does use snapshots

b) never heard that MS doesn't support snapshots and could cause problems. I guess I either missed that in the reading or the people that helped us with our install didn't mention it.

So far its been about 7-8 months since we finished our conversion and haven't seen any problems yet. I know that a lot of people keep a DC on a physical box outside of the virtual world, seems like a smart thing considering that MS doesn't support snapshots and ntp etc...

- Kyle

[khughes]

I think in time we are looking for a DR site but right now we got what we got. This contract was all setup before I joined the company and we had to completely re-work it to adjust for our VMware setup. Our test will be at their facility though

- Kyle

[Dave_Mishchenko]

Here's the KB article - support.microsoft.com/kb/888794

To roll back the contents of Active Directory to a previous point in time, restore a valid system state backup. A system state backup can be restored up to the tombstone lifetime number of days after the backup was performed. The backup must have also been made on the same operating system installation as the operating system that you are restoring.

*Active Directory does not support other methods to roll back the contents of Active Directory. In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent. *

We also do not support using "undo" and "differencing" features in Virtual PC on operating system images for domain controllers that run in virtual hosting environments.

There's a good document here that goes into more detail. http://www.microsoft.com/downloads/thankyou.aspx?familyId=64db845d-f7a3-4209-8ed2-e261a117fc6b&displ...

The condition they describe would be more evident if you say lost just one of two domain controllers and then restored a week old snapshot. If you're taking the snapshots of the 2 DCs very close together and then restoring them both you could potentially avoid the problem, but I'm not an expert on how USN are changed if for example you back them up at 3 AM and there are no AD changes made at that time and then would subsequently restore them together at your DR site. Having dealt with USN problems in the past though it's definitely something you don't want to have to deal with.

[khughes]

Thanks for the articles Dave. My thoughts of restoring the bulk backup of our DC's was:

a) if we needed to bring both back up at the same time, restore both of them back to the ESX hosts, then power them on at the same time or within 30 seconds of eachother. I have tested this and replication and everything seemed to work together.

b) if we don't need to restore the 2nd DC which doesn't have any roles on it, only our "print server" which are just shared printers and in a DR most likely wont have the exact same printers anyways, would be restore the primary dc and then build out a new VM and promote it.

I'm actually in the process of trying my thought process from the origional post to see if that will work.

Besides the obvious things you pointed out about the domain controllers which -knock on wood- haven't seemed to happen yet, is there anything wrong with my thought process of bringing everything back up?

Install license server -> connect esx hosts to license server -> power on DC(s) -> join VC box to the domain -> install sql / VC -> continue DR bulk restores

- Kyle

[azn2kew]

Kyle,

Assuming your primary site is functional with ESX hosts, virtual center servers with SQL 2000 installed and valid license servers with bunch of production VMs and using esXpress for backup solutions?

In a DR scenario, if your primary site is down, everything should be replicated or transfer at least all the .vmdk disk images of all VMs to your DR site via SAN replication software like SANCopy, SANMirror, vReplication, Platespin etc.. If you dont' have those enterprise level of application due to $ involve than manually configure your DR is reasonable straight forward.

Components required during DR scenarion:

1. Backup solution ->esXpress in your case to back up all VMDK files to DR site

2. Standby virtual center server preconfigured and virtualized. Or you can reconfigure VC server very quickly from scratch with .lic file in hand.

3. All ESX hosts server standby and ready to be connected to the network with new IP schemes and attach to SAN storage through HBA. Present all the backup VMDK images LUNs to standby ESX hosts and register .vmx file to cluster.

If you dont' have DR SAN, then where are VMDK backup to? NFS I supposed, or any means should be fine as long as you can see .vmdk and .vmx files than it can be register anytime. Make sure your virtual machines IP addresses and DNS is configured correctly. You might have your DNS admin takes care of DR DNS zone so when it kicks to DR drill, all DNS and IP should be taking care of.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant