VMware Communities: Message List - ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

ODOCChuck — Thu, 06 Aug 2009 15:09:01 GMT

Is this still an issue, have updates to the driver or an Update release resolved the issue with the synchDriver, or is it "works as designed" and dont expect a fix? I am not so concerned about a corrupted AD, but it does cause issues when the DB is corrupted. There are more reasons to do a snap shot than restoring a full VM.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Josh26 — Sat, 04 Apr 2009 03:19:10 GMT

alastairc wrote:Hi Mike,

. Effective immediately I'm removing the sync driver from all our VMs which host databases of any type.

Could this be described as a bug, or is it just bad practice to install the filesystem sync driver in VM's which host databases?

I see this advise thrown around a lot, with no regard for the consequences. Basically what the sync driver does is to freeze IO, in a "best effort" at getting an application consistent snapshot. Most databases, eg Exchange and Oracle have major issues with this write lag.

The "just disable the driver" approach can be just as dangerous. It means nothing is remotely consistent in the snapshot. Try restoring one of those snapshots, and doing a chkdsk. You have about a 50% chance of something requiring a repair.

The bigger warning is, don't try restoring a snapshot of yoru domain controller. It's not pretty.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

piyush1414 — Fri, 03 Apr 2009 17:47:07 GMT

Try Edb Repair tool to repair corrupted Active Directory.

Download the free Trial from here http://www.edbrepair.org/edb-active-directory-recovery.php

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Jabadakkas — Tue, 15 Jul 2008 07:41:04 GMT

A few months ago I had a chat with a VMware pre-sales consultant. He told me that the upcoming ESX 3.5 Update 2 which is due Q3 2008 will include Volume Shadow Copy Service (VSS) support. This will hopefully eliminate any problems ESX admins are experiencing in regards to snapshots on AD vm's. I googled for VSS support and found that the Virtual Server 2.0 betas already include VSS support. Ofcourse I realize that Virtual Server is a hosted virtualization product and accesses storage in a different way than ESX does, but still you could give it a try.

Are there any forum readers that tested VSS on Virtual Server? Were you able to snapshot an AD vm without the corruption problems on ESX that alastairc identified at the start of this forum topic?

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Tounet — Fri, 07 Mar 2008 13:54:11 GMT

Alternative method could be using the pre-freeze script of vcb (C:\windows\pre-freeze-script.bat) and post-thaw script (C:\windows\post-thaw-script.bat) inside the virtual machine.

You could easily stopping the services database, or run a locally backup of the database if you can't stop the services for few seconds / minutes.

For AD Controller, you could use these script for run systemstate backup locally with ntbackup. Don't forget that systemstate is the only supported and recommended microsoft solution for backup AD Controller.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

vmcms — Fri, 29 Feb 2008 14:18:39 GMT

Justin,

It looks like you've got a process for recovering from a corrupted DC such that AD corruption seems a non-event should it transpire.

Does this take into account DNS and AD cleanup? My experience is that though these objects are supposed to clean themeselves up, they do not always. And occasionally there is a DC setting or two in Microsoft Exchange 00/03/07 that gets stuck on an old DC that has to be manually tracked down and corrected.

With these things explicitly considered, how significant an event do you feel AD corruption and recovery to be?

VMCMS

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Thu, 28 Feb 2008 15:13:01 GMT

Yes it is, I completely concur and that is what I suggested on the first post I made on this thread.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

FunkyD — Thu, 28 Feb 2008 10:55:09 GMT

It seems to me that the golden rule with any server that hosts a database is to backup the database first either locally or using your file backup solution and then snapshot the server. To recover, restore the snapshot and then resotre the databases following whichever method suits. You cannot reliably snapshot a server with a database and simply restore it and expect it to work - it might work but it's better not to risk it.

On that basis my strategy is to do a daily snapshot of file servers and SQL (they all have a local backup of the database) and sftp them to the remote site. For database servers (AD, Exchange) I do a monthly snapshot, sftp to the remote site and to recover use the last nights backup to disc files for the databases.

I am hoping to improve things by upgrading Veritas 9 to version 11 so I can use the VCB module.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Mon, 25 Feb 2008 21:57:47 GMT

Yes. Now we are on the same page. This paper identifies the issue of not knowing which writes need to be commitied at the Hypervisor layer. Now the part we need to know is does the VMware SCSI driver work with the Hypervisor to write FUA flagged requests immediately or not. Once that is known the we can determine if we can trust the underlying host OS with this task or does this only occur with the LGTOsync driver. And I completely agree that todays DB's will deal with these issues much better than 1st and 2nd gen ones did and will recover to a usable state. They do still get in trouble when under extreem stress as the requests start saturating cache and can bottle neck with a transaction log and data items in the write-back cache at the failure point.

Thats a good paper, thanks for sharing it.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Justin King — Mon, 25 Feb 2008 21:23:06 GMT

Huh, I think we're saying the same thing but comming to different conclusions

The meat of what I'm saying is that AD is simply stored in a jet database. Just like any other modern database it keeps log files of data to be commited and thus this level of data security is already partially accounted for. A partially commited DB change would still have a log present even in a circular setup and thus I consider the issue minor. Perhaps I'm unique in this area, but I've had a number of problems occur when a snapshot occurs on a DC and it has the sync driver installed. Observational data implies the entire VM is paused while the host commits vmdisk writes, simply uninstall the driver and do a snap and look at the difference in time. At least on any of my esx hosts it's easily visible, and the entire host is essentially "paused" for a few seconds.

Only VMware docs I can find that reference FUA is this one: http://www.vmware.com/files/pdf/usenix07.pdf

Still need to read through it though.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Mon, 25 Feb 2008 19:34:45 GMT

I can see why we are not on the same page. If VMware would provide a clear description of the functional parts of LGTOsync we would not have these issues or this discussion.

Here is how I understand the functional side of this driver.

Since ESX can not determine what the VM is doing with disk writes and memory cache at the time a snapshot is requested it cannot correctly freeze the I/O state of the vmdk for that snapshot to provide integrity of any cached SCSI operations.

VMware needed to montior this SCSI disk write and memory cache activity and feed that info back to the ESX physical hardware and complete those disk I/O operations in order to provide the integrity requirements of backup functions. They provided this capability by developing the LGTOsync driver with Legato. So when I think of the LGTOsync driver I am not looking at the VM flushing it's writes, It's the underlying host that needs to do this function. If you disable the driver you lose this capability all together. So unless you can tell me that you know for certain that the VMWare SCSI drivers support forced unit access down to the host OS then it is better to run the driver.

So I consider the following when I build VM's for AD or any DB's

http://support.microsoft.com/kb/888794

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Justin King — Mon, 25 Feb 2008 18:22:28 GMT

My understanding of this driver is It was made to directly address the fact that you may snap at a time where all writes havn't been comitted to disk, so it does a sort of "pause" to the MV to allow it to commit all writes before you snap it. So yes, it is supposed to address and keep you form having a non-crash consistent backup.

However, from my experiance, unless you're using some odd DB this is a non-issue anyway as databases already have a built in resilience to such problems and have for a long time. Not only that, but the length of time the entre OS is "paused" seems to cause more problems than it helps (from personal experiance at least), so I've resorted to removing it in most cases.

In the context of AD, I'm not worried. I dont run only one AD server, so in the event of a localized failure I can easily force FSMO onto one of the other DCs and simply deploy from template then DCPromo and be back up in 20minutes. I wouldn't even bother with a restore. If the problem is widespread and I am forced to do an authoritive restore ... having an inconsitent backup that forces it to be another 15minutes older (as it rolls back the change logs) is the least of my worries.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Mon, 25 Feb 2008 17:38:58 GMT

I have not seen any official KB either. I would tend to think that if this were a serious issue it would much more wide spread. The corruption is more likely external to the sole existence of the LGTOsync driver.

I would like to look at it more closely in the LAB.

And if you turn it off you have another more serious issue.

http://supportforums.vizioncore.com/forums/permalink/943/986/ShowThread.aspx#986

You will not be crash consistent. Then you will see DB corruption at the time you need to recover the VM. Ugly

Best bet NTBackup AD locally and then snapshot it.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

JRink — Mon, 25 Feb 2008 16:54:27 GMT

I mentioned yesterday that I've never even heard of this sync driver prior to reading this thread. I did some searches on VMWare's support site and really didn't pull up anything.

Is there a document from VMWare in regards to best practices of snapshotting VMs that explains this and other "gotchas" we should know about? If not, how would I have even known about this?

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

joergriether — Sun, 24 Feb 2008 22:26:35 GMT

it´s generally a good idea to disable the sync driver inside any ad domain controller and inside any exchange server.
when it comes to sql i never ever had any problems with the sync driver when snapshots were taken. strange, but that´s what i experienced.

best regards
Joerg

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

alastairc — Sun, 24 Feb 2008 22:08:55 GMT

Yes, at least with versions 3 and above of ESX server, the Sync driver is included as part of the default vmware tools install. Following the problems I had with Active Directory, I don't really trust it anymore and make a point of uninstalling it from any VM that will be running a database of any description.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

JRink — Sun, 24 Feb 2008 18:32:33 GMT

Is the Sync Driver automatically installed when you do a typical installation of the VMTools on a Windows server?
I never heard about this driver before...

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mrbrown66 — Fri, 22 Feb 2008 21:23:55 GMT

In an Active Directory environment I would recommend using the combination of an NTBackup follow by a file level based VCB backup.

Aside from the quiesing issues mentioned above, restoring a domain controller from a VCB snapshot could result in corruption within AD. Corruption could occur when the domain controller appears back in your forest with the clock time totally different from other domain controllers.

We have setup a nightly NTbackup on each domain controller, and this gets backed up using a VCB file level backup. The AD restore process is then very straight forward and well documented/support by Microsoft.

Hope that helps.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Justin King — Tue, 29 Jan 2008 18:26:31 GMT

I'll also toss in my own "me too". I've remove the sync driver by default form all of my VMs at this point ... they snap much much faster that way anyway. Ironicallythe sync driveris supposed to prevent that problem (basically verifying all queued writes have occured before the final pause) but it seems to cause more trouble than it solves in my case.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Sangokan — Tue, 29 Jan 2008 18:23:03 GMT

I had a dismounted store once in Exchange 2003 after a snapshot, also disabling the sync driver in Vmware tools fixed it for us as advised on the vizioncore thread.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

DigitalVoodoo — Tue, 29 Jan 2008 17:02:22 GMT

A quick "me too" - I had a similar problem with my domain controller VMs, and removed the sync driver from those VMs. I haven't had a problem with them since, and wholeheartedly recommend it for DC VMs. I haven't had such problems yet with the few SQL VMs I have, but I would think it to be a good idea as well if you have any concerns at all about it.

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Tue, 29 Jan 2008 15:51:31 GMT

That's great info. Thanks very much!

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

alastairc — Tue, 29 Jan 2008 11:22:42 GMT

Hi Mike,

Thanks for your help - I think I may have found the cause of the problem. Our ESX Server is currently massively under-utilised, so I/O contention is probably not an issue, but the text: "The supplied user buffer is not valid for the requested operation" in the event description got me wondering if some other system event was causing the IO operation to fail. Looking through the event log, I noticed that at the exact same time as the NTDS error occurred, the following error was logged in the System log:

Event Type: Information
Event Source: LGTO_Sync
Event Category: None
Event ID: 1
Date: 20/01/2008
Time: 6:14:24 AM
User: N/A
Computer: MELDC1
Description:
The description for Event ID ( 1 ) in Source ( LGTO_Sync ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: , Sync Stop done.

LGTO_Sync appears to be the VMWare Tools file system sync driver, which (if my understanding is correct) is supposed to quiesce the filesystem before the snapshot is taken. After a bit of googling, it looks like this driver is responsible for quite a bit of grief where databases are concerned:

Here someone is having exactly the same problems as me:
http://supportforums.vizioncore.com/forums/thread/2472.aspx

Here it would appear the Sync driver is causing an Oracle database dismount:
http://support.p2v.net/boards/read.php?1,1163,1165

And here's a VM KB article detailing issues with the sync driver and MS Exchange:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=5962168

So it would seem that VMware Tools Sync driver + Databases = Trouble. Effective immediately I'm removing the sync driver from all our VMs which host databases of any type.

Could this be described as a bug, or is it just bad practice to install the filesystem sync driver in VM's which host databases?

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

mike.laspina — Tue, 29 Jan 2008 07:30:07 GMT

Hello,

While it is true that snapshots on databases are not a best practice, this should work without corrupting the system because in theory the system is suspended at a point in time.

Now if you were restoring to that point in time I would say that all bets are off because of transactions that are in progress may or may not be complete.

To protect Active directory simply add an NTBackup of AD before the snapshot.

I would start looking at one of three possible tracks here.

1) It could be a bug in the snapshot code, the paint is still wet on ESX 3.5 (Snapshots are quite mature this is not very likely)

2) You may have a hardware/firmware problem. (SAS firmware and drivers are not fully matured, very very likely)

3) Resource starvation, too much I/O at the time of snapshots. (Seen it happen but not very likely)

Hope this helps.

BTW I have been snapshotting DC's and DB's every day for 2.5 years and never seen a corruption to date. (Hmmm now that I said that ....... I hope I don't eat my words)

Re: ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

Dave.Mishchenko — Tue, 29 Jan 2008 06:52:25 GMT

Microsoft does not support snapshot type backups for Domain Controllers - http://support.microsoft.com/kb/888794.

In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent.

For apps like SQL Server, you should ideally do something to quiesce the file system. That might involve stopping SQL Server just before you create the snapshot and then restarting it after the snapshot is created. That will get the DB files in a crash consistant state before you start to backup the vmdk. Otherwise it's a bit like pulling the plug on the server. http://communities.vmware.com/thread/115035. If the SQL dbs are critical, you might consider using the native sql backup tools to backup the db and transaction logs to a network drive, seperate vmdk, etc.

ESX Server 3.5 - Corrupted Active Directory after taking a snapshot

alastairc — Tue, 29 Jan 2008 06:27:56 GMT

Hi there,

I've just had a ESE database corruption, which is pretty concerning and I'm wondering if anyone else has seen this or advise what may be happening here:

One of our Windows Server 2003 R2 Domain Controllers runs as a VM Guest under ESX Server 3.5 build 64607. The host hardware is a HP Proliant DL380G5 with 16GB RAM, running two quad core Intel Xeon E5345, hyperhreading is disabled so we have a total of 8 HEC's from ESX's perspective. We don't have a SAN, so all VM storage is on the locally attached disks (6 HP 15KRPM SAS drives in a RAID-5 configuration, attached to a Smart Array P400 Controller)

Anyway, our domain started acting strangely one Monday morning, so I had attempted to login to our ESX hosted domain controller to find it was rejecting the login credentials. After forcing a restart of the guest VM I was able to login, and from looking at the event logs I could see that the Active Directory database had become read-only after the following error was logged:

Event Type: Error
Event Source: NTDS ISAM
Event Category: General
Event ID: 482
Date: 20/01/2008
Time: 6:14:24 AM
User: N/A
Computer: MELDC1
Description:
NTDS (384) NTDSA: An attempt to write to the file "C:\WINDOWS\NTDS\edb.log" at offset 3230720 (0x0000000000314c00) for 512 (0x00000200) bytes failed after 0 seconds with system error 1784 (0x000006f8): "The supplied user buffer is not valid for the requested operation. ". The write operation will fail with error -1011 (0xfffffc0d). If this error persists then the file may be damaged and may need to be restored from a previous backup.

The timing of this error concides with a cron job which executes vcbSnapAll to backup our VMs. This leads me to suspect that the action of taking the snapshot has somehow caused a write operation to the AD database to fail. Perhaps windows tried to write during the file system queisce?

My concern here is that this could happen again. We also host production MS SQL Databases on the same ESX host, and I would hate for these to become corrupted in a similar fashion. Can anyone advise, is this problem likely to have been caused by vcbSnapAll taking a snapshot of the VM? If so, are there best practices for snapshotting domain controllers or database servers?