Creating a snapshot with quiesce guest succeedes, but triggers vss-error in eventvwr on server 2008 r2 DC

[citybird]

Hi

I'm trying to backup my virtual DC, running Server 2008 R2. The Computer has all windows updates and Vmware Tools installed. When creating a snapshot (quiesce guest file system is active, but snapshop the VM's memory is not), the creation of the snapshot succeeds, but i get an error and a warning in the event viewer of the VM. The errors are triggered during the snapshot processing.

The error and warning are followed by a series of informational ESENT-Events, freezing all other Shadow copy instances. So the lsass AD is the only one raising an error.

-

---

Error 489, Source ESENT:

lsass (480) An attempt to open the file "c:\Windows\NTDS\ntds.dit" for read only access failed with system error 32 (0x00000020): "The process cannot access the file because it is being used by another process. ". The open file operation will fail with error -1032 (0xfffffbf8).

-

---

Warning 8229, Source VSS:

A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,

the error is likely to reoccur.

. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:

PostSnapshot Event

Context:

Execution Context: Writer

Writer Class Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Writer Name: NTDS

Writer Instance ID: {8231a194-b132-41b1-97e9-7c3f8333780d}

Command Line: C:\Windows\system32\lsass.exe

Process ID: 480

-

---

Because of that, I'm afraid that the backup of my AD might be inconsistent.

Any ideas on how to resolve this?

Thanks in advanced!

[kcucadmin]

vmware has not implemented APPLICATION level VSS support for Win2k8

it is File Level ONLY.

I was trying to find the Support Table that shows what vmware has implented for Each guest OS.

http://www.vmware.com/pdf/vdr_11_admin.pdf

page 8 has a table of the levels supported. basicly VDR is unable to provide a Active Directory backup at this time with out using a in guest backup as well.

[pcmanning]

I've just installed VDR 1.2 on vSphere 4.1 and am trying to backup two Windwos 2008 R2 DCs and another with an ADAM install on (virtual center 4.1). All three fail to with the similar errors as above. The PDF says that application level backup is supported assuming the UUID attribute is set - this is true in my case as I have created the VMs on ESXi 4.1. Another w2k8r2 server with no applications on backs up fine.

VDR error: Failed to create snapshot for Home2, error -3960 ( cannot quiesce virtual machine).

Event log shows errors Esent 402, 482 and VSS 8229 with a code of 0x800423f4

So basically this is still broke?

Paul

[friedeva]

I can confirm the exact same behaviour than described on previous post (vSphere 4.1, VDR 1.2, 2008 R2 DCs).

If you list the VSS writers on the servers, you can see that the NTDS writer fails with a non-retryable error:

Writer name: 'NTDS'

Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Writer Instance Id: {ed528947-a91f-40a7-b4c8-417226caf7be}

State: Failed

Last error: Non-retryable error

I found no workaround for the moment!

[Raudi]

I got this errors on my 2008 R2 DC and the VMware Shapshot fails:

Quelle: ESENT

Ereignis-ID: 482

Ebene: Fehler

Beschreibung:

lsass (464) Versuch, in Datei "
?\Volume{e8d7073f-a6fb-11df-9786-005056a20001}\Windows\NTDS\edb.log" bei Offset 3761664 (0x0000000000396600) für 512 (0x00000200) Bytes zu schreiben, ist nach 0 Sekunden mit Systemfehler 19 (0x00000013): "Das Medium ist schreibgeschützt. " fehlgeschlagen. Fehler -1032 (0xfffffbf8) bei Schreiboperation. Wenn dieser Zustand andauert, ist die Datei möglicherweise beschädigt und muss aus einer vorherigen Sicherung wiederhergestellt werden.

Quelle: ESENT

Ereignis-ID: 408

Ebene: Fehler

Beschreibung:

lsass (464) In Protokolldatei '
?\Volume{e8d7073f-a6fb-11df-9786-005056a20001}\Windows\NTDS\edb.log' konnte nicht geschrieben werden. Fehler -1032 (0xfffffbf8).

Quelle: VSS

Ereignis-ID: 8229

Ebene: Warnung

Beschreibung:

Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f4, Für den Generator wurde ein nicht vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird,

tritt der Fehler höchstwahrscheinlich erneut auf.

" zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll.

Vorgang:

PostSnapshot-Ereignis

Kontext:

Ausführungskontext: Writer

Generatorklassen-ID: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Generatorname: NTDS

Generatorinstanz-ID: {3538f908-7deb-4bfa-8a53-8140842bb670}

Befehlszeile: C:\Windows\system32\lsass.exe

Prozess-ID: 464

Very strange, VSS made a snapshot, it seems that ESE want's to check the AD Database in the snapshot and this is failing because ESE can't wite to the edb.log.

The only workaround i found is to disable the application level vss snapshot by settings this value:

disk.EnableUUID = "false"

Is this a Microsoft or a VMware problem? Sonds like Microsoft...

[cmacmillan]

According to the docs for VMware DR 1.2, application consistent backups in 2008 R2 are possible:

Windows 2008 32-bit/64-bit

Windows 2008 R2

Application-consistent quiescing. For application-

consistent quiescing to be available, three conditions must be met:

1 The UUID attribute must be enabled. This is

enabled by default on virtual machines created

on ESX 4.1 hosts. For virtual machines created

on other hosts, complete the procedure “Enable

Windows 2008 Virtual Machine Application

Consistent Quiescing,” on page 38.

2 The virtual machine must use only SCSI disks.

For example, application-consistent quiescing

is not supported for virtual machines with IDE

disks. There must as many free SCSI slots in the

virtual machine as the number of disks. For

example, if there are 8 SCSI disks on SCSI

adapter 1, there are not enough SCSI slots free

to perform application quiescing.

3 The virtual machine must not use dynamic

disks.

However, in at least one installation, ADAM fails to quiesce under VM DR 1.2. However, a manual snapshot appears to provide the desired results... Running "vssadmin list writers" shows 'ADAM (VMwareVCMSDS Write' as failed, non-retryable error.

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

VMware vExpert 2010

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[MKguy]

First off, you should check everything mentioned in http://kb.vmware.com/kb/1007696 .

I recently had a support case for an issue like this which appeared on our vCenter 4.1 VM (which uses ADAM).

Apparantly, it was somehow related to a non-matching disk UUID in the .vmdk disk descriptor file and the UUID actually presented to the VSS writer in the GuestOS. According to support, one possible reason could be SVmotioning a snapshoted VM.

The vmware tools debug log logged:

[http://...|http://...]
00000184	13:05:03.813	[4296] CVmSnapshotRequestor::ParseUUIDs: got disk uuid from host: 6000c291-b256-5990-5cf5-edc9b3905769
[http://...|http://...]
00000339	13:05:08.474	[4296] SnapRequestor: CVmSnapshotRequestor::QueryStatus	
00000340	13:05:09.378	[4296] VssProvider_IsDiskSupported: vendor id is null.	
00000341	13:05:09.379	[4296] VssProvider_IsDiskSupported: VMware disk uuid not found: 6000c293-299f-0a21-e775-c968596e5793	
00000342	13:05:09.380	[4296] VDSHelper::ForEachVolume: return (0)	
00000343	13:05:09.381	[4296] SnapRequestor: CVmSnapshotRequestor::UnregisterProvider	
00000344	13:05:09.387	[4296] Provider: *** CVmSnapshotProvider::OnUnload(0000000002E97DE0)	
00000345	13:05:09.388	[4296] SnapRequestor: Provider unregistered.
[http://...|http://...]

Anyways, the first suggestion was to make the host cache the UUID again by powering the VM off, removing it from inventory and adding it back in. This step didn't work however.

In the next step, the ddb.uuid parameter in the vmdk file was modified from

ddb.uuid = "60 00 c2 91 b2 56 59 90-5cf5-ed c9 b3 90 57 69"

to

ddb.uuid = "60 00 C2 93 29 9f 0a 21-e7 75 c9 68 59 6e 57 93"

Note that these are the 2 UUIDs from the debug logs.

This step yielded the desired success and we are able to take quisced snapshots of our vCenter VM now.

If you want to make sure, you should contact support for this issue.

Btw. whether snapping a DC is really useful, as an actual restore could trigger large-scale wreckage to your domain, is another question.

[cmacmillan]

Wisdom of snapping DC notwithstanding, our problem seems to be related to ADAM... these other steps made no difference.

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

VMware vExpert 2010

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[MKguy]

I'm pretty sure you're having the exact same issue like me with this ADAM and you didn't do the last step as it was supposed to be done.

I hope you didn't just arbitrarily copy the UUIDs from the logs I posted myself...

I mixed things up a bit, the posted log excerpts of mine were not exactly extracted from vmwaretools logfiles but from a debugview session after editing some vmware ini file with advanced logging instructions or something. I unfortuneately fail to recall what exactly it was, so you should consult VMware support for this issue/instructions how to obtain this info.

[mrlucmorin]

@MKGuy,

Hi, where did you get that log readout from ? You mention the "vmware tools debug log", but I can't find this anywhere.

I'm trying to troubleshoot a similar issue, and I'd like to verify if the UUID is good.

Thanks

[MKguy]

As mentioned in my last post above, I confused things up a bit and the info is not from a plaintext logfile somewhere on the system, but from a debugview capture of the whole process during an attempt to create a quiesced snapshot.

I unfortuneately do not remember the exact procedure, the VMware support handled it for the most part (my bad I didn't record it exactly). I advise you to contact support and specificially ask for these steps.

Support also mentioned that they are working on an internal KB article for the whole issue, and may (or may not) release a public one too.

In case they ask for a reference case or something, I could PM you my case number so you can forward it to support as a reference.

Btw, this looks like the logging parameter the support edited prior to the actual debugview session:

http://kb.vmware.com/kb/1007873

[lucmorin]

@MKGuy,

yes please, PM me the case number, as I've also entered an SR for this. Maybe this can speed up things.

Thanks again

[HendersonD]

I am having exactly the same problem. My disk UUIDs do not match. You mention that the ddb.uuid parameter was edited. Did you do this edit or did VMWare do it for you? If you did it, how?

[MKguy]

VMware support did it for me. But really, all they did was basically described in this post: http://communities.vmware.com/message/1595294#1595294

It should be easy enough to do it yourself if you have both UUIDs.

[HendersonD]

Below is a screen shot of one of the

affected VMs, it happens to house my departments Help desk solution. How do I go about editing the vmdk file? The only vmdk file I see is 9.4GB. I am running ESXi 4.1. If I login to the console (via Tech Support Mode) would I see some other files that I am not seeing just browsing the datastore

[Paul_Brannon]

Yes, in your datastore in /vmfs/volumes/, you'll see that a virtual disk consists of two files: a .vmdk and a -flat.vmdk.

The ddb.uuid parameter is in the .vmdk text file. You can edit this with vi. The flat file is your virtual disk data.

I'm having all the above problems with the VMware snapshot provider provided by ESXi 4.1 and any windows server 2008 that is a domain controller or has ADAM. I'm trying to monitor the vmware tools with DebugView after modifying the tools.conf file and restarting them and such but I don't get any useful debug traffic to confirm whether I have this mismatched uuid problem. Any tips on running DebugView beyond what was mentioned in the KB article?

This has got to be affecting more than just the few people who have posted so far.

thanks,

paul

[HendersonD]

I found the trick to running DebugView. Right click on it and run as Adminstrator. Once open, go to the View menu and make sure everything is checked except Log Boot. Now take the VMWare snapshot of the VM you have DebugView running and it will show the disk uuid values.

If you look at the graphic I had in my previous post, I do not have a flat.vmdk file at all. I only have one vmdk file per VM. I think this might be a function of ESXi 4.1. I am not sure how to edit a single multi gigabyte vmdk file to change the ddb.uuid parameter.

[HendersonD]

I am making progress. I launched FastSCP and browsed my datastore and am seeing both vmdk and flat-vmdk files. I am not sure why I do not see this when browsing the datastore using the VI client. I will now attempt to edit the vmdk file and change the ddb.uuid parameter.

[HendersonD]

Below is the output of VMWare Tools log file:

ParseUUIDs: got disk uuid from host: 6000c29d-d0d5-8bfb-02d3-49f81d73c621

ParseUUIDs: got disk uuid from host: 6000c299-5236-6c99-5a66-75dbe1ec161f

VMware disk uuid not found: 6000c299-5236-6c99-5a66-75dbe1ec161f

VMware disk uuid not found: 6000c29d-d0d5-8bfb-02d3-49f81d73c621

When I opened the Help.vmdk file this is the disk ddb.uuid:

ddb.uuid = "60 00 C2 9d d0 d5 8b fb-02 d3 49 f8 1d 73 c6 21"

Two questions:

1. This VM only has one disk, I am not sure why the log files appear to show two disks, any ideas? The one disk does have two volumes, a System Reserve and a C drive but I think this is typical of a Win2008 server.

2. The ddb.uuid in the file is very close to one of the disk uuids shown in the VMWare tools log file. The only difference is a capital C in one is a small c in the other. What do I change the ddb.uuid parameter to? Do I match it with the one from the log file by changing the case on the letter c or do I match it with the other one which ends in 61 1f

[MKguy]

The vSphere Client abstracts the datastore views so you will only see one vmdk file in it even though it actually consists of the flat-vmdk and the separate descriptor-vmdk.

>1. This VM only has one disk, I am not sure why the log files appear to show two disks, any ideas? The one disk does have two volumes, a System Reserve and a C drive but I think this is typical of a Win2008 server.

No idea about. I'm sure it's not related to partitioning though. So this non-existent ID (6000c29d-d0d5-8bfb-02d3-49f81d73c621 ) could very well be your issue here. Do all your VMs show 2 IDs? You could try powering the VM off, removing it from inventory and adding it back in, which was also suggested by Vmware. (Maybe the VM had another VMDK mounted and hot-removed at some point? Check the device manager with DEVMGR_SHOW_NONPRESENT_DEVICES=1 if there's a ghost disk-device or anything).

>2. The ddb.uuid in the file is very close to one of the disk uuids shown in the VMWare tools log file. The only difference is a capital C in one is a small c in the other. What do I change the ddb.uuid parameter to? Do I match it with the one from the log file by changing the case on the letter c or do I match it with the other one which ends in 61 1f

I'd say capitalization doesn't matter here, it seems support did the same in my case with capitalizing the C in the ddb.uuid parameter (see http://communities.vmware.com/message/1595294). So yea, try the step above first.

I also advise you to contact VMware support too in order to make sure of all these things are correct.