Windows 2008R2 snapshot problem on 4.1

[AntonVZhbankov]

I have problem with snapshots on one of my VMs. And what's strange it started only last friday.

One of my DC's on Win2008R2 refused to take quiesced snapshot after clod migration from one cluster to another. Remaining DC, absolutely the same configuration still can take snapshots.

I've made new VM, built from nothing, clean installation, promoted to DC instead of failed on and problem still persists.

What I've found in Windows logs:

1) Volume Shadow Copy Service error: Unexpected error DeviceIoControl(
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 000000000000042C,0x00560000,0000000000000000,0,00000000003C7F70,4096,[0]). hr = 0x80070001, Incorrect function.

.

Operation:

Exposing Recovered Volumes

Locating shadow-copy LUNs

PostSnapshot Event

Executing Asynchronous Operation

Context:

Device:
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Examining Detected Volume: Existing -
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Execution Context: Provider

Provider Name: VMware Snapshot Provider

Provider Version: 1.0.0

Provider ID: {564d7761-7265-2056-5353-2050726f7669}

Current State: DoSnapshotSet

2) lsass (460) An attempt to write to the file "
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log" at offset 7525888 (0x000000000072d600) for 512 (0x00000200) bytes failed after 0 seconds with system error 19 (0x00000013): "The media is write protected. ". The write operation will fail with error -1032 (0xfffffbf8). If this error persists then the file may be damaged and may need to be restored from a previous backup.

3) lsass (460) Unable to write to logfile
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log. Error -1032 (0xfffffbf8).

---

MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

[kswail]

Hi there,

I'm having the same problem here only with my 2008R2 box with the ADDS role. I get the same errors as you in my event log.

What's even stranger is that all of this was working fine a week ago, when my VMs were on local DAS. I've since SVMotioned my VMs to an NFS store, and my problem seems to have coincided with this relocation. I haven't tried installing a new DC as you have though. The SVMotion worked fine for the DC and a myriad of other Windows 2008 R2 VMs, and all of them backup with a snapshot quiesce just fine. Only the DC is being a b#tch.

After the snapshot failure, if you run "vssadmin list writers" from your DC you should see this for the NTDS writer:

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {14cf4f25-2ae0-467e-9bcd-de24a5e00ef2}
State: Failed
Last error: Non-retryable error

I can rule out vss as the cultprit since I can use Windows Server Backup to get a full system state backup, and when I run "vssadmin list providers" after that there are no errors reported. I can also confirm I've taken the DC offline to run consistenty checks and all is golden in terms of the AD databases.

I've tried literally everything to resolve this, uninstall, clean registry, reinstall, ensure the VMware snapshot provider is registered correctly... all for nothing.

I've also read another chap is having problems with snapping his

vCenter server due to ADLDS (ADAM). Clearly it has something to do with

an incompatibility with VMware vss and the ADDS VSS Writer.

http://communities.vmware.com/thread/277587

The only fix for now is to not quiesce when you take your snaps, and so I'm using Windows Server Backup to backup system state to a local vmdk which get backed up along with the VM.

Anyone else seeing this or know of a fix?

Cheers.

kswail

[KEPete]

The only way I've gotten quiesced snaps to work was to reinstall vmtools while unchecking the VMWare VSS driver option. After that, all snap based backups are working fine. This issue only seems to be affecting Windows 2008 running AD or AD LDS roles.

[AntonVZhbankov]

kswail, I've contacted VMware support on this issue.

---

MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

[kswail]

KEpete, be careful since my undestanding is that by not installing the VSS provider, even though you select to quiese during the snapshot doesn't mean that it is actually doing it. In fact the disk is being snapped, but the application like ADDS or ADLDS are not being quiesced and you could have a problem if you even need to restore from a backup.

[kswail]

Thanks Anton, please share with us what you find! Much appreciated!

kswail

[KEPete]

Ugh, I guess I thought it was working since a manual snapshot using quiesce was working fine. Or at least the event message indicated success. I'll take a look at my backups and see what's there. Thanks.

[RobMokkink]

We also got the same issue.

Really anoying problem.

[KEPete]

Well, I got around to checking my backups and found that it did include the ntds.dit file with size and time stamp correct. However, I could not mount it using dsamain to look at the content. I'm using Windows backup for system state now but still curious about the file that did get backed up using VDR. Any other ways I can check validity of that backup? Thanks.

[Rob_Bohmann1]

Found this post : ** I have not tested this to confirm if this is the solution to the problem described in thsi thread **

Here is the relevant part:

" Application consistent quiescing of Windows 2008 virtual machines is only available when those virtual machines are created in vSphere 4.1.

Virtual machines created in vSphere 4.0 can be updated to enable application consistent quiescing by modifying a virtual machine’s enableUUID attribute. "

Enable Windows 2008 Virtual Machine Application Consistent Quiescing

1 Start the vSphere Client, and log in to a vCenter Server.

2 Select Virtual Machines and Templates and click the Virtual Machines tab.

3 Right‐click the Windows 2008 virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off. The virtual machine

powers off.

4 Right‐click the virtual machine, and click Edit Settings.

5Click the Options tab, and select the General entry in the settings column.

6Click Configuration Parameters… The Configuration Paramters window appears.

7Click Add Row.

8 In the Name column, enter disk.EnableUUID.

9In the Value column, enter TRUE.

10 Click OK and click Save.

11 Power on the virtual machine.

if this works hopefully there will be a way to script/cli this and let it take affect on next powercycle op as needed.

Message was edited by: Rob.Bohmann

added disclaimer about not testing. You should of course

[Borja_Mari]

Hi,

as OP pointed, i would recommend you to check this carefully:

http://communities.vmware.com/thread/282199

(Especially page 8 from the "vdr admin" vmware pdf doc)

You can check this too:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100769...

http://www.vladan.fr/troubleshoot-virtual-machine-vss-snapshot/?utm_source=feedburner&utm_medium=fee...EsxVirtualization(ESX+Virtualization)

I hope maybe any of this links helps you

Regards/Saludos,
Pablo

Please consider awarding

any helpful or corrrect answer. Thanks!! -

Por favor considera premiar

cualquier respuesta útil o correcta. ¡¡Muchas gracias!!

Virtually noob blog

[RobMokkink]

I just did a test: disk.enableuuid = false

I can now do a backup with vranger. I will investigate further.

[RobMokkink]

I created a new template in 4.1, but still the problem occurs. In the VM there are no more VSS error, so that is good news.

But it looks like a serious bug!

[kswail]

Hello all and thanks for the input!

Sadly, in our cases, the problem is not related to disk.EnableUUID = True in our VM configs, which is the essence of all the link provided above.

I have this setting in my VMs, and quiescing works accross the board for all Windows 2008 R2 machines EXCEPT for DCs or ADLDS servers.

Thanks for the tips though!

kswail

[mrjlturner]

Gang,

I'm having the same issues on a newly built 2k8 x64 VM built on a 4.1 host. I am trying to backup vCenter server VM and the ADAM writer keeps failing. I eventually added the disk.EnableUUID=false and was able to get a successful snapshot and backup via VDR. Now, I just need to test a restore to see if it actually backed everything up successfully... If anyone else has any input it'd be appreciated. I hope that VMware fixes this issue!

Thanks!

mrjlturner

[lucmorin]

Hi all,

I can confirm the "quiesced snapshot" problem on a Win 2008 R2 VM where vCenter is installed.

This is a VM I created from scratch using Microsoft ISOs.

All along during install, I'd take quiesced snapshots, and it worked fine. The minute I installed vCenter, then quiesced snapshots stopped working. After reverting back to a previous snapshot, quiesced snapshots were again working fine.

As a side note, at first I had installed a custom instance of MS SQL Standard, so I thought maybe this was causing the problem with vCenter. So I reverted even further back, and let vCenter install its own MS SQL Express instance. This didn't change anything as quiesced snapshots were still failing.

I have 2 other Win 2008 R2 VMs, and I can take quiesced snapshots without a problem.

As others have mentionned, the problem seems to originate in the ADAM writer. if you stop the VMWareVCMSDS service, then the snapshot finishes without error. Please be aware that this service is required for vCenter to properly start at boot time, so I'm not advocating this as a solution, just as a fact to help us pinpoint the cause of this annoying problem.

VMWare, are you reading this ?????

Regards

Luc Morin

[KEPete]

disk.EnableUUID = false might make it appear to be quiescing the application state, I believe all that does is force the file system state backups as I had done by removing the VSS Support from VMTools.

See this thread: http://communities.vmware.com/thread/284367;jsessionid=AB53FF4C35DB254C3DCF01F07BDD084D?tstart=0

Now I've also tried disk.EnableUUID=true as the article http://www.blueshiftblog.com/?p=473, http://vknowledge.wordpress.com/2010/08/18/application-consistent-quiescing-and-vdr/, and http://www.sys-con.com/node/1544145 suggests and what was posted above, but that did not work for me either.

So far I have not found anyone getting this to work or at least have a real backup/restore work even with the work arounds.

[lucmorin]

I opened an SR with VMWare, and I was talking with the tech support rep yesterday, and he was convinced that the problem was caused by having SQL server installed on the same machine as vCenter.

I told him this was preposterous, as vCenter install SQL Express by default if you don't have an existing DB.

I obliged him anyway, and I installed my SQL server on another VM, created a new DSN pointing to the remote DB, and installed vCenter on a fresh install of Windows 2008 R2 + VMware tools.

Well, the problem is still there. The VMwareVCMSDS writer is always reporting an error 11 ( 'vssadmin list writers' at DOS prompt), and trying to take quiesced snapshots ends with an error message:

"Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine."

I reported this back to the tech rep, and I'm still waiting. I sure hope they will elevate this because in my opinion, it is unthinkable that I should not be able to snapshot my vCenter VM.

Regards

[pciccone]

Locmorin, we are in the exact same situation at the moment. For me, this is fresh "text book" install of a new vSphere cluster, having this issue "out of the box". let me know what they ultimately tell you as well. Without disabling the whole VSS system, I cannot find a way to exclude the ADAM Writer only despite my many attempts.

I can also confirm that the original tech speculation that SQL on the same box could be the cause is not the case, as our install has a dedicated SQL server, similar to your new install. I can't believe that was even suggested - what does an ADAM writer have to do with SQL?

One more thing, it also affects their VMWare View product line as well, another fresh install server that uses the same VmwareVCMSDS service causes the same VSS ADAM writer problems.

Thanks.

-- Phil

[RobMokkink]

I have sql installed inside vm's and i can backup using vRanger without any probs.

As long as i set enblediskuuid to false (see previous reply)

Can you run a chkdsk inside the vm? This can help solve VSS problems, because VSS problems are also a big issue on physical servers and using backup solutions like MS DPM.