Windows 2008R2 snapshot problem on 4.1

[AntonVZhbankov]

I have problem with snapshots on one of my VMs. And what's strange it started only last friday.

One of my DC's on Win2008R2 refused to take quiesced snapshot after clod migration from one cluster to another. Remaining DC, absolutely the same configuration still can take snapshots.

I've made new VM, built from nothing, clean installation, promoted to DC instead of failed on and problem still persists.

What I've found in Windows logs:

1) Volume Shadow Copy Service error: Unexpected error DeviceIoControl(
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 000000000000042C,0x00560000,0000000000000000,0,00000000003C7F70,4096,[0]). hr = 0x80070001, Incorrect function.

.

Operation:

Exposing Recovered Volumes

Locating shadow-copy LUNs

PostSnapshot Event

Executing Asynchronous Operation

Context:

Device:
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Examining Detected Volume: Existing -
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Execution Context: Provider

Provider Name: VMware Snapshot Provider

Provider Version: 1.0.0

Provider ID: {564d7761-7265-2056-5353-2050726f7669}

Current State: DoSnapshotSet

2) lsass (460) An attempt to write to the file "
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log" at offset 7525888 (0x000000000072d600) for 512 (0x00000200) bytes failed after 0 seconds with system error 19 (0x00000013): "The media is write protected. ". The write operation will fail with error -1032 (0xfffffbf8). If this error persists then the file may be damaged and may need to be restored from a previous backup.

3) lsass (460) Unable to write to logfile
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log. Error -1032 (0xfffffbf8).

---

MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

[MKguy]

Have you also looked into the UUID mismatch problem discussed here in this thread:

http://communities.vmware.com/message/1595294#1595294

Might be it if you have the same problems with the vCenter ADAM instance the VMWareVCMSDS service runs..

[jb238]

We have 4 Win2008R2 DC VMs and use VDR 1.2. After upgrading to 4.1, only one DC had this problem.

I noticed that the first snapshot after a reboot works. Subsequent snapshots fail until the next reboot.

[TK22]

As soon as i get the time i´ll look into that UUID mismatch thing.

Our workaround right now is just to disable the VMware Snapshot Provider service in the vCenter VM.

Then we can both snapshot and VDRbackup the vCenter 4.1 server on 2008R2

[Borja_Mari]

Hi,

the same workaround (it seems that can happens in w2k3 too) is explained here too.

Regards/Saludos,
Pablo

Please consider awarding

any helpful or corrrect answer. Thanks!! -

Por favor considera premiar

cualquier respuesta útil o correcta. ¡¡Muchas gracias!!

Virtually noob blog

[ITWheel]

OK Everyone, let me know if this does not work for you.

The VMware Snapshot Provider is the problem. Reviewing and implementing

the fixes in the current version of the following VMware KB does not

resolve the issue. They may fix it soon when they see this.

Cannot create a quiesced snapshot because the snapshot operation

exceeded the time limit for holding off I/O in the frozen virtual

machine

Use this method if you are getting the following error in the system

event log when trying to start the VMware Snapshot Provider Service in

the guest VM:

Error 1053: The service did not respond to the start or control request in a timely fashion VMWare Snapshot Provider

Also, you will probably be getting the following error when trying to

take a snapshot of the VM with Memory unchecked and Quiesce checked or

when trying to do a VDR backup:

Cannot create a quiesced snapshot because the snapshot operation

exceeded the time limit for holding off I/O in the frozen virtual

machine.

To fix the problem Uninstall and Reinstall VMWare Snapshot Provider as follows:

1. Uninstall Volume Shadow Copy Service (using Modify from VMware Tools setup)

2. Delete VSS folder from C:\Program Files\VMWare\VMWare Tools\Drivers

3. Delete VMWare Snapshot Provider Service by deleting the key

HKLM\System\Current Control Set\Services\vmvss (Just delete the vmvss

key)

4. Delete VMware Snapshot Provider

{564d7761-7265-2056-5353-2050726f7669} from HKLM\system\current control

set\services\VSS\Providers

(If you do not have this key, just omit this step.)

5. Reboot

6. Start the two COM+ services mentioned in kb.vmware.com/kb/1018194

7. Install Volume Shadow Copy Service (using Modify from VMWare Tools setup.exe)

8. Reboot

VMWare Snapshot Provider should now be Manual, and you can start it.You

should now be able to take a snapshot and back up with vdr while the VM

is powered on.

( Also see this Symantec article with a full step by step walkthrough of steps 1-4: )

Marcus

IT Wheel

www.itwheel.com

Consider awarding points for "helpful" and/or "correct" answers.

[hugo5113]

i had the same error.

enviroment:

vSphere 4.1

W2K8R2

DC

Backup with VDR

i repaired it:

1.) uninstall only vmware-tools-option VSS-driver...(see attatched file)

2.) backing up registry

3.) regedit: deleting all vmware-related(!) entries with substring "vcbsnap", e.g. CLSIDs (by using /search...)

4.) reboot

5.) reinstall vmware-tools, userdefined/change, VSS-driver

6.) reboot

it works fine

regards Peter

[KEPete]

FYI, not sure how many people on here use DFS but my Windows 2008 R2 Standard DFS boxes are also showing VSS errors after being backed up by VDR. It does not show errors in VDR but shows on the server's event logs. What's funny is that when I first upgraded my hosts to 4.1, VDR would not backup at all if it had AD or ADLDS installed. but the others were fine. After I removed VSS and reinstalled VMTools, VDR backups started running but backups were no good for AD. I installed VSS again and backups still ran. But VSS errors continue. Has anyone tested restores with these various fixes being posted here?

[KEPete]

Hugo, I've also tried reinstalling the VSS drivers and have been running successful VDR backups as far as VDR is concerned. However, did your VSS errors go away in your Windows event logs? I started noticing these issues back in July: http://communities.vmware.com/message/1577208#1577208

So far, nothing seems to really work as far as getting a real backup of AD. I'm currently using Microsoft's built in tool for backing up the System State to protect myself.

[bingofox]

I have the exact same issue.

ESXi 4.1

vCenter 4.1

All VM's (mix of 2008 and 2003) created from scratch on 4.1

All of them work and can be quisced prior to taking a backup except those running A/D, SQL, or any relational Database for that matter.

It also affect Windows 2003 Servers as well - not just 2008.

I'm wondering why there still isn't a hotfix/patch for this?

[mdgunther]

4.1 U1 came out on 2/10 and look what's in the release notes:

* Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1

When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

I'll hopefully be updating before the month is over to see that it works.  After the update, I'll have to undo the workaround.  It's been so long, I can't remember what it was!

[randomotten]

I hate to say this. But we did the update and the problem is still there.

We use Acronis Virtual Edition with the virtual Appliance and still get this kind of errors:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Entry Details
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type:          Error
Date and time: 2/14/2011 12:16:04 PM
Backup plan:   Backup All Virtual Machines
Task:          VCENTER01 - Incremental backup
Code:          5,439,591(0x530067)
Module:        83
Owner:         Administrator@BKV-GROEP
Message:      
  Failed to open the virtual machine ([BIGFOOT] VCENTER01/VCENTER01.vmx).
Additional info:
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91dcf6
Fields:
Message: Failed to open the virtual machine ([BIGFOOT] VCENTER01/VCENTER01.vmx).
--------------------
Error code: 138
Module: 83
LineInfo: a859dd78cc91dce8
Fields:
Message: VMware failed to create a quiesced snapshot of the selected virtual machine. To back up the machine, power off the machine before the process or uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the VM.
--------------------
Error code: 138
Module: 83
LineInfo: c7610e0a857bed85
Fields:
Message: Awaiting task 'CreateSnapshot' has failed. Reason: Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine..
--------------------
Acronis Knowledge Base: http://kb.acronis.com/errorcode/

Event code: 0x00530067+0x0053008A+0x0053008A

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[frankLT]

Hello,

we had exactly the same problem on our vCenter server. Upgrading from ESX 4.1 to U1 and installing the appropriate version of VMware Tools in our vCenter VM solved the issue. Cloning and VDR backup using quiesced snapshot now succeed. Unfortunately, there are some errors/warnings left, so I don't exactly know if we get fully consistent snapshots, especially because of NTFS warnings.

Always numerous warnings of this kind:

Protokollname: System
Quelle:        Ntfs
Datum:         23.02.2011 14:14:15
Ereignis-ID:   57
Aufgabenkategorie:(2)
Ebene:         Warnung
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      LTVC1.domain.de
Beschreibung:
Die Daten konnten nicht in das Transaktionsprotokoll verschoben werden. Die Daten sind möglicherweise beschädigt.
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Ntfs" />
    <EventID Qualifiers="32772">57</EventID>
    <Level>3</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-02-23T13:14:15.176483200Z" />
    <EventRecordID>20355</EventRecordID>
    <Channel>System</Channel>
    <Computer>LTVC1.domain.de</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Binary>0000000001000000020000003900048000000000100000C000000000000000000000000000000000</Binary>
  </EventData>
</Event>

And always this error:

Protokollname: System
Quelle:        Ntfs
Datum:         23.02.2011 14:14:15
Ereignis-ID:   137
Aufgabenkategorie:(2)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      LTVC1.domain.de
Beschreibung:
Auf dem Volume "\\?\Volume{5f561ae3-3f4d-11e0-8393-0050568115d9}" konnte der Transaktionsressourcen-Manager aufgrund eines nicht wiederholbaren Fehlers nicht gestartet werden. Der Fehlercode ist in den Daten enthalten.
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Ntfs" />
    <EventID Qualifiers="49156">137</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-02-23T13:14:15.205769200Z" />
    <EventRecordID>20360</EventRecordID>
    <Channel>System</Channel>
    <Computer>LTVC1.domain.de</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>\\?\Volume{5f561ae3-3f4d-11e0-8393-0050568115d9}</Data>
    <Binary>1C0004000200300002000000890004C000000000100000C000000000000000000000000000000000100000C0</Binary>
  </EventData>
</Event>

I just upgraded the OS (W2K8R2) to SP1. Generally running fine, but previous described messages are still there. Anyone else having the same problem?

Best regards,

Frank

[mdgunther]

I updated to 4.1U1 yesterday and it appears to have fixed the problem for me.  I was able to take a snapshot without error and vDR successfully backed up the vCenter Server VM.  I had to set the EnableUUID back to true to enable the VSS snapshots.  Support had advised me to remove the VM from the inventory and re-add it when I set it to false (the workaround until the bug was fixed), so I did the same for the true edit.

To be clear, I confirmed with support that this is an ESX/Vmware-tools level problem and not a vCenter Server problem.  Just updating the vCenter Server isn't going to fix it, you have to do everything.

[kipz]

hello,

i have also upgraded to the latest binary but the problem is still there. and for me, even few Windows 2003 VMs fails with the same error.

i have also noticed that sometimes rebooting VM helps for a short period. right after reboot i can take quiesced snapshot few times. but eventually it starts to fail again.

regards,

kipz

[wadood]

frankLT,

I too have the same error messages as you describe. on a windows 2008 r2 64 bit domain controller. have you had any luck with knowing the cause of this or fixing it?

[frankLT]

Hey wadood,

no, not so far. But this problem is obviously not specific to certain server roles or applications. Each of our 2008 R2 VMs produce these same errors when being backed up with vdr. But those event log entries are the only problem I noticed. Backup jobs are running fine, and VMs can be restored without any problem.

Frank

[wadood]

hi frank,

thanks for the answer. yes, i just confirmed they are on all windows 2008 r2 vms. i was not looking in the system logs for the rest.

at least you confirm the vdr backups are fine, thats what was scaring me.

wadood.

[tsinfra]

Hey guys,

I'm running ESXi 4.1 build 381591 and still having VMware VSS probs on a W2K8 R2 SP1 virtual machine.

Uisng Symantec Netbackup 7.1.0.1 to do application consistent / file quiesced backups of multiple W2K3 R2 and W2K8 R2 SP1 vm's.

The 2003 vm's snapshot ok, however the 2008 snapshot always failed with the I/O message displayed.

Worked with Symantec who in the end advised me to completely uininstall VMware VSS and use the Symatec VSS provider instead;

Symantec notes:

Install the symantec vss provider in the VM .

Refere the guide.

http://www.symantec.com/business/support/index?page=content&id=DOC3663

Page 167 onwards.

This seems to have worked!

I've logged an SR with VMWare, but I don't think the VMware VSS provider installed as part of a clean vmware tools install on a 4.1 U1 build 381591 host works with W2K8R2!

Thought I'd share what I'm finding.

[TK22]

Jag är borta på semester och kommer troligast inte att läsa e-post förrän jag förhoppningsvis är tillbaka igen, Onsdagen den 7:e September.

För teknisk assistans hänvisar jag till kontoret på 060 - 14 95 00.

Mvh

/ Kenneth

I´m on vacation and will probably not read mail until i´m back, Wednesday September 7:th.

Kind Regards

/ Kenneth

[rb51]

hi all,

we still get this error on our vcentre server.

event ID: 12289, 8193

hosts: 4.1U1

vcentre: 4.1U1

OS: Windows 2008 R2

vmware tools: 8.3.7 build 341836 (U1)

Just wondering what you guys would suggest. We will soonish be applying U2 to our hosts.

Help is much appreciated.

rb51