riki78
Contributor
Contributor

Problem with quiesced snapshot on Server 2008 R2 with ESXi 4.1 U1

Hello

Since we have updatet from ESXi 4.1 to Update 1 we have problems with taking quiesced snapshot. Before we have installed the update 1 quiesced snapshot worked witout problems.

When we take a quiesced snapshot, Windows 2008 R2 mount a new volumes and remove them after the snapshot is taken.

In the Eventlog i can see following errors and infos:

multiple entrys from this 2 event:

Log Name:      System
Source:        Virtual Disk Service
Date:          11.04.2011 22:40:26
Event ID:      3
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A

Description:
Service started.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Virtual Disk Service" />

and

Log Name:      System
Source:        Service Control Manager
Date:          11.04.2011 22:40:26
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      TESTTEMPLATE.gef.be.ch
Description:
The Virtual Disk service entered the running state.

and

Log Name:      System
Source:        Ntfs
Date:          11.04.2011 22:20:57
Event ID:      57
Task Category: (2)
Level:         Warning
Keywords:      Classic
User:          N/A
Description:
The system failed to flush data to the transaction log. Corruption may occur.

and

Log Name:      System
Source:        Ntfs
Date:          11.04.2011 22:20:57
Event ID:      137
Task Category: (2)
Level:         Error
Keywords:      Classic
User:          N/A
Description:
The default transaction resource manager on volume \\?\Volume{d533451f-6478-11e0-94e8-005056ae007a} encountered a non-retryable error and could not start.  The data contains the error code.

and

Log Name:      Application
Source:        VSS
Date:          11.04.2011 18:05:48
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      G3110SRV014APL.gef.be.ch
Description:
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000418,0x00560000,0000000000000000,0,000000000027CDB0,4096,[0]).  hr = 0x80070001, Incorrect function.
.

Operation:
   Exposing Recovered Volumes
   Locating shadow-copy LUNs
   PostSnapshot Event
   Executing Asynchronous Operation

Context:
   Device: \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
   Examining Detected Volume: Existing - \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
   Execution Context: Provider
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-7265-2056-5353-2050726f7669}
   Current State: DoSnapshotSet
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">12289</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-04-11T16:05:48.000000000Z" />
    <EventRecordID>6430</EventRecordID>
    <Channel>Application</Channel>
    <Security />
  </System>
  <EventData>
    <Data>DeviceIoControl(\\?\fdc#generic_floppy_drive#6&amp;2bc13940&amp;0&amp;0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000418,0x00560000,0000000000000000,0,000000000027CDB0,4096,[0])</Data>

When we remove the snapshot the vm freeze for about 30second (no Ping no RDP)

Have anyone the same issue?

I have try to install a new clean server 2008 r2 but have the same problem.

Best Regards
Simon

0 Kudos
59 Replies
idle-jam
Immortal
Immortal

have you updated with the latest vmware tools (just in case you might have missed that out)?

0 Kudos
riki78
Contributor
Contributor

Hello

Yes we have the last vmware tools installed.

0 Kudos
mulio
Contributor
Contributor

Hey there!

We got the same problem @ vsphere 4.1 u1 and after a lot cups of coffee´s later, a workaround to fix this issue.

We are thinking that the problem is vmware-tools related! :smileygrin:

First wie opened the "Volume Management" @ the VM that, and then we startet a quiesced snapshot.

10 sec after the start of the snapshot - we have seen the same system volume from our VM as a clone for 3 seconds :smileyconfused:.

After 3 seconds the cloned system volume disappears. Now we got a fail @ the event logs in our server.

After this show, we are uninstalled the vmware tools and installed the tools from version 4.1.

After this - quiesced snapshots works fine and vDR works too.

1. Uninstall VMware Tools 4.1 U1

2. Install VMware Tools 4.1

3. :smileylaugh:

Greetz

0 Kudos
riki78
Contributor
Contributor

Hello

Thank you for replay Smiley Happy

I have now open a support case

Have you also open a support case by vmware? When Yes can you tell me the case number?

I have also found that when I tkake an snapshot the snapshot add 2 snapshotfiles files in the datastore.

Best Regards
Simon

0 Kudos
mulio
Contributor
Contributor

Cheerio Simon!:smileygrin:

No, we have not openend a case @ vmware.

We are waiting for the new tools version. After this we do some tests on a non critical vm´s with new vmware-tools.

Finaly the new tools will be installed on all other vm´s.

Interesting thing with the 2 snapshots in the datastore :smileygrin:. We looked not yet at the datastore while creating snapshots. The workarounds for some problems thats vmware related are very funny. e.g. ->  delete a vm from the inventory and after this add the vm to the inventory :smileydevil: -> yes, vmware is really funny :smileygrin:

greetz

0 Kudos
EL-JK
Contributor
Contributor

We are experiencing this problem while making quiesced snapshot via VDR.

Manual snapshots, no problem, allso with the quisced option turned on, tried them both.

Our problems allso started after upgrading the VMware Tools!

Any news on the support case?

0 Kudos
wadood
Contributor
Contributor

same problem here

windows 2008 r2 64 bit domain controller

anyone know how we can solve this?

0 Kudos
riki78
Contributor
Contributor

Hello

VMware Support tries currently to reproduce the problem in the Labs.

As soon I obtain a response I will poste again.

I think a good idea are when any people with the same problem open a case @ VMware Support.

My Case number are: 11057600304

Best Regards

Simon

0 Kudos
wadood
Contributor
Contributor

eiki78,

this is happening only on windows 2008 r2 domain controllers for you too? because i have several other windows 2008 r2 vms and i dont get this errors, get them only on the domain controller vm. and btw, this is not manual snapshots...this is during the time vdr 1.2 runs.

wadood.

0 Kudos
mulio
Contributor
Contributor

In our enviroment occurs this mainly on all windows 2008r2 sp1 machines (member-server) there are thin provisioned!

3DC´s

DC 1 - 2008R2 - sp1 - Thick provisioned- no problems with newest vmwaretools

DC 2 - 2008R2 - sp1 - Thick provisioned- no problems with newest vmwaretools

DC 3 - 2008R2 - sp1 - Thin provisioned - problem

Exchange2010 sp1 - @server 2008r2 sp1 - Thick provisioned - no problem

Again --> with the old vmware-tools from version 4.1 -> there are no problems with thin provisioned machines.

Hope it helps

Greetz

:smileygrin:

0 Kudos
wadood
Contributor
Contributor

my mistake, this error are happening on all windows 20008 r2 some not sp1 some sp1. mine are all thick provisioned. was just looking in application logs instead of system.

0 Kudos
EL-JK
Contributor
Contributor

.

0 Kudos
BrianHendrickso
Contributor
Contributor

I am having the same issue and would love to hear if anyone has a resolution to this. I am seeing the problem on all my ESX 4.1 U1 virtual servers running Windows 2008/R2. All are thick provisioned and have the 4.1 U1 Tools installed. I didn't have this issue prior to upgrading!

-Brian

0 Kudos
mulio
Contributor
Contributor

Hey Folks Smiley Happy

I´ve got some news about our snapshot problems.

1) The DeviceIoControl Error

Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000418,0x00560000,0000000000000000,0,000000000027CDB0,4096,[0]).  hr = 0x80070001, Incorrect function.



I solved this issue by disabling the floppy device @ the "Device Manager".

After this --> The error is gone away!

2) Some NTFS Errors - EventID 57 / EventID 137

Log Name:      System
Source:        Ntfs
Date:          11.04.2011 22:20:57
Event ID:      137
Task Category: (2)
Level:         Error
Keywords:      Classic
User:          N/A
Description:
The default transaction resource manager on volume \\?\Volume{d533451f-6478-11e0-94e8-005056ae007a} encountered a non-retryable error and could not start.  The data contains the error code.

For these error i haven´t found yet a possible solution - but maybe a explanation for this.

@ http://technet.microsoft.com/de-de/library/ee923636(WS.10).aspx i found some interesting thing


Source: http://technet.microsoft.com/de-de/library/ee923636(WS.10).aspx

If the shadow copy is successfully created, the Volume Shadow Copy Service returns the location information for the shadow copy to the requester. In some cases, the shadow copy can be temporarily made available as a read-write volume so that VSS and one or more applications can alter the contents of the shadow copy before the shadow copy is finished. After VSS and the applications make their alterations, the shadow copy is made read-only. This phase is called Auto-recovery, and it is used to undo any file-system or application transactions on the shadow copy volume that were not completed before the shadow copy was created.

This is a solution why a volume appears in the "Volume Manager" while a snapshot is taken and this could be the answer to the error´s thats ntfs related.

tjaster
Enthusiast
Enthusiast

Here is what we got from the VMware Support :

Uninstall VMware Tools. Install VMware Tools again with Custom Installation and do not install VSS Driver. See attached Screenshot. So the problem is sitting inside the VSS driver used to quiesced the machine.

newinstall.png

--------- VMware Certified Professional 3/4
0 Kudos
mulio
Contributor
Contributor

Hi tjaster,

if you disable vss support - you´ll disable quiesced snapshots and the result of this is an backup which is not application consistent.

tjaster schrieb:

Here is what we got from the VMware Support :

Uninstall VMware Tools. Install VMware Tools again with Custom Installation and do not install VSS Driver. See attached Screenshot. So the problem is sitting inside the VSS driver used to quiesced the machine.

14087_14087.pngnewinstall.png

0 Kudos
riki78
Contributor
Contributor

Hello

This is the last responds from VMware Support:

Regarding the following:

(EventID 137 ntfs), EventID 57 ntfs) (EventID 12289 vss)

These can be safely ignored, That's a false alarm from the NTFS driver.

event 12289 is in normal operation.  I believe it happens when VSS is querying for supported disks and sees our floppy drive and VSS will skip it.

engineering have reproduced the other two, they are the result of a fix engineering made to allow quiesced snapshots on VMs with an ADAM or NTDS instance. please see this: http://support.storagecraft.eu/kb_details.aspx?id=KB10059, which looks like they made the same fix we did and the resulting event logs show up as well.  However, I could not find anything from Microsoft about this.

Engineering took a quiesced snapshot and ran chkdsk both after the snapshot and after reverting to the snapshot and they both came up clean.

So, the messages can be ignored.

Best Regards

Simon

0 Kudos
tjaster
Enthusiast
Enthusiast

OK i think this problem is fixed now. We just checked the updates beeing provided with update manager. See here :

  • If you are using a backup application that utilizes Changed Block Tracking (CBT) and the ctkEnabled option for the virtual machine is set to true, the virtual machine becomes unresponsive for up to 30 seconds when you remove snapshots of the virtual machine residing on an NFS storage.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=103510...

We will try to install the update and revert for feedback

--------- VMware Certified Professional 3/4
0 Kudos