riki78
Contributor
Contributor

Problem with quiesced snapshot on Server 2008 R2 with ESXi 4.1 U1

Hello

Since we have updatet from ESXi 4.1 to Update 1 we have problems with taking quiesced snapshot. Before we have installed the update 1 quiesced snapshot worked witout problems.

When we take a quiesced snapshot, Windows 2008 R2 mount a new volumes and remove them after the snapshot is taken.

In the Eventlog i can see following errors and infos:

multiple entrys from this 2 event:

Log Name:      System
Source:        Virtual Disk Service
Date:          11.04.2011 22:40:26
Event ID:      3
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A

Description:
Service started.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Virtual Disk Service" />

and

Log Name:      System
Source:        Service Control Manager
Date:          11.04.2011 22:40:26
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      TESTTEMPLATE.gef.be.ch
Description:
The Virtual Disk service entered the running state.

and

Log Name:      System
Source:        Ntfs
Date:          11.04.2011 22:20:57
Event ID:      57
Task Category: (2)
Level:         Warning
Keywords:      Classic
User:          N/A
Description:
The system failed to flush data to the transaction log. Corruption may occur.

and

Log Name:      System
Source:        Ntfs
Date:          11.04.2011 22:20:57
Event ID:      137
Task Category: (2)
Level:         Error
Keywords:      Classic
User:          N/A
Description:
The default transaction resource manager on volume \\?\Volume{d533451f-6478-11e0-94e8-005056ae007a} encountered a non-retryable error and could not start.  The data contains the error code.

and

Log Name:      Application
Source:        VSS
Date:          11.04.2011 18:05:48
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      G3110SRV014APL.gef.be.ch
Description:
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000418,0x00560000,0000000000000000,0,000000000027CDB0,4096,[0]).  hr = 0x80070001, Incorrect function.
.

Operation:
   Exposing Recovered Volumes
   Locating shadow-copy LUNs
   PostSnapshot Event
   Executing Asynchronous Operation

Context:
   Device: \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
   Examining Detected Volume: Existing - \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
   Execution Context: Provider
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-7265-2056-5353-2050726f7669}
   Current State: DoSnapshotSet
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">12289</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-04-11T16:05:48.000000000Z" />
    <EventRecordID>6430</EventRecordID>
    <Channel>Application</Channel>
    <Security />
  </System>
  <EventData>
    <Data>DeviceIoControl(\\?\fdc#generic_floppy_drive#6&amp;2bc13940&amp;0&amp;0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000418,0x00560000,0000000000000000,0,000000000027CDB0,4096,[0])</Data>

When we remove the snapshot the vm freeze for about 30second (no Ping no RDP)

Have anyone the same issue?

I have try to install a new clean server 2008 r2 but have the same problem.

Best Regards
Simon

0 Kudos
59 Replies
tjaster
Enthusiast
Enthusiast

The problem still persists. We will open a new ticket to VMware Support and keep this community article updated.

--------- VMware Certified Professional 3/4
0 Kudos
aafcu
Contributor
Contributor

Are there any updates on this issue?  We are struggling to find a solution to this problem as well.

0 Kudos
max70
Contributor
Contributor

Same problem here, on two different Esxi 4.1 U1 systems

0 Kudos
Bubs
Contributor
Contributor

The work around for this is to just install the old Vmware tools on your problem machines as previously posted in this thread.

0 Kudos
BrianCHW
Contributor
Contributor

Keep in mind to never use snapshots on AD controllers, AD does not play well with snaps and cannot be used to restore them.

Brian Kirsch Instructor, MATC VMUG Global Board of Directors M.Ed, VCP3, VCP4-DV, VCP5-DCV, EMCISA MCP, BADDC, CCA, VAI, VCC & vExpert 2013
0 Kudos
markusepp
Contributor
Contributor

We had the same problems again and got the below answer from VMWare Support:

Hope it also helps solve your problem.

"I have seen that the majority of the failures are Win2k8 VM's can you please try the following on the Win2k8 VMs that you are having issues with.

a) Power off the Windows Server 2008 virtual machine through vSphere Client.
b) Click Settings > Options >General > Configuration Parameters.
c) Set disk.EnableUUID to False.
d) Save the settings.
e) Power on the virtual machine.

This appears to be a common problem backing up Win2k8 VMs which the above steps normally fixes"


0 Kudos
riki78
Contributor
Contributor

Hello

News from the VMware Support:

Here is a quick update on where our engineering team are with Microsoft regarding those VSS errors we see in ESX 4.1 and Windows Server 2008.

Engineering have confirmed that the fix for this will will be available with ESX 4.1 update 2 (due in late Q3 2011 or early Q4 2011).

While the messages were never causing an issue with the VM or its data the fix will ensure those messages don't show up in the Windows Event Viewer again.

As we have a confirmed fix due shortly are you happy to close our this SR or do you still have any questions or concerns regarding the issue or the fix which is due shortly?

Best Regards

Simon

0 Kudos
vmbud
Contributor
Contributor

By default Win 2008 R2 / Win 7 creates 100MB system partition (no drive letter), the rest OS in C: drive.

That is what creates quiesced snapshot failure.

1.JPG

Removing system reserved partition will solve the problem.

For existing VM’s you can use instructions from here:

http://www.geekshangout.com/?q=node/103

If you want new VM template without this partition you can use similar  instructions from here:

http://www.mydigitallife.info/hack-to-remove-100-mb-system-reserved-partition-when-installing-window...

- leave VM  tolls shadow copy service installed

- leave disk.EnableUUID to true

In my scenario VDR backups and application quiesecing is working great including DB backups.

Performing quiesced snapshots on Win 2008 R2 event viewer logs VSS errors for floppy drives.

(I think someone mentioned already)

- go to Device manager

- disable floppy drive, then disable floppy disk controller (floppy drive disappears in Device manager)

In Win 2008 R1 there is error 12, I guess nothing can be done with it, LOL ignore it…

http://support.microsoft.com/kb/958669

I hope this helps,

Cheers

0 Kudos
BrianHendrickso
Contributor
Contributor

I am currently out of the office and will return Monday, October 3rd. I will intermittently check and respond to emails and voicemails during this time.

-Brian

0 Kudos
gtlscot
Contributor
Contributor

Hi

Does anyone know if is resolved as we appear to be having some strange situations with a couple of 2008 R2 boxes since we moved to 4.1 update 1.  We are actually having to go into the VM and add a new drive and copy data from the previous one has it has become read only

Thanks

0 Kudos
wentworth33
Contributor
Contributor

I'm having the same problem.  When our backups run, the F: drive on two of my Windows 2008 R2 VM's dissappears.

I need to go into Disk Management and reassign the drive letter.  Then I need to use diskpart to remove a read only flag from the volume.

This happens every time the backups try and run.

I'm getting VSS event ID# 12289 and Ntfs events with ID#s 57, 136, and 137.

These are thick provisioned member servers on an ESXi 4.1 cluster running current VMTools.

I really need this fixed, but I'm not sure how.  Any good ideas?

0 Kudos
gtlscot
Contributor
Contributor

Wentworth.  This is exactly what we are having.  I have raised a ticket with support so hoping for a quick resolution

0 Kudos
markusepp
Contributor
Contributor

Have you tried my above suggestion about the disk.EnableUUID setting?

Having the same problem this has really solved it for us...

0 Kudos
gtlscot
Contributor
Contributor

Not yet.  Will look into this whilst waiting on support.

0 Kudos
gtlscot
Contributor
Contributor

Tried this setting now.  Support provided the following article which refers to your suggestion.  I have asked when engineering will be releasing a fix

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=103129...

0 Kudos
wentworth33
Contributor
Contributor

Did that fix your problem?  I haven't tried it yet, and have rather just stopped backing up the two bad VMs.  Let me know your results and maybe I'll give it a try too.

0 Kudos
tjaster
Enthusiast
Enthusiast

We did a testing on fresh released 4.1 U2 Version. The error still persist if you do not make any changes on the virtual machine. Any news from VMware?

--------- VMware Certified Professional 3/4
0 Kudos
vmbud
Contributor
Contributor

Did anyone tried to use the info I posted October 1st?

0 Kudos
gtlscot
Contributor
Contributor

Yes, from the testing we have conducted we are happy that it has been resolved.  Still awaiting proper patch however VMware have stated that it is quite a difficult one to correct

0 Kudos
gtlscot
Contributor
Contributor

Yes,  we discovered this issue a while back and removed the 100MB partition.  After we done this we upgraded to Netbackup 7.1 which did not encounter this problem anyway.  The issue we were having recently was completey different.

0 Kudos