Bug - Snapshot of 2012R2 VM Takes 10min When Shadow Copies are enabled

[DrWhy]

When taking a snapshot of a running Server 2012R2 VM with the option "quiesce guest file system" checked, it takes 10min.  That status of the task hits 100% after only 30 seconds, but it does not show as "Complete" until exactly 10min after the status hits 100%.  It does this every time.  It appears to be hitting some sort of timeout.  The cause for this is that within the Windows OS, I have Shadow Copies configured to be saved to a volume other than itself.  Please see the image below.  In this situation Shadow Copies for volume E are being saved to volume T.  If shadow copies for volume F are being saved to volume F, this problem DOES NOT occur.  This problem also DOES NOT occur if I select snapshot Virtual Machine Memory nor does it occur when the VM memory and Quiesceing are left unchecked.  I have taken screen shots of the VSS errors I receive in the VM.  I can recreate this problem on 2 different hosts with different VM's.  Both ESXi 6 hosts have the latest ESXi patch that fixes various VSS issues.

I have three questions:

1. Is snapshoting a Windows VM with Shadow Copies enabled on a secondary volume supported?

2. This operation eventually completes, but because of the errors, is it really completing correctly?  Is the data really getting quiesed properly?

3. Is there any way to fix this?  Please advise

[cykVM]

The first thing which jumps into my eyes is that the 3rd screenshot of error event 12289 VSS mentions "fdc#generic_floppy_drive" as an accessed volume.

Maybe removing the floppy drive from the VM helps to solve at least this.

Also check if VSS is really working following the troubleshooting hints (especially step 3 and 4) in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100769...

As mentioned in the notes for Windows 2012 of above KB also check if DTC (ditributed transaction coordinator) service is running.

If you have some backup software handy (e.g. Windows Backup) you may try a backup from within the VM to check if VSS is working properly inside Windows.

Last but not least maybe a re-installation of VMWare Tools might help.

[DrWhy]

The floppy drive error is benign.  I've removed it on the other VM and the problem continues.  VSS is working fine as a quiesed snapshot works fine if shadow copies are not enabled on a secondary volume. Furthermore, I've reproduced this problem on 4 VM's across 2 different hosts, all with the same results.  I believe that is enough to rule out VSS, vmware tools, and pretty much all other possibilities and narrows it down to how the VMware VSS driver talks with the Microsoft VSS driver.

[DrWhy]

Does anyone have additional info on this?

[FrankMunich]

ESXi6.0 + latest patch:

No time difference when creating snapshots with or without Shadow Copies enabled, also no Errors in the Event Viewer. 2012 R2 as well.

In short, I cannot reproduce your issue. On a side note, does it make sense to use shadow copies in a virtual Environment?

[DrWhy]

Did you have the shadow copies set to be stored on a secondary volume?  It can be nice to have shadow copies enabled as it supplements our backups in a manner of speaking.

[FrankMunich]

Yes, my 2nd disk is E:\ and that's where I created the shadow copies. No problems and no errors. It's a Windows 2012 R2 that was created in vSphere from scratch (boot from ISO) and has a lot of data and applications on it.

[DrWhy]

I don't know what to say... I've recreated this error on 4 different VM's and 2 different hosts.  Any ideas?

[FrankMunich]

My Versions are:

vSphere Client

Version 6.0.0
Build 2502222

VMWare ESXi

Version 6.0.0

Build 2715440

German-000

The latest patch is ESXi600-201505001.zip - did you install that?

[DrWhy]

I have the same ESXi version as you.  I used the vSphere Web Client - Version 6.0.0 Build 2559277.  Do you think that could make a difference?  What version of VMware tools are you running?

[FrankMunich]

I think it is unlikely that the client Version affects the internal shadow copy handling. I can't find my vmware login details in the moment, otherwise I would have tested the latest vsphere Client as well.

I would guess this is more a Hardware/Driver issue on the ESXi Server itself. What Server/Hardware are you using for ESXi?

I am using a HP ProLiant Microserver N54 for which customized vSphere ISO Images are available. That means, I have the best Drivers and files specific for this Server.

If you use the Standard vSphere installer on legacy Hardware, this might be some sort of a Driver/Hardware issue.

[DrWhy]

Both hosts are HP ML350p Gen8 Servers.  I am using the HP ESXi image.

[FrankMunich]

You might check if there is a BIOS update or contact HP Support/Vmware Support in parallel. It appears to be a specific issue on this Server type then.

Maybe someone else using this Server can test this as well.