VMware Cloud Community
Supi_du
Contributor
Contributor

Vmware Tools defect since Version 10.x.x with server 2012R2 msg.snapshot.error-QUIESCINGERROR

Hello Community,

I am facing with a Snapshot Problem with one Single Server 2012R2 VM with Exchange 2013.

Environment:

Dell R430, ESXi 6.0 U3 5572656.

The Error Message is the following and the Snapshot failes:

Create virtual machine snapshot VM-Exchange

An error occurred while saving the snapshot:

msg.snapshot.error-QUIESCINGERROR.

I hoped the latest update fixed this as the release notes says:

VMware Tools 10.1.10 Release Notes

Quiesced snapshots of Windows Server 2012 and Windows Server 2012 R2 virtual machines with VMware Tools 10.1.0 fails with an error.

This issue is resolved in this release.

But again last night, the Problem still exist.

To be more complicated, the Problem is rather intermitted.  3-4 Backups with snapshot works, the error occurs.

I just testet 3 Snapshot after upgrade to 10.1.10 an "no error" , ? Problem solved? ... not yet.

So now i am back with VmTools 9.10.5.

Im gettting this Problem with every Tools Version 10.x .  After 5-7 Days (Nightime Backup Windows)  the Problem occurs,

And before my Test of Version 10.1.10 yesterday the Backup runs for about 4 Weeks with Version 9.10.5 without any snapshot error.

So for me it is rather clear the Problem has to do with Version 10.x and some changes.

Is there someome with a hint?  🙂

16 Replies
daphnissov
Immortal
Immortal

Although it may not be helpful in your case, here's a hint:  Don't rely on VMware's quiescing ability. It has been notoriously problematic for years and continues to be today. My recommendation is that whatever backup application you're using, use an in-guest agent to signal quiescing and don't use the ability built-in to VMware tools. If you're using something like Veeam where they built their own engine to avoid issues exactly like this, make sure you turn on Application Aware Image Processing.

Reply
0 Kudos
StephenWagner7
Enthusiast
Enthusiast

I did a blog article on this one:

vSphere ESXi 6.5 – Unable to take quiesced snapshot | The time I've wasted on technology...

I usually see 2 things cause this, but this is the most usual culprit.

Log on to the 2012 VM, and delete the volume shadow copies. Restart the VM (may not be needed), and then try taking the snapshot.

I see this occur when the VM has TONS of volume shadow copies. I regularly delete a bunch of mine to keep things running fast and smooth (and to allow the snapshots to work).

Reply
0 Kudos
Supi_du
Contributor
Contributor

Hi,

thanks for your answer.

I try to get some further input.

I use Quest Vranger 7.5 as Backup Program. The Backup also includes application aware processing via vzshadow.exe

When and how to use quiescing with SQL or Exchange servers and other VSS aware applications with (82...

But Vmware should also be Application Aware since the parameter "disk.enableUUID=true"  is possible.

And I deal since ESXi 4.1 with this option, as it was not working very well in the beginning with Server 2008 VM's and Vmware DataRecovery.

The other hint with "delete shadow copies": For this Vm shadow copies are disabled.

As i notet before, the snapshot success rate is rather inconsistent with the 10.x branch. But i have to qualify it to this "specific" VM. 2 other 2012R2 VM's  ( One simple Domain Controller and one File Server with deduplication enabled) work flawless with every 10.x VMware Tools-Version.

Just to clarify: After install Version 10.1.10 yesterday 3 test snapshot worked. Then the first Backup Job also was successful. But the during the second job ( Backup to another repository), i got again this error message. Vsphere client shows that vranger tries 3 times to take the snapshot.

When i then restart the VM, Backups working for 3-5 days.

Tis our Mail Server VM and therefore an essential VM, I am not able to do here extensive Tests. Especially because a VMTools Downgrade is not that much Fun.

(Don't uninstall VMWARE Tools with paravirtual SCSI Disk and reboot without reinstall the old tools. My Exchange just missed the Data Disk 😞   )

Last Time i tested Version 10.0.17 for about 3 Weeks. And not later that 6-7 Day's I got this error message during VM Backup.

Right after going back to 9.10.5 the Problem disappears . (for the last 4 Weeks). Testing Version 10.1.10...  Problem/Error is back again.

Reply
0 Kudos
StephenWagner7
Enthusiast
Enthusiast

I would recommend using the command prompt on the VMs to check the health of your VSS providers...

I'm wondering if 3rd party backup software, or other miscellaneous software is interfering or causing issues with the VMs VSS providers.

Also, some backup applications actually disable VMware tools VSS providers. I would check the 3rd party documentation as it probably might explain what's causing these issues...

Reply
0 Kudos
Supi_du
Contributor
Contributor

I verfied according this guide:

How to check the Volume Shadow Copy Service (VSS) provider/writer status.

All Providers are ok and have status stable.

Also, either with or without vzshadow,exe as "Backupscrip.d" , the snapshot error happens. Already when i trigger a snapshot within the Vsphere Client.

(4-5 Times ok, then Error "msg.snapshot.error-QUIESCINGERROR" occurs.) Not only when i Backup with vranger.

Reply
0 Kudos
StephenWagner7
Enthusiast
Enthusiast

Sorry for the late reply...

I just noticed you mentioned Shadow Copies were disabled. Could you try enabling them for the heck of it, restart the VM, and then try processing a backup?

Reply
0 Kudos
Supi_du
Contributor
Contributor

Hello,

I will try to test it this weekend. But is rather a pain to get the vm back to working tools.

At the Moment now, one Month after my last post, not a single error Message. "msg.snapshot.error-QUIESCINGERROR“ with vmtools 9.10.5.

One think i also noticed with the latest Tools 10.1.10 :

When login after reboot via RDP the first time, vmtools are not displayed in task bar. I tested all the reg repairs regarding this.

When I login via Vcenter Console, the Vmtools are visable, and also after that via RDP.

Reply
0 Kudos
Supi_du
Contributor
Contributor

Hello again,

another try with 10.1.10 and ShadowCopies, sadly another failure. Even with shadow copies enabled.

I give it up and stay with Tools 9.10.5.

First snapshot try ok, second failure. What a pity.

exchange failure.JPG

Reply
0 Kudos
nparas5
Enthusiast
Enthusiast

I have faced the same issue in my Virtual Infra and worked with VMware Technical support the same, below is what they suggests and it resolved the issue:

To resolve this issue, start the Windows Virtual Disk service:

  1. Log in to the Windows operating system as an Administrator.
  2. Click Start, type services.msc, and press Enter.
  3. Right-click the Virtual Disk service and click Start.

If this does not resolve the issue, disable VMware snapshot provider in the affected virtual machine:

  1. Disable the VMware Snapshot provider service in the guest operating system.
  2. Restart the Volume shadow copy service.

“msg.snapshot.error-QUIESCING-ERROR” in vCenter Server (2069952) | VMware KB

Sure it will help you in your case.

Reply
0 Kudos
Supi_du
Contributor
Contributor

Hello nparas5 and thanks for answering.

With enabling the VD Service i tried before with no luck.

I see, there is also a thread about the same Problem:

Unable to take quiesced snapshot

The Following  VMware KB Artikel: Quiesced snapshots of Windows 2012/2012R2 VMs with Tools 10.1.0 fails (2149640) | VMware KB says it is fixed with tools 10.1.10, what i can not confirm.

Reply
0 Kudos
nparas5
Enthusiast
Enthusiast

Have you tried the below steps as well?

If this does not resolve the issue, disable VMware snapshot provider in the affected virtual machine:

  1. Disable the VMware Snapshot provider service in the guest operating system.
  2. Restart the Volume shadow copy service.
Reply
0 Kudos
Supi_du
Contributor
Contributor

  1. Disable the VMware Snapshot provider service in the guest operating system.

Thats what i do not want to do.

I tried today a little bit more with Vmware Tools 10.1.10.

So far I can say for my environment the Problem is the Service VMSS, called Vmware Snapshot Provider.

Even if the service has the option "manual", the service will be started after reboot.

I tried the  PS-Script from  tomsie1000 here:  Unable to take quiesced snapshot

and also a shorter Version to stop VMSS after reboot.

sleep 60 #give the system time to start the services - may need tweaking for slower systems

if ((Get-Service -Name vmvss).Status -eq 'Running'){stop-service -inputobject $(get-service -Name vmvss)}

but this helps not 100% > All my test show no (full) consistent behaviour.

I can reproduce after reboot, the service is started. I stop VMSS. When i then take a new snapshot, this snapshot failes with the Message:

msg.snapshot.error-QUIESCINGERROR.

and VMware log:

vmx| I120: Msg_Post: Warning

vmx| I120: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.

vmx| I120+ The error code was: 5

vmx| I120+ The error message was: 'VssSyncStart' operation failed: IDispatch error #8454 (0x80042306)

vmx| I120: ----------------------------------------

vmx| I120: Vigor_MessageRevoke: message 'msg.snapshot.quiesce.vmerr' (seq 2641570) is revoked

vmx| I120: ToolsBackup: changing quiesce state: STARTED -> ERROR_WAIT

vmx| I120: ToolsBackup: changing quiesce state: ERROR_WAIT -> IDLE

vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE

vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot

vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (31).

The VMSS Service is running (again).

When i then stop VMSS and try a new snapshot, it is possible, that a snapshot works. But not in all tries.

When the snapshot is possible, the VMSS Service will be stopped 1-2 Minutes later. Also when i then delete the snashot, i have to wait 1-2 Minutes, that the service gets stopped. (trigger comes from vmware tools service? )

The Key seems the VMSS Service... but with this Tools Version the Service behaves like a DIVA. 🙂

I haven't indentify a reliable workaround.

As my snapshots are working right now, i will stay with this VM at the moment with tools version 10.1.10.  But I think it will working for some time and after 7-10 Days i get a new failure.

Reply
0 Kudos
nparas5
Enthusiast
Enthusiast

How are you installing VMTools Interactive or automatic? Are you suppressing the drivers like Audio, Filesharing, Sync, Network Introspection ?

Try disabling there drivers and if possible disable VSS driver as well during VMTools re-installation? The above fix which i provided earlier was given by VMware Technical Support, in your case its not so suspecting the above given drivers if not disabled .

NOTE: Network Introspection feature comes with NSX, if not using NSX's this feature disable this driver, i have seen many backup issues due to this as well, its also a VMware Finding.

Reply
0 Kudos
Supi_du
Contributor
Contributor

Hello nparas5,

i install /update Tools 10.1.10 interactive. The only disabled option is VMCI-Driver, so no NSX and File Share.

Of course I can disable VSS, but I want the Backup-Snapshot with the VMWARE VSS. And it works.... more often than not with Tools 10.1.10.

I have 2 further Server 2012R2 VM's... Both working great and do not suffer from this "Bug". Maybe its a combination of 2012R2 and Installed Exchange 2013?

But, like i wrote before, with Version 9.10.5 these snapshot Problems do not appear. ( See screenshot some Posts above)

Reply
0 Kudos
ccandia
Contributor
Contributor

This was a long time ago, has anyone found any tool versions or ways to get this issue corrected?

Reply
0 Kudos
greedj1
Contributor
Contributor

We are still seeing this on all Tools 10.2.5 and previous, it's intermittant and always fails with error 5 when debugging is enabled.

Installing tools 9.10.5 resolves the issue.

Index of /45848/tools/esx/6.0u1/windows

Reply
0 Kudos