VMware Cloud Community
Shoomy
Contributor
Contributor

Unable to take quiesced snapshot

After a reboot of a server, it can happen that it is not possible to take a quiesced snapshot.It happened after an update to 6U1 from 6.0. Before the update there were no problems.

We did the following to troubleshoot the problem.

* Reinstall VMWare Tools

* Check VSS Providers for errors (no errors)

What we found out was that it can happen after a reboot of the server and only if the service "VMware Snapshot Provider" is running (service is set to manual). To correct it we need to stop "VMware Snapshot Provider" and "VMware Tools". If all is stable, ie no running services, which should not run, like "Volume Shadow Copy" we start the service "VMware Tools". After this we can take a quiesced snapshot again.

While this ain't a huge problem, it is annoying at best. Any one have a solution or have the same problem.

32 Replies
DavoudTeimouri
Virtuoso
Virtuoso

Hi,

IF your VM's OS is Windows 2008 R2, please read the below KB and do the mentioned instruction:

VMware KB: Cannot take a quiesced snapshot of Windows 2008 R2 virtual machine

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
Reply
0 Kudos
Shoomy
Contributor
Contributor

Thanks for your reply and I have read this article several times when in search for a solution.

Unfortunately this is not the solution as our backup relies on application awareness (SQL, Exchange etc.). Disabling this feature will have impact on our backup process.

All our servers (w2k8 and w2k12) have the same problem. After a reboot, the service "vmware service provider" keeps running and wont stop (not always but it can happen), which blocks quiesced snapshot.

Odin_13
Contributor
Contributor

Hi Shoomy,

I can confirm your observations.

We also experience the issue with some of our Windows 2012 servers and stopping the 'VMware Snapshot Provider' service resolves the issue.

After stopping the service it is possible again to create a quiesced snapshot of the server.

Thanks!

Reply
0 Kudos
hussainbte
Expert
Expert

what is the startup type for the service ? Manual or Automatic?

I believe it should be manual.

If you found my answers useful please consider marking them as Correct OR Helpful Regards, Hussain https://virtualcubes.wordpress.com/
Reply
0 Kudos
Odin_13
Contributor
Contributor

Startup for the 'VMware Snapshot Provider' service is indeed set to manual.

Strange thing is that the issue is completely random and so far only a couple of servers (all Win2K12) are affected.

Reply
0 Kudos
Shoomy
Contributor
Contributor

What happens is that after a reboot VMware tools service starts and triggers "VMware Snapshot Provider" and "Volume Shadow Copy" after a while, few minutes at the most, both services should stop running. Both service are on manual as VMware best practice. Exactly the same behaviour as when taking an quiesced snapshot.

In some cases "VMware snapshot provider" just does not stop and this only happens after a reboot.

Thanks Odin for confirming. May I ask you which version of VMware you are running. We are on 6.0U1 (10240) and yes it is completely random. I have it on w2k8 and w2k12, not only on w2k12.

Reply
0 Kudos
Odin_13
Contributor
Contributor

Hi Shoomy,

So far i have only encountered the issue with a couple of Win2K12 servers.

Still trying to figure out why this occurs...

Reinstall of the VMware Tools does not solve the issue as you already stated in an earlier post.

Build  : Releasebuild-3380124

Patch  : 26

Update  : 1

Version : 6.0.0

VMware Tools: 10240

Reply
0 Kudos
a11112f
Contributor
Contributor

Hey i'm having the same problem with Server 2012 R2 backups (Veeam 9 Update 1, using VeeamZIP full method). I think the problem started around ESXi 6 Update 1 or Update 1b, I'm running ESXi 6 Update 2 now.

I have reinstalled VMWare tools (with setup /C), and upgraded to ESXI 6u2 but it didn't help.

I get this in the vSphere client:

An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR.

An error occurred while saving the snapshot: msg.snapshot.error-QUIESCINGERROR.

or

A timeout occurred while communicating with VMware Tools in the virtual machine.

The only error I get on the Windows guest is:

Event ID 7011 Service control manager

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VMTools service.

No VSS or any other errors.

As of now the only fix I have is to stop the VMWare Snapshot Provider service before I run a backup.  This seems to work 100% for me.

The VMWare Snapshot Provider seems to starts and stops with the VMware tools service, it is set to manual (default setting) start but never seems to stop on it's own.

Reply
0 Kudos
bwernli
Contributor
Contributor

@a11112f

Same Problem here with ESXi 5.5 U3b!! We have >1000 VM's and a day certainly 2-3 VM's with this problem :smileyconfused:

Reply
0 Kudos
a11112f
Contributor
Contributor

I run a Task Scheduled Powershell script targeted at Veeam for my backups so I was able to add some lines to stop the VMware Snapshot Provider service when the backup starts.  For the first time my backups all completed.

$ServiceStat = Get-Service -Name vmvss -ComputerName $VMName

if ($ServiceStat.Status -eq "Running"){

stop-service -inputobject $(get-service -ComputerName $VMName -Name vmvss)

}

I was also thinking of using Group Policy with a WMI filter targeted at Server 2012 R2 installs to stop the services, but I have the feeling that having the service stop during a backup would be...bad.  Once that service is stopped though, quiesced snapshots work.

I attached the full Veeam script if anyone wants it, I didn't write the original but I have made a few of my own changes.  There is more info in the script comments as to the author.

HendersonD
Hot Shot
Hot Shot

I am seeing the exact same thing under Win 2008 R2 and Win 2012 R2. We are running ESXi 6.0 U2. The VMWare Snapshot provider service is set to manual but upon reboot of the VM the service starts and does not stop. At backup time a quiesced snapshot cannot be taken. If I go in and stop this service, then a snapshot can be taken. We are using Commvault for backup but will be jumping to Veeam in the near future so I will test the script

Reply
0 Kudos
tadsmith
Contributor
Contributor

We are seeing the same thing on all version of the 10.x tools, even the 10.0.8 version. They say this will be resolved in 10.0.9 released in June, but I am pushing for a faster resolution. I'm also trying to get them to give me some details on what exact tools / OS versions are affected and what is causing the issue.

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

I have 41 Windows VMs running under ESXI 6 Update 2

Yesterday I applied 2 Windows updates to all of them, they all had to restart

Last night my backup ran and it shows this morning that for 13 of the VMs a quiesced snapshot could not be taken

I restarted all 13 VMs this morning

I logged into each one and found the VMware Snapshot Provider service still running on 6 of them even though it is set for manual. I stopped this service on all 6

My backup should run fine tonight

The big fix for this cannot come soon enough

Reply
0 Kudos
Vitaly91
VMware Employee
VMware Employee

Do you by chance run VMTurbo?

Reply
0 Kudos
tomsie1000
Contributor
Contributor

Having the same issue here with 5.5 and 6U2 - we have several environments that share a base template, and since the tools have moved to 10.0.x we've been having this issue across all of them. We also use NetBackup which uses the snapshot functionality, so this meant no backups for us too.

I've found two ways around it:

1. Remove the shadow copy (VSS driver) from the VMware Tools. Bare in mind that you are removing the quiescing abilities so choose this option carefully where it works in tandem with the application and how it uses the filesystem.

2. Restart the services just after startup. I've implemented a script via gpo to roll to any affected machines - not ideal but this way it'll be easy to pull it back out later if/when the bug is fixed.

Here's the code if it helps anyone out:

sleep 60 #give the system time to start the services - may need tweaking for slower systems

if ((Get-Service -Name vmvss).Status -eq 'Running'){stop-service -inputobject $(get-service -Name vmvss)} #stop the VMware Snapshot Provider service

if ((Get-Service -Name VMTools).Status -eq 'Running'){stop-service -inputobject $(get-service -Name VMTools)} #stop the VMware Tools service

Sleep 10 #wait for the services to stop - may need tweaking but this works for us - i was too lazy to write a check loop Smiley Happy

Start-Service -InputObject (Get-Service -name VMTools) #start VMware Tools service

Reply
0 Kudos
Bleeder
Hot Shot
Hot Shot

The 10.0.9 tools were released last week and the notes say this issue should be fixed.

VMware Tools 10.0.9 Release Notes

Reply
0 Kudos
diegoazevedo
Contributor
Contributor

Can anyone confirm whether this is fixed with VMware Tools 10.0.9?

Reply
0 Kudos
Agevis
Contributor
Contributor

We have the same problem here trying to take backup using Acronis.

Several 2012 R2 server with no problem except for just one SQL Server.

The problem started suddenly maybe after a windows update (not sure) but there are other SQL Server which get updated every week.

Disabling vmware snapshot provider service works for us but don't know if there are any drawbacks.

Updated to tools v10.0.9 build 3917699 did NOT resolve the issue.

Tried also to update ESX host to last patch 6.0.0 4192238 and did NOT resolve the issue.

Problem arise only when take snapshot quiesce=yes memory=no with vmware snapshot provider service not set to disabled.

Reply
0 Kudos
Supi_du
Contributor
Contributor

I can confirm this for latest ESXi 5.5 4345813 and Server 2003 32bit vm. When i take Snapshot, i get : msg.snapshot.error-QUIESCINGERROR.

But is worked before for some time. And also 2 other 2003 VM's dont show this Problem?! Downgrade to Tools 9.4.15 helped.

Reply
0 Kudos