VMware Cloud Community
sjuanlu
Contributor
Contributor

Quiescing snapshot error

Hi,

I have a problem with Data Protector and vSphere 6.0 update 1b (all security and upgrade patches are applied). When I try to do a backup for Linux vms (Debian 7 64 bits) I receive this error:

An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR.

An error occurred while saving the snapshot: msg.snapshot.error-QUIESCINGERROR.

Vms has VMware Tools upgraded and runnning. The virtual machine log (included in datastore)  vmware.log  don't show outstanding information:

2016-03-08T14:30:14.693Z| vcpu-0| I120: ToolsBackup: changing quiesce state: COMMITTED -> ERROR_WAIT

2016-03-08T14:30:21.786Z| vcpu-1| I120: HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-000001.vmdk'

2016-03-08T14:30:21.787Z| vcpu-1| I120: DDB: "longContentID" = "354a813059f4ddfaa79886a54c498d25" (was "874985b5fda1d68f5181e3296c173168")

2016-03-08T14:30:21.836Z| vcpu-1| I120: DISKLIB-CHAIN : DiskChainUpdateContentID: old=0x6c173168, new=0x4c498d25 (354a813059f4ddfaa79886a54c498d25)

2016-03-08T14:31:05.288Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox-dnd timed out.

2016-03-08T14:31:07.793Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox-dnd timed out.

2016-03-08T14:31:09.860Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox-dnd timed out.

2016-03-08T14:31:14.688Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox-dnd timed out.

2016-03-08T14:45:12.649Z| vmx| I120: Msg_Post: Warning

2016-03-08T14:45:12.649Z| vmx| I120: [msg.snapshot.quiesce.timeout] Timed out while quiescing the virtual machine.

2016-03-08T14:45:12.649Z| vmx| I120: ----------------------------------------

2016-03-08T14:45:12.653Z| vmx| I120: Vigor_MessageRevoke: message 'msg.snapshot.quiesce.timeout' (seq 10972793) is revoked

2016-03-08T14:45:12.653Z| vmx| I120: ToolsBackup: changing quiesce state: ERROR_WAIT -> DONE

2016-03-08T14:45:12.659Z| vmx| I120: SNAPSHOT: SnapshotDeleteWork '/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG.vmx' : 32

2016-03-08T14:45:12.660Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-000001-delta.vmdk" : open successful (21) size = 16863232, hd = 0. Type 8

2016-03-08T14:45:12.660Z| vmx| I120: DISKLIB-LIB_BLOCKTRACK   : Resuming change tracking.

2016-03-08T14:45:12.660Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-000001-delta.vmdk" : closed.

2016-03-08T14:45:12.661Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-flat.vmdk" : open successful (21) size = 42949672960, hd = 0. Type 3

2016-03-08T14:45:12.661Z| vmx| I120: DISKLIB-LIB_BLOCKTRACK   : Resuming change tracking.

2016-03-08T14:45:12.665Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-flat.vmdk" : closed.

2016-03-08T14:45:12.665Z| vmx| A115: ConfigDB: Setting displayName = "SYSLOG"

2016-03-08T14:45:12.730Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-000001-delta.vmdk" : open successful (1114133) size = 0, hd = 0. Type 8

2016-03-08T14:45:12.730Z| vmx| I120: DISKLIB-LIB_BLOCKTRACK   : Resuming change tracking.

2016-03-08T14:45:12.730Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/56b0a861-923cbd6c-cace-0017a477d80c/SYSLOG/SYSLOG-000001-delta.vmdk" : closed.

2016-03-08T14:45:12.730Z| vmx| A115: ConfigDB: Setting displayName = "SYSLOG"

2016-03-08T14:45:12.784Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot '_DP_VEPA_SNAP_': 32

2016-03-08T14:45:12.784Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 32 failed: Failed to quiesce the virtual machine (31).



Any idea?


Thanks in advanced.

Reply
0 Kudos
10 Replies
MauroBonder
VMware Employee
VMware Employee

Check out if this topic apply in your case

https://communities.vmware.com/thread/513185?start=0&tstart=0

*Please, don't forget the awarding points for "helpful" and/or "correct" answers. *Por favor, não esqueça de atribuir os pontos se a resposta foi útil ou resolveu o problema.* Thank you/Obrigado
Reply
0 Kudos
sjuanlu
Contributor
Contributor

That patch is the 14 May 2015 but I have the lastest version 6.0 Update 1b (6.0.0, 3380124) (Feb, 2016) with all lastest patch and upgrades applied by Update Manager. The patch

VMware ESXi 6.0, Patch Release ESXi600-201505001 (2116125)  should be already applied.


this patch is applied separately?


Thanks.

Reply
0 Kudos
UmeshAhuja
Commander
Commander

Hi,

Check it out if you have vSphere 6.0 U1a in your environment.

Backup Issues

  • Snapshot quiescing fails on Linux VMs
    When you perform a quiesced snapshot of a Linux virtual machine, the VM might fail after the snapshot operation. The following error messages are logged in the vmware.log file:

    <YYYY-MM-DD>T<TIME>Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'smvi_UUID': 0
    <YYYY-MM-DD>T<TIME>Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).
    <YYYY-MM-DD>T<TIME>Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
    <YYYY-MM-DD>T<TIME>Z| vmx| I120: Vix: [18631 guestCommands.c:1926]: Error VIX_E_TOOLS_NOT_RUNNING in MAutomationTranslateGuestRpcError(): VMware Tools are not running in the guest

    This issue is resolved in this release ( VMware ESXi 6.0 Update 1a ).

Ref :- VMware ESXi 6.0 Update 1a Release Notes

Thanks n Regards
Umesh Ahuja

If your query resolved then please consider awarding points by correct or helpful marking.
Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

ESXi patches are always cumulative. If you're on 6.0U1b, then you have all the fixes from U1a and previous patches included. There is no need/no way to apply patches separately (forcing it would actually result in a complete system downgrade).

Can you create a snapshot manually with and without quiescing, or do you only get that error when you backup application tries to snapshot the VM?

Do you see anything in the VM OS logs, especially messages/dmesg/selinux?

Do you see anything in the vmkernel.log on the host?

Have you re-run the vmware-config-tools.pl  script?

Can you check the Tools version with: # vmware-toolbox-cmd -v


-- http://alpacapowered.wordpress.com
Reply
0 Kudos
sjuanlu
Contributor
Contributor

Thanks for the replies.

Can you create a snapshot manually with and without quiescing, or do you only get that error when you backup application tries to snapshot the VM?

- Yes, manually with and without quiescing. No problem. The problem appear whith HP Data Protector software backup.

Do you see anything in the VM OS logs, especially messages/dmesg/selinux?

- No, I have not seen anything related to quiescing.

Do you see anything in the vmkernel.log on the host?

- Only what it is in the first post.

Have you re-run the vmware-config-tools.pl  script?

- Yes. I have reinstalled the tools twice. The tools are the last version.

The error only occurs in debian 7 64 bits.

It's rare...

Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

- Yes, manually with and without quiescing. No problem. The problem appear whith HP Data Protector software backup.

If manual quiescing works fine, I can only assume the issue lies with the VDDK your backup software ships with. VMware routinely updates the VDDK to fix bugs, but it usually takes a lot of time before backup software vendors incorporate the newer versions. To make sure check if your backup software is up to date and supports 6.0 U1.

Do you see anything in the vmkernel.log on the host?

- Only what it is in the first post.

Your first post only contains info of the VM's own vmware.log file. I am referring to the host's vmkernel log file you should find at /var/log/vmkernel.log.

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
sjuanlu
Contributor
Contributor

In this post and other, always said that it is support for Vsphere 6.0 ( vSphere 5.5, 5.1, 5.0, 4.1) without updates or pacha:

Backup Interoperability Matrix - vBrain.info

In vmkernel.log only this:

2016-03-15T19:52:29.504Z cpu3:4215657)VSCSI: 2590: handle 8727(vscsi0:0):Reset request on FSS handle 384096822 (0 outstanding commands) from (vmm0:STA500)

2016-03-15T19:52:29.504Z cpu3:4215657)VSCSI: 2590: handle 8728(vscsi0:1):Reset request on FSS handle 102423099 (0 outstanding commands) from (vmm0:STA500)

2016-03-15T19:52:29.505Z cpu25:33115)VSCSI: 2868: handle 8727(vscsi0:0):Reset [Retries: 0/0] from (vmm0:STA500)

2016-03-15T19:52:29.505Z cpu25:33115)VSCSI: 2661: handle 8727(vscsi0:0):Completing reset (0 outstanding commands)

2016-03-15T19:52:29.505Z cpu25:33115)VSCSI: 2868: handle 8728(vscsi0:1):Reset [Retries: 0/0] from (vmm0:STA500)

2016-03-15T19:52:29.506Z cpu25:33115)VSCSI: 2661: handle 8728(vscsi0:1):Completing reset (0 outstanding commands)

2016-03-15T19:52:30.599Z cpu6:4215664)VSCSI: 6726: handle 8727(vscsi0:0):Destroying Device for world 4215657 (pendCom 0)

2016-03-15T19:52:30.600Z cpu6:4215664)VSCSI: 6726: handle 8728(vscsi0:1):Destroying Device for world 4215657 (pendCom 0)

2016-03-15T19:52:30.612Z cpu6:4215664)WARNING: CBT: 2056: Unsupported ioctl 44

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 3996: handle 8735(vscsi0:0):Using sync mode due to sparse disks

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 4038: handle 8735(vscsi0:0):Creating Virtual Device for world 4215657 (FSS handle 384096822) numBlocks=83886080 (bs=512)

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 273: handle 8735(vscsi0:0):Input values: res=0 limit=-2 bw=-1 Shares=1000

2016-03-15T19:52:30.612Z cpu6:4215664)WARNING: CBT: 2056: Unsupported ioctl 44

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 3996: handle 8736(vscsi0:1):Using sync mode due to sparse disks

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 4038: handle 8736(vscsi0:1):Creating Virtual Device for world 4215657 (FSS handle 102423099) numBlocks=104857600 (bs=512)

2016-03-15T19:52:30.612Z cpu6:4215664)VSCSI: 273: handle 8736(vscsi0:1):Input values: res=0 limit=-2 bw=-1 Shares=1000

2016-03-15T19:52:38.135Z cpu6:4215659)VSCSI: 2590: handle 8735(vscsi0:0):Reset request on FSS handle 384096822 (0 outstanding commands) from (vmm0:STA500)

2016-03-15T19:52:38.135Z cpu6:4215659)VSCSI: 2590: handle 8736(vscsi0:1):Reset request on FSS handle 102423099 (0 outstanding commands) from (vmm0:STA500)

2016-03-15T19:52:38.135Z cpu25:33115)VSCSI: 2868: handle 8735(vscsi0:0):Reset [Retries: 0/0] from (vmm0:STA500)

2016-03-15T19:52:38.135Z cpu25:33115)VSCSI: 2661: handle 8735(vscsi0:0):Completing reset (0 outstanding commands)

2016-03-15T19:52:38.135Z cpu25:33115)VSCSI: 2868: handle 8736(vscsi0:1):Reset [Retries: 0/0] from (vmm0:STA500)

2016-03-15T19:52:38.136Z cpu25:33115)VSCSI: 2661: handle 8736(vscsi0:1):Completing reset (0 outstanding commands)

Thanks.

Reply
0 Kudos
sguk
Contributor
Contributor

Are you using a Linux VM with Centos 6?

If so, quiescing does not work, it crashes the VM. This is due to a bug in the Centos 6 kernel so VMWare Tools quiescing will not work.

Centos 7 works fine.

Reply
0 Kudos
GoingAlone
Enthusiast
Enthusiast

I had this issue on a vm, the disks were set to independent mode in the configuration for the vm. Removing that fixed it for us.

Reply
0 Kudos
shane1973
Enthusiast
Enthusiast

Have you ruled out the performance of your storage?  What else is happening during backup?  We've had a few instances of this error type.  Ironically in one case it completely went away when we removed a host from the 2 host cluster. VMware has no explanation why only having 1 host completely gets rid of this error/problem.

Reply
0 Kudos