Solved: An error occurred while taking a snapshot: Failed ...

renwong · ‎02-20-2014

Hi

I'm using vCenter 5.5 with WEB vSphere GUI to take a snapshot of a RHEL 5.4 32 based VM on a ESXi 5.5. When I choose the "Quiesce the guest file system" instead of the "snapshot the virtual machine memory", the snapshot creation failed at 90% with "An error occurred while taking a snapshot: Failed to quiesce the virtual machine" .

I have no problem using the "snapshot the virtual machine memory" option to create the snapshot.

Your help is appreciated

Ren Wong

renwong · ‎02-28-2014

It is a RHEL 5.4 kernel bug. Thanks everyone who helped.

https://bugzilla.redhat.com/show_bug.cgi?id=539328

View solution in original post

renwong · ‎02-20-2014

As it turns out that we have two root partitions on the VM. If the VM is up and running from the first root partition (file system label /) there is no issue taking the snapshot but if the VM is up on the second partition (/2), the snapshot fails and causes the VM to panic.

Re-installing and re-configuring the VMware tool on the second partition does not make any difference. After the snapshot approaching 95%, the VM panic.

Shrikant_Gavhan · ‎02-23-2014

Hi Renwong,

Please check virtual machine usage , if it is generating high IO inside the OS

Please check the below link if it helps you,

VMware KB: Creating a quiesced snapshot in Windows 2008 and Windows 2008 R2 fails with the error: Sn...

Just for the information, RHEL / any of the linux or Unix VM server does not support VMware tools quiescing.

Unix / Linux file system does not quiesce the machine in order to take the application consistent backup, it is only crash consistent copy of the OS.

Thanks and Regards,

Shrikant Gavhane.

Thanks and Regards, Shrikant Gavhane

ManjunathV · ‎02-23-2014

I'm not sure how you took snapshot for the first machine which VM installed on first partition, since Linux OS doesn't support for Quiesce, since which require VSS to quiesce, so where ever we have VSS components, in those machines only we can take snapshot with Quiesce.

renwong · ‎02-24-2014

Hi ManjunathV,

Let me provide more information. I was taken a VM backup using vSpehere WEB client from a RHEL 5.4 based VM. The backup failed due to the Quiesce error. That is why I try manual snapshot using the "Quiesce the guest file system" instead of the "snapshot the virtual machine memory".

So the question now is why the EBR plug-in quiesce the RHEL VM if it is not supported ?

The KB article provided by Krishant did not help. I cannot find the snapshot.redoNotWithParent parameter.

Thanks

Ren

renwong · ‎02-24-2014

Here is the hostd.log of the ESXi:

2014-02-24T18:41:20.030Z [3D8C2B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:41:20.030Z [3D8C2B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details

2014-02-24T18:41:22.812Z [3D881B70 verbose 'SoapAdapter'] Responded to service state request

2014-02-24T18:41:24.615Z [3D881B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root

2014-02-24T18:41:25.710Z [3C240B70 verbose 'Hostsvc.DvsManager'] PersistAllDvsInfo called

2014-02-24T18:41:25.939Z [3D881B70 info 'Hostsvc' opID=hostd-9f65] VsanSystemVmkProvider : GetRuntimeInfo: Start

2014-02-24T18:41:25.939Z [3D881B70 info 'Hostsvc' opID=hostd-9f65] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

--> dynamicType = <unset>,

--> accessGenNo = 0,

--> }

2014-02-24T18:41:31.114Z [3BEC2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Updating current heartbeatStatus: yellow

2014-02-24T18:41:32.031Z [3D8C2B70 verbose 'SoapAdapter'] Responded to service state request

2014-02-24T18:41:35.396Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2014-02-24T18:41:35.404Z [3D8C2B70 verbose 'Locale' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2014-02-24T18:41:35.423Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2014-02-24T18:41:35.425Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2014-02-24T18:41:35.437Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2014-02-24T18:41:35.437Z [FFDE4B70 verbose 'Vimsvc.ha-license-manager' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:41:35.452Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6836-77a33b2-eb user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

2014-02-24T18:41:40.029Z [3D8C2B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:41:40.030Z [3D8C2B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details

2014-02-24T18:41:45.422Z [3D881B70 verbose 'Default' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2014-02-24T18:41:45.430Z [3D881B70 verbose 'Locale' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2014-02-24T18:41:45.452Z [3D881B70 verbose 'Default' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2014-02-24T18:41:45.454Z [3C240B70 verbose 'Default' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2014-02-24T18:41:45.466Z [FFDE4B70 verbose 'Default' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2014-02-24T18:41:45.466Z [3D881B70 verbose 'Vimsvc.ha-license-manager' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:41:45.481Z [3D881B70 verbose 'Default' opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

2014-02-24T18:41:51.393Z [3C240B70 verbose 'Default' opID=1381cd91-97 user=vpxuser] AdapterServer: target='vim.VirtualMachine:8', method='createSnapshot'

2014-02-24T18:41:51.394Z [3C240B70 info 'Vimsvc.TaskManager' opID=1381cd91-97 user=vpxuser] Task Created : haTask-8-vim.VirtualMachine.createSnapshot-228252373

2014-02-24T18:41:51.394Z [3D8C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx' opID=1381cd91-97 user=vpxuser] Create Snapshot: Avamar-1393267311a6d576b91e72d43fdb15e1a7fc3dea10e9a47796, memory=false, quiescent=true state=4

2014-02-24T18:41:51.394Z [3D8C2B70 info 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx' opID=1381cd91-97 user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_CREATE_SNAPSHOT)

2014-02-24T18:41:52.641Z [3BEC2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Handling message _vmx3: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

-->

2014-02-24T18:41:52.641Z [3BEC2B70 info 'Vimsvc.ha-eventmgr'] Event 243 : Message on slu003 on esxi14. in ha-datacenter: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

-->

2014-02-24T18:41:52.642Z [3BEC2B70 info 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Answered question _vmx3

2014-02-24T18:41:52.813Z [3D8C2B70 verbose 'SoapAdapter'] Responded to service state request

2014-02-24T18:42:00.029Z [3D581B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:42:00.029Z [3D581B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details

2014-02-24T18:42:01.116Z [3D8C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Updating current heartbeatStatus: green

2014-02-24T18:42:02.033Z [3D581B70 verbose 'SoapAdapter'] Responded to service state request

2014-02-24T18:42:03.555Z [FFD815B0 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Version status of tools changed to: 3

2014-02-24T18:42:05.401Z [3D581B70 verbose 'Default' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2014-02-24T18:42:05.409Z [3D581B70 verbose 'Locale' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2014-02-24T18:42:05.428Z [3D581B70 verbose 'Default' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2014-02-24T18:42:05.430Z [3D581B70 verbose 'Default' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2014-02-24T18:42:05.441Z [3D581B70 verbose 'Default' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2014-02-24T18:42:05.441Z [3D581B70 verbose 'Vimsvc.ha-license-manager' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:05.456Z [3D581B70 verbose 'Default' opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

2014-02-24T18:42:08.332Z [3C240B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 30597MHz/60241MB to 30597MHz/60239MB

2014-02-24T18:42:08.335Z [3D581B70 verbose 'Default' opID=SWI-dda157cf user=vpxuser] AdapterServer: target='vim.ResourcePool:ha-root-pool', method='GetConfig'

2014-02-24T18:42:08.336Z [3BEC2B70 verbose 'Default' opID=SWI-dda157cf user=vpxuser] AdapterServer: target='vim.ResourcePool:ha-root-pool', method='GetName'

2014-02-24T18:42:15.402Z [3C240B70 verbose 'Default' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2014-02-24T18:42:15.410Z [3C240B70 verbose 'Locale' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2014-02-24T18:42:15.429Z [3C240B70 verbose 'Default' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2014-02-24T18:42:15.430Z [3C240B70 verbose 'Default' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2014-02-24T18:42:15.442Z [3C240B70 verbose 'Default' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2014-02-24T18:42:15.442Z [3C240B70 verbose 'Vimsvc.ha-license-manager' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:15.457Z [3C240B70 verbose 'Default' opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

2014-02-24T18:42:15.711Z [3C240B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Version status of tools changed to: 3

2014-02-24T18:42:18.569Z [3C5D2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Create Snapshot translated error to vim.fault.GenericVmConfigFault

2014-02-24T18:42:18.569Z [3C5D2B70 info 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Create Snapshot failed: vim.fault.GenericVmConfigFault

2014-02-24T18:42:18.569Z [3C5D2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Create Snapshot message: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

--> An error occurred while taking a snapshot: Failed to quiesce the virtual machine.

-->

2014-02-24T18:42:18.569Z [3C240B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] guest operations are not ready

2014-02-24T18:42:18.569Z [3C240B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Version status of tools changed to: 3

2014-02-24T18:42:18.571Z [FFD815B0 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Tools are not auto-upgrade capable

2014-02-24T18:42:18.571Z [FFD815B0 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Skipped tools ManifestInfo update.

2014-02-24T18:42:18.574Z [3D8C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Time to gather Snapshot information ( read from disk, build tree): 2 msecs. needConsolidate is false.

2014-02-24T18:42:18.597Z [3D8C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Time to gather config: 22 (msecs)

2014-02-24T18:42:18.597Z [3D581B70 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 8

2014-02-24T18:42:18.599Z [3D8C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Time to gather snapshot file layout: 0 (msecs)

2014-02-24T18:42:18.617Z [3D8C2B70 warning 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] Failed operation

2014-02-24T18:42:18.617Z [3D8C2B70 info 'Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx'] State Transition (VM_STATE_CREATE_SNAPSHOT -> VM_STATE_ON)

2014-02-24T18:42:18.618Z [3D881B70 verbose 'Hostsvc'] Received state change for VM '8'

2014-02-24T18:42:18.618Z [3D881B70 info 'Guestsvc.GuestFileTransferImpl'] Entered VmPowerStateListener

2014-02-24T18:42:18.618Z [3D8C2B70 info 'Vimsvc.TaskManager'] Task Completed : haTask-8-vim.VirtualMachine.createSnapshot-228252373 Status error

2014-02-24T18:42:18.618Z [3D881B70 info 'Guestsvc.GuestFileTransferImpl'] VmPowerStateListener succeeded

2014-02-24T18:42:18.618Z [3D581B70 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 8 not configured for replication

2014-02-24T18:42:18.618Z [3D881B70 info 'Hbrsvc'] Replicator: powerstate change VM: 8 Old: 1 New: 1

2014-02-24T18:42:18.619Z [3D881B70 verbose 'Hbrsvc'] Replicator: Remove group no matching entry for VM (id=8)

2014-02-24T18:42:20.029Z [3C240B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:42:20.030Z [3C240B70 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details

2014-02-24T18:42:22.815Z [3BEC2B70 verbose 'SoapAdapter'] Responded to service state request

2014-02-24T18:42:25.405Z [3C240B70 verbose 'Default' opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2014-02-24T18:42:25.413Z [3C240B70 verbose 'Locale' opID=HB-host-516@6847-48fcca09-b user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2014-02-24T18:42:25.433Z [3C240B70 verbose 'Default' opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2014-02-24T18:42:25.434Z [3C240B70 verbose 'Default' opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2014-02-24T18:42:25.446Z [3C240B70 verbose 'Default' opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2014-02-24T18:42:25.446Z [3C240B70 verbose 'Vimsvc.ha-license-manager' opID=HB-host-516@6847-48fcca09-b user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:25.461Z [3C240B70 verbose 'Default' opID=HB-host-516@6847-48fcca09-b user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

renwong · ‎02-24-2014

More tests reveal that if our application is shutdown, there is no problem taking a quiesce snapshot.

renwong · ‎02-28-2014

It is a RHEL 5.4 kernel bug. Thanks everyone who helped.

https://bugzilla.redhat.com/show_bug.cgi?id=539328