VMware Cloud Community
SebastianGrugel
Hot Shot
Hot Shot

Virtual machine disks consolidation failed / The virtual machine has exceeded the maximum downtime of 12 seconds for disk consolidation

For Community information or maybe somebody will advice something interesting.

My Environment:

ESXi 5.5 u2

vCenter 5.5 u2

only 1 VSAN datastore in cluster

2 cluster in vCenter

Hi.

We have VM which from some time cant be consolidated. What we know about alerts and this VM.

Screenshot_2a.png

Screenshot_1.png

Screenshot_2.png

Screenshot_5.png

Screenshot_3.png

Screenshot_6.png

Screenshot_7.png

Screenshot_8a.png

Screenshot_8.png

Screenshot_9.png

Screenshot_10.png

In logs on this VM (vmware.log) we found:

2016-05-04T20:03:57.420Z| vcpu-0| I120: DISKLIB-LIB  : Free disk space is less than imprecise space neeeded for combine (0x1fbd3800 < 0xd3714800, in sectors). Getting pre

cise space needed for combine...

2016-05-04T20:04:10.494Z| vcpu-0| I120: SnapshotVMXConsolidateHelperProgress: Stunned for 13 secs (max = 12 secs). Aborting consolidate.

2016-05-04T20:04:10.494Z| vcpu-0| I120: DISKLIB-LIB :DiskLibSpaceNeededForCombineInt: Cancelling space needed for combine calculation

2016-05-04T20:04:10.495Z| vcpu-0| I120: DISKLIB-LIB  : DiskLib_SpaceNeededForCombine: failed to get space for combine operation: Operation was canceled (33).

2016-05-04T20:04:10.495Z| vcpu-0| I120: DISKLIB-LIB  : Combine: Failed to get (precise) space requirements.

2016-05-04T20:04:10.495Z| vcpu-0| I120: DISKLIB-LIB  : Failed to combine : Operation was canceled (33).

2016-05-04T20:04:10.495Z| vcpu-0| I120: SNAPSHOT: SnapshotCombineDisks: Failed to combine: Operation was canceled (33).

After fast troubleshooting with Vmware engineer we receive summary:

"I tracked down that KB I mentioned, and it's somewhat related but it also seems to be related to a newer VSAN-specific bug.

I mentioned this calculation that must be done in certain cases.

When a consolidation is called, the host does a basic check - is the snapshot file larger than the amount of free space on the datastore. If there is definitely enough free space, itt proceeds with the consolidation.

However, if the snapshot file is larger than the amount of free space then we need to do a more detailed calculation to determine if we will be able to consolidate without filling up the datastore.

We allow 12 seconds to perform this calculation - if it takes any longer, then we abort the consolidation.

This is exactly what we see in your logs.

As we saw, there is more than enough free space on the VSAN datastore

Screenshot_9.png


so we should not need to be doing this precise calculation. I've found a bug in VSAN where there is a problem with the calculation, meaning that consolidation may fail for VMs with large snapshot files:

https://kb.vmware.com/kb/2118663

We don't see the exact symptoms here - your consolidation is not failing with 'insufficient space', it is failing with 'operation was cancelled'. But it seems to be related to the same root cause.

The fix for the problem went into 5.5U3b so obviously that is not an option right now.

As far as I know, the workaround mentioned in the KB may also not be an option as I don't think you would have a suitable datastore to send the clone to.

However, there is another workaround which has helped some other customers:

- Shut down the virtual machine

- Right-click the virtual machine and click Edit Settings.

- Click the Options tab.

- Under Advanced, click General.

- Click Configuration Parameters and add snapshot.maxConsolidateTime = 120

  This should allow more time for the calculation to complete and may allow consolidation to proceed."


After this investigation i additional KB which can confirm this way of working:



VMware KB:    Snapshot consolidation in VMware ESXi 5.5.x and ESXi 6.0.x  fails with the error: maxi...

If somebody was some similiar experience ? and can advice something?

We have only VSAN datastore then we cant migrate this VM to some other local datastore.

If i will have some updates i will put information on post.

Greetings

Sebastian    

vExpert VSAN/NSX/CLOUD | VCAP5-DCA | VCP6-DCV/CMA/NV ==> akademiadatacenter.pl
5 Replies
SebastianGrugel
Hot Shot
Hot Shot

Today we shutdown VM and add to VM advanced parameter:

snapshot.maxConsolidateTime = 120

After turn ON VM again and start consolidation completed with succesfull after 2 minutes.

vExpert VSAN/NSX/CLOUD | VCAP5-DCA | VCP6-DCV/CMA/NV ==> akademiadatacenter.pl
Prakas
Enthusiast
Enthusiast

I faced this error. I used to consolidate the snapshot by powering off the VM - it will complete real quick.

Reply
0 Kudos
martinmurugan
Contributor
Contributor

Hi All,

Method 1: Need to shutdown the VM

  • Shut down the virtual machine. 
  • Right-click the virtual machine and click Edit settings
  • Click the Options tab. 
  • Under Advanced, right-click General, click Configuration Parameters and then click Add Row
  • In the left pane, add this parameter:
    snapshot.asyncConsolidate.forceSync
  • In the right pane, add this value:
    TRUE
  • Click OK to save the changes.  
  • Power on the virtual machine.

Method 2: No downtime

To set the parameter snapshot.asyncConsolidate.forceSync to TRUE without shutting down the virtual machine, run this PowerCLI command:

get-vm virtual_machine_name | New-AdvancedSetting -Name snapshot.asyncConsolidate.forceSync -Value TRUE -Confirm:$False

Thanks in Advance

PaulB2
Enthusiast
Enthusiast

Method 2 worked for me, thanks!

Reply
0 Kudos
TEST1981
Contributor
Contributor

method 2 worked,but need disk move between datastorage to loss locked file.then ok.

Reply
0 Kudos