VMware Cloud Community
racom
Enthusiast
Enthusiast

"Snapshot hierarchy is too deep" after vSphere Replication crashed

We use remote NFS server as target datastore for standalone vSphere Replication. It crashed a week ago. It looked like replication was reestablished well after restoring. But I've found two VM's in Virtual machine Consolidation Needed status. After stopping vSphere Replication and vSphere Data Protection I was able to consolidated one of them but the other one failed with "Snapshot hierarchy is too deep" error message. I've checked datastore and I can see 256 *-delta.vmdk and *-ctk.vmdk files. I've found some howto here http://networkadminkb.com/KB/a331/how-to-fix-snapshot-hierarchy-is-too-deep.aspx. But the affected VM is Active Directory PDC W2012 so I can't power it off now and I'm afraid to delete snapshost of running VM. Do you have any experience, please? Or is some other way how to consolidate snapshots of running VM?

Reply
0 Kudos
15 Replies
admin
Immortal
Immortal

Hi,

This problem is a tricky one to handle since VMware supports only 32 level of snapshots. but we have our options,

this is what I do when the snapshots is out of control

Option 1 :

Step 1 : Take an SSH to any of the Host with access to the data store containing the snapshots and cd to the location.

Step 2 : Run vmkfstools -qv3 <last snapshot >

eg : vmkfstools -qv3 test-00256.vmdk --- This will let us know if the snapshot chain is complete.

Then commit the disks using vmksfstools -i last < snapshot.vmdk >  VMname. vmdk name

eg :  vmkfstools -i test-00256.vmdk VMName.vmdk

The below Kb should be helpful :

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100454...

Note : If have to repeat the same for all the hard disk. ie vmdks.and requires downtime.

Option 2 :

use the VMware converter and do V2V of the machine on the fly - No downtime required.

I would prefer to use the second option.

Download link for converter :

https://my.vmware.com/web/vmware/info/slug/infrastructure_operations_management/vmware_vcenter_conve...

Links of the KB article’s below which would help you to increase the performance of the converter and speed up the process.

http://kb.vmware.com/kb/1004588

http://kb.vmware.com/kb/2020517

Hope this was helpful.

Thanks,

Avinash

continuum
Immortal
Immortal

You will need a downtime some time when the vmsd is invalid.
To keep the downtime as short as possible you can use vmkfstools -i to consolidate all but the very last snapshot.
Then you still need about 3 minutes of downtime to prepare the final step that gets rid of the last snapshot.
Merging the last snapshot into the new basedisk that you created with vmkfstools -i then again can be done while the VM is running.

I can give you detailed instructions when you post details (file-listing, vmware.logs ...)


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

racom
Enthusiast
Enthusiast

Many thanks. I'll convert VM for sure first and try commit the disks then. The snapshot chain looks to be complete.

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Many thanks. Archive with vmware.log, file-listing and vmkfstools -qv3 output is here http://uloz.to/x3EG8SeW/vm-logs-tgz

Reply
0 Kudos
racom
Enthusiast
Enthusiast

"Cannot convert a powered-on machine" message. I must omit this step Smiley Sad

Reply
0 Kudos
iangelov
VMware Employee
VMware Employee

Actually convert "Powered on machine" can be used in this case.

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Thanks. But I have no idea how to proceed. The "Next" button is not avalaible.

VM_Convertor.png

Reply
0 Kudos
iangelov
VMware Employee
VMware Employee

I see, back to the first wizard page and choose "Powered on machine" source.

In this case you need admin credentials for the machine, also result will not have any snapshots (they are invisible for the guest os).

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Thanks for your tip. But I'm not able to contact the specified VM. It's probably out of scope this discussion but I can ping this VM, I can log to it via remote desktop but using the same credentials Converter can't contact it from the same PC :smileyconfused:

Reply
0 Kudos
racom
Enthusiast
Enthusiast

It's probably because the host this VM running on is in disconnected state. Can't reconnect host to vCSA: Host name does not match the subject name(s) in certificate. I've put support request but it's not resolved yet.

Reply
0 Kudos
iangelov
VMware Employee
VMware Employee

It depends on network/firewall/etc. settings.

Via remote desktop you can install converter and choose "This local machine" radio button.

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Thanks again. It works. I can't see the most simple solution sometimes Smiley Sad

I converted VM for backup purposes and now I try to consolidate snapshots. There are two disks in different storages with the same amount of snapshots.

Reply
0 Kudos
continuum
Immortal
Immortal

the download link does not work - please simply attach the files here


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Thanks, logs attached. I used simple editor before and I couldn't find Attach Files icon I'm sorry.

Some more info:

There are two disks in different storages with the same amount of snapshots in this VM. I've did for both of them yet:

1. cd <disk directory>

2. mkdir ../restore

3. vmkfstools -i <disk name>-000255.vmdk ../restore/<dik name>.vmdk

I can't copy the last snapshots to restore directory now 'cause <disk name>-000256-delta.vmdk files are busy. Shall I turn off VM first, copy files, turn on VM and proceed with consolidation then? But what about new changes? Or is better to have VM turned off for all next steps? Anyway I must wait for maintanace hours before turn this VM off.

I'm not sure with "9. Remove the existing snapshot vmdk ..." in VMware KB: Committing snapshots in vSphere when more than 32 levels of snapshots are present fails w... Shall I do "rm <disk name>-000*.vmdk"?

Reply
0 Kudos
racom
Enthusiast
Enthusiast

Resoved. I've turned VM off and finished consolidation then:

1. cp <disk name>-000256*.vmdk ../restore

2. grep CID ../restore/<disk name>.vmdk ../restore/<disk name>-000256.vmdk

3. vi ../restore/<disk name>-000256.vmdk

4. vmkfstools -i ../restore/<disk name>-000256.vmdk ../restore/<disk name>_1.vmdk

5. rm *.vmdk

6. mv <disk name>.vmsd <disk name>.oldvmsd

7. vmkfstools -i ../restore/<disk name>_1.vmdk <disk name>.vmdk

8. vi <disk name>.vmx

I've remove VM from inventory and add it back again before the 8. step. I had to change disk names in the 8. step. VM looks well after powering on. Thanks for help.

Reply
0 Kudos