Virtual machine with snapshots that are not showing in snapshot manager

[chuckado1]

Hello,

I have a virtual machine with a 100 plus snapshots on it probably taken by some backup appliance.  These snapshots do not appear in snapshot manager and disk consolidation fails due to file locking.  I have tried to use vim-cmd vmsvc/snapshot.get vmid to see the snapshots but it does not see them.  This machine is running very slowly due to the number of snapshots.  I was hoping for some advice on how to get these removed.

[TheBobkin]

Hello Chuck,

First stop any back-up job/process that is running against this VM.

Next check which snapshots are being used and part of the disk-chain of each attached Hard Disk - Right-click VM in inventory > Edit Settings > Click each Hard Disk and note the Disk File location (e.g VMName-000001.vmdk).

(Alternatively you can SSH to the host that this VM is registered on and run # cat VMName.vmx | grep vmdk)

Then get an SSH session to the host that the VM is currently running on and use cd to change directory into the folder(s) the disks/snapshots are located and check which disks/snapshots are locked:

# vmkfstools -v10 -q VMName-000001.vmdk (change this to the last snapshot in the chain as per above of course)

This should tell you what is locked.

Then you should be able to determine the locker of this disk (proxy VMs are typical culprits!) using vmfsfilelockinfo, lsof | egrep and esxcli vm process list to determine exactly what is locking disks:

kb.vmware.com/kb/2110152

Bob

[vasan22in]

Hello,

Browse the datastore VM folder, you can see all snapshot files inside. For File lock issue,

SSH'd to the ESXi server and checked the hostd.log

tail -f /var/log/hostd.log

Run this to identify which ESXi host the vmdk file was locked

vmkfstools -D /vmfs/volumes/yourvolume/yourVM/yourlockedVM.vmdk

You can see a single entry for "RO Owner" which had the lock and the MAC address.This is the ESXi server which has the lock on the VMDK file. Next locate which ESXi host has a network adaptor with that MAC address.

Once confirmed I placed the host in maintenance mode, DRS vMotioned all VMs to another host in the cluster and restarted the hostd service.

/etc/init.d/hostd restart

Once the hostd service had restarted I performed a Snapshot -> Consolidate on the VM and it completed successfully.

Refer the url for more information.

Investigating virtual machine file locks on ESXi (10051) | VMware KB

[continuum]

When you have a VM with more than the officially allowed snapshots you DO NOT USE any buildin consolidation features.
Thats way too risky and prone to failures that will make matters even worse.
Instead you use vmkfstools -i to clone each snapshot-chain into a new vmdk.
If you need top do that while the VM is busy - look up the current vmdk.
Open the descriptor of the current vmdk and look up its parent.
Then clone the parent to a new vmdk.
This can be done while the VM is running.
Once the vmkfstools -i clone is done you only need a small maintenance window to attach the current vmdk to the newly created clone.
When that is done the consolidation of the remaining delta can be done on the fly.
If you need  detailed instructions post:
filelisting including date and size
descriptor of cuurent vmdk
vmx-file
vmsd-file
all vmware.logs

[TheBobkin]

Good point Continuum,

I was thinking of this question as 100 snapshots over multiple disks but if it is all on one disk-chain then better to use clone in batches of 30 or so (works for ~50 at a time but size matters too I have noticed). Been too long since I worked in Infrastructure team - this kind of thing used to be my bread&butter!

Bob

[continuum]

Hello Bob

> this kind of thing used to be my bread&butter!
This stuff is bread&butter for me now.
Unfortunately its only bread most of the times - need to learn how to write bigger invoices :smileycry:
Ulli

[chuckado1]

Good news, I was able to find the backup proxies in the environment that were causing the file locking issue and I was able to successfully consolidate the virtual machine (it took 17 hours).  I am now concerned that this issue may be widespread in the environment and was wondering if anyone had a powercli script handy to search through the datastores to find additional machines with these hidden snapshots?

Thanks

[TheBobkin · Accepted solution]

Hello Chuck,

"proxy VMs are typical culprits!" - Glad that helped!

This will show you the directory path of all snapshots in the environment:

#cd /vmfs/volumes/

# find . iname *000*vmdk

Then you can trim this to just print the name portion of the VM/namespace and 'uniq' so the list is shorter.

Bob