Orphaned vmdk files after Virtual Machines have been “Deleted from Disk”

[SKIRK505]

Recently we deleted a few virtual machines that were no longer being used. We powered off the virtual machines, right clicked on them, and selected the “delete from disk” option in the menu. We noticed later that the .vmdk files for the virtual machines were left behind on the SAN.

We have been reviewing KB articles about file locks and such, but everything we had read assumes that there is a virtual machine associated with the vmdk and has it locked. Since we deleted the virtual machines we haven’t been able to locate what has the files locked.

We received no error message at the time we deleted the virtual machines, but now that we attempt to delete the orphaned vmdk files we get .. “Cannot delete file [<datastorename>] <vmname-flat.vmdk>”.. with no explanation as what is preventing it.

I have rebooted all the hosts in the environment in an attempt to clear any file locks, created new virtual machines and attempted to associate the vmdk file with it, and even look into the SAN, but without success. The files range from 16-31GB and since want the storage back, I need to find a way to remove them.

[steveb05]

When you say "looked into the SAN", what did you do? What SAN do you have?

You can SSH into a host, and do a long list (ls -l)  of that datastore directory to see if there are any hidden lock files.

[SKIRK505]

Good question about the SAN.. I looked to make sure that we didn't have any SAN level file locking turned on and also to see if we could do anything at the SAN level to remove the files. This environment is using a HP P2000 MSA, fairly basic SAS connected SAN.

[steveb05]

I only have two suggestions:
1) If you have any backup systems attached to the SAN, check/reboot them

2) Try changing the ownership of the vdisk in the SAN to see if the controller has some kind of lock on the file

Aside from that, if you have support, you might call VMware or HP. With that SAN direct attached over SAS, it greatly reduces the list of things that could be locking the files.

[SKIRK505]

I did a manual fail-over of the Storage controllers, but it did not change anything.

Vmware got back in touch with me yesterday afternoon, but for the most part had me repeat steps I had already attempted earlier from the KBs on File Locks. Since the original virtual machine is gone thought, these KBs really don't apply.

I think the next Option will Roll Shut Downs. Vmware indicated that Rebooting the host doesn't always remove file locks.. interesting.

[SKIRK505]

Ran a few commands look for the file locks.. here is what I got, but since the VM is gone, I'm unsure what direction to go next.

Locked Vile VSTORE203-flat.vmdk

~ # vmkfstools -D vmfs/volumes/Raid5Vol6/VSTORE203/VSTORE203-flat.vmdk
Lock [type 10c00001 offset 217247744 v 561, hb offset 3158016
gen 113, mode 0, owner 00000000-00000000-0000-000000000000 mtime 103782 nHld 0 nOvf 0]
Addr <4, 493, 22>, gen 533, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 32104251392, nb 13531 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

Locked Vile VSTORE206-flat.vmdk

~ # vmkfstools -D vmfs/volumes/Raid5Vol6/VSTORE206/VSTORE206-flat.vmdk
Lock [type 10c00001 offset 154261504 v 306, hb offset 3158016
gen 113, mode 0, owner 00000000-00000000-0000-000000000000 mtime 39254 nHld 0 nOvf 0]
Addr <4, 339, 131>, gen 276, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 24763170816, nb 8034 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

Locked Vile VSTORE207-flat.vmdk

~ # vmkfstools -D vmfs/volumes/Raid5Vol6/VSTORE207/VSTORE207-flat.vmdk
Lock [type 10c00001 offset 154296320 v 318, hb offset 3158016
gen 113, mode 0, owner 00000000-00000000-0000-000000000000 mtime 103724 nHld 0 nOvf 0]
Addr <4, 339, 148>, gen 288, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 24015536128, nb 7241 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

Locked Vile VSTORE209-flat.vmdk

~ # vmkfstools -D vmfs/volumes/Raid5Vol6/VSTORE209/VSTORE209-flat.vmdk
Lock [type 10c00001 offset 217333760 v 595, hb offset 3158016
gen 113, mode 0, owner 00000000-00000000-0000-000000000000 mtime 104202 nHld 0 nOvf 0]
Addr <4, 493, 64>, gen 541, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 17319329792, nb 2709 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

[continuum]

If vmdks that are no longer in use cant be deleted - try to rename them ( I use WinSCP )
If that works - also reame the directory where they are stored in.
If that also works - wait half an hour and try to delete the directory again.

That worked for me a few times when I run into the same problem

[SKIRK505]

Renaming worked.. I'll wait an hour and see if it lets me delete them.

[Jrlouk]

THANK YOU for this post!  Just saved me from hours of head scratching and trial and error!