VMware Cloud Community
GreyhoundHH
Enthusiast
Enthusiast

Can't delete files of destroyed VM

Hi!

I have a problem deleting some files of an already destroyed virtual machine. This is what I get when attempting to delete the folder:

########################

root@xxx yyy_zzz# rm -rf VM/

rm: cannot remove `VM//VM_2-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM_1-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM-53afe79d.vswp': Device or resource busy

root@xxx yyy_zzz# cd VM/

root@xxx yyy_zzz# vmkfstools -U VM_1-flat.vmdk

Failed to delete virtual disk: Device or resource busy 1048585

#######################

The files are located on a LUN shared by seven ESX 3.02 Servers. What can I do to find out which process/system is locking those files? And how can I finally remove this folder?

Any help appreciated. Thanks!

Message was edited by: GreyhoundHH

Tags (1)
Reply
0 Kudos
29 Replies
LarsLiljeroth
Expert
Expert

As far as i know there is no way to see what is locking up your files.

We had a simillar issue. Our solutions was to restart every host in our cluster once at a time. After the 7 host out of 9 we were able to delete the files... Smiley Wink

This can be done without down time... Try it out.

// Lars Liljeroth -------------- *If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!
GreyhoundHH
Enthusiast
Enthusiast

Hi!

Yeah, I had this also in my mind, but I was hoping to find an alternative, since these are customer-systems.

If I wanted to avoid rebooting the hosts, what would be the services on the hosts that might be locking those files?

thx

Reply
0 Kudos
TCP
Enthusiast
Enthusiast

Hi!

Try to use vmware-cmd tool from service console

for example

vmware-cmd /vmfs/volumes/......../vmname/vmname.vmx getstate - current state of your virtual machine

vmware-cmd /vmfs/volumes/......../vmname/vmname.vmx stop(reset) - stop or reset vm

than try to remove files.

VCP3, VCP4
Reply
0 Kudos
GreyhoundHH
Enthusiast
Enthusiast

Hi!

The VM has already been destroyed, there are no other files left than those *.vmdk and the *.vswp.

cya

Reply
0 Kudos
stuart_ling
Contributor
Contributor

Is it possible that these disks are mounted on another VM?

Reply
0 Kudos
demz
Expert
Expert

Are you sure that the disk isn't attached to another VM ?

It was my issue...

Reply
0 Kudos
GreyhoundHH
Enthusiast
Enthusiast

I was thinking of this possibility too and asked the customer if he could confirm that this disk is nowhere else attached. The answer is still pending...

But what I'm wondering now is that there are only the *-flat.vmdk files left, not the one without "-flat". Since this is missing it can't be attached to any other VM, can it?

thx

Reply
0 Kudos
aldin
Enthusiast
Enthusiast

is that VM cloned? we have similar issue with machines that was cloned. we restart esx and than delete files. i think it is in some way linked to its origin.

Reply
0 Kudos
demz
Expert
Expert

Just create a new VM with similar settings and replace its -flat.vmdk files

(edit the .vmdk without flat in order to correct issues if there are...)

or just copy another .vmdk and change all settings.

Reply
0 Kudos
demz
Expert
Expert

If there isn't .vmdk without -flat, it's maybe an old VM created with ESX 2.5

Reply
0 Kudos
GreyhoundHH
Enthusiast
Enthusiast

I'm afraid I can't give further details on this machine's history. I've just taken over this environment and can't tell if it has been cloned, but I doubt that it has been creatied in ESX 2.x.

Since I'm not supposed to do any work-arounds in the VirtualCenter I think I'll advice the customer to sequentially reboot all their hosts.

cya

Reply
0 Kudos
demz
Expert
Expert

Beware ! If there's no .vmdk (descriptor), the concerned VM won't be able to start after a shutdown !

Try first the way I told you recreating a VM and replace the -flat.vmdk.

Reply
0 Kudos
LarsLiljeroth
Expert
Expert

Did you solve this issue ? Just curious for future reference... /lars

// Lars Liljeroth -------------- *If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!
Reply
0 Kudos
GreyhoundHH
Enthusiast
Enthusiast

No, not really 😕

I've informed the customer that we should try to reboot the host on which that destroyed VM was registered. I hope that will release the lock on those files.

I'll keep this thread updated as soon as I can see any progress..

cya

Reply
0 Kudos
Jasemccarty
Immortal
Immortal

Try the following command from the ESX console:

lsof | grep filename-you-wish-to-look-for.vmdk

This will show you which process has the files open.

If it is an old VM pid, then kill the pid with:

kill -9 PID

Good luck,

Jase McCarty

http://www.jasemccarty.com

Jase McCarty - @jasemccarty
Reply
0 Kudos
GreyhoundHH
Enthusiast
Enthusiast

In fact, that was one of the first things I've tried, but I didn't get any hits even on "lsof | grep vmdk". I've tried that command on several environments and hosts...

cya

Reply
0 Kudos
Kris_H
Contributor
Contributor

I just had the same problem with one of the VM's running on one of our ESX 3.0.2 servers. I tried virtually everything i could think of (and not think of :P) but in the end I had to reboot the ESX server the VM was last registered to. After I did that I could delete the remaining files without any problem. Smiley Happy

Reply
0 Kudos
LarsLiljeroth
Expert
Expert

Excactly the same as i have seen.. Sadly as it is...

Has anyone been able to recover from a file lock like this without having to reboot their host. ?

// Lars Liljeroth -------------- *If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!
Reply
0 Kudos
Jasemccarty
Immortal
Immortal

I had an issue one time, that dealt with this.

Because I had tried a vMotion somewhere in the mix, I had to run lsof on all of my hosts, to determine which one had it open.

I killed the process on another host (the one I had attempted to vMotion to), and I was good to go.

Jase McCarty

http://www.jasemccarty.com

Jase McCarty - @jasemccarty
Reply
0 Kudos