VMware Cloud Community
GreyhoundHH
Enthusiast
Enthusiast

Can't delete files of destroyed VM

Hi!

I have a problem deleting some files of an already destroyed virtual machine. This is what I get when attempting to delete the folder:

########################

root@xxx yyy_zzz# rm -rf VM/

rm: cannot remove `VM//VM_2-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM_1-flat.vmdk': Device or resource busy

rm: cannot remove `VM//VM-53afe79d.vswp': Device or resource busy

root@xxx yyy_zzz# cd VM/

root@xxx yyy_zzz# vmkfstools -U VM_1-flat.vmdk

Failed to delete virtual disk: Device or resource busy 1048585

#######################

The files are located on a LUN shared by seven ESX 3.02 Servers. What can I do to find out which process/system is locking those files? And how can I finally remove this folder?

Any help appreciated. Thanks!

Message was edited by: GreyhoundHH

Tags (1)
Reply
0 Kudos
29 Replies
bhoriuchi
Enthusiast
Enthusiast

This is the procedure I followed in a similar situation and couldnt find the process that was lockign the files. My environment is ESX 3.5, VC 2.5

The VM that had locks on it was server01

First I logged into the host the VM was running on

Then I did an lsof | grep /server01 which gave the following output

bash 7213 root cwd DIR 0,9 2660 1056 /vmfs/volumes/46780987-011c6b63-c596-0015605393f4/server01

then I did a kill -9 7213

Reply
0 Kudos
tfcatalanojr08
Contributor
Contributor

what if you were to power down the vm clone it to another host and datastore bring it up and deltel the old vm. I would think this could resolve the issue?

Reply
0 Kudos
habibalby
Hot Shot
Hot Shot

Hello ,

This won't solve the problem. I'm facing the same issue right now, DataStore become without space and I have to clone the VM using vConverter to another DataStore. The old VM was with undeleted SnapShot, I have tried all the possible thing before I attampted to V2V it.

After I did the cold clone using vConverter I went to delete the entire folder for the effected DataStore of the Old VM. But unforturently, I cannot delete the file.

Even when I do vmkfstools -u /vmfs/volumes/VM/VM-Flat.VMDK i cannot delete and I got an error Error: Device or resource busy

Any help?

Best Regards,

Hussain Al Sayed

Revisit your posts and award points for "correct" or "helpful".

Best Regards, Hussain Al Sayed Consider awarding points for "correct" or "helpful".
Reply
0 Kudos
habibalby
Hot Shot
Hot Shot

Hello Once Again,

I have tried all the possible way to get

rid of the flat.vmdk file with no luck, the only way that solved this

issue is to Hot Storage vMotion the remaining VMs on that Store once all

of them removed, I have deleted the LUN directly for the EMC CX

storage:), rescan all the ESXs hosts to make sure that this LUN doesn't

show in the ESX.

Best Regards,

Hussain Al Sayed

Revisit your posts and award points for "correct" or "helpful".

Best Regards, Hussain Al Sayed Consider awarding points for "correct" or "helpful".
Reply
0 Kudos
AlbertWT
Virtuoso
Virtuoso

How do I kill the PID from ssh root console ?
which one is th correct PID to kill ? I'm confused here and do not want to kill the other production VM in this ESXProd 08 host ?
[root@ESXProd08 VMFS_03_LUN]# rm WebTest04-VM/
rm: cannot remove directory `WebTest04-VM/': Is a directory

[root@ESXProd08 VMFS_03_LUN]# rmdir WebTest04-VM/
rmdir: WebTest04-VM/: Directory not empty

[root@ESXProd08 WebTest04-VM]# ps -ef | grep -i WebTest04-VM
root     15521     1  0 Apr25 ?        00:00:51 /usr/lib/vmware/bin/vmkload_app64 --sched.group=host/user /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX;version=4.1.0;buildnumber=433742;licensename=VMware ESX Server;licenseversion=4.0 build-433742; -@ pipe=/tmp/vmhsdaemon-0/vmxc878b4382a283d56; /vmfs/volumes/4beb8b7c-803df79e-1047-0012793aef65/WebTest04-VM/WebTest04-VM.vmx

[root@sESXProd08 WebTest04-VM]# lsof | grep WebTest04-VM
lsof      10142   root  cwd       DIR               0,18       980     274976 /vmfs/volumes/4beb8b7c-803df79e-1047-0012793aef65/WebTest04-VM
grep      10143   root  cwd       DIR               0,18       980     274976 /vmfs/volumes/4beb8b7c-803df79e-1047-0012793aef65/WebTest04-VM
lsof      10144   root  cwd       DIR               0,18       980     274976 /vmfs/volumes/4beb8b7c-803df79e-1047-0012793aef65/WebTest04-VM
bash      20645   root  cwd       DIR               0,18       980     274976 /vmfs/volumes/4beb8b7c-803df79e-1047-0012793aef65/WebTest04-VM

[root@ESXProd08 WebTest04-VM]#
/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
gguereque
Contributor
Contributor

Hi,

I had this same problem after trying to do a restore of a vm and I had to restart the host. After doing that, I was able to delete those VMDK files.

Thanks a lot!

Reply
0 Kudos
mshapiro
Contributor
Contributor

Hi,

I experienced the same exact issue where I could not delete some vmdk files from an original VM that I had cloned for the same "Device or Resource" still in use messages.  In my case I found that my VMware Data Recovery appliance had integrated those vmdks into its configuration so was locking them.

That's what I get for not cleaning up old backups.  =P

Reply
0 Kudos
gguereque
Contributor
Contributor

Hi Team,

Just to let you know; your answers have helped me a lot. We are starting operations in China and I was able to pull up a plan to build, export, import all our Operations Server (VMs) in our vCenter in China .

Best regards!

Reply
0 Kudos
Scott610
Contributor
Contributor

I know this is an old thread, but I recently tripped over this same problem, unable to delete VMDK files for a VM that had been deleted.

After poking around for a bit, I went to the vmhost and looked at the directory from there, and I saw that the files the GUI wouldn't delete, also had "DELTA" files that weren't listed in the datastore browser.

I deleted the files from the command line without an issues.  Hope this helps someone else.

Reply
0 Kudos
dekoshal
Hot Shot
Hot Shot

To make sure you reboot the right host verify which host is holding the lock on the VM file

Run command vmkfstools -D <file name>

Out put should show the MAC address of the NIC holding the lock on the file. In below screen shot you can see that highlighted part of output is MAC of VMNIC0 on esx-02

Once Host is identified you may go-ahead with reboot and that should release the lock.

screenlock.jpg

If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.

Best Regards,

Deepak Koshal

CNE|CLA|CWMA|VCP4|VCP5|CCAH