Unable to delete or unmount a datastore because file system is busy

[Sam30]

ESXi 5.5 VMFS3

I'm trying to delete or unmount a datastore but both are failing with below error messages.

UNMOUNT. - Out of the 5 hosts in cluster it is unmounted from 4 while it's still mounted on one host & gives below error

Call "HostStorageSystem.UnmountVmfsVolume" for object "storageSystem-3994" on vCenter Server "xyz" failed.

Cannot unmount volume 'Datastore Name: abc VMFS uuid: 507978c7-f56a954c-b170-782bcb507614' because file system is busy. Correct the problem and retry the operation.

DELETE - if trying to delete it's giving me below error.

Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-3994" on vCenter Server "xyz" failed.

Cannot remove datastore 'Datastore Name: abc VMFS uuid: 507978c7-f56a954c-b170-782bcb507614' because file system is busy. Correct the problem and retry the operation.

I browsed through the datastore, there's only folder named .vSphere-HA while if I check the datastore hearbeating in HA properties, this datastore is not a part of Datastore heartbeats as well.

Tried restarting mgmt services - Didn't work

Tried disabling HA & then deleting/unmounting datastore - Didn't work

Tried http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=206917... - Didn't work

Rebooting the host would be last option, but I'm looking for a fix on this.

[Dee006]

Sam,to be frank i didn't come across the issue if it happens I will request for downtime :smileysilly: to restart the host.After googling it may happens due to default Syslog.global.logDir directory.

Löschen eines lokalen VMFS schlägt fehl – Thomas-Krenn-Wiki

VMware had open a KB for this issue,but no resolution for the issue.

Hope those links helps you.

[dhanarajramesh]

May be HA using for Data store Heart-beating, check the cluster setting whether its been using and also check whether any of the hosts are using as Host cache.

[Sam30]

Dee, I already checked that & my hosts scratch partition is pointing to local disk which has alot of free space.

[Sam30]

Dhanraj, like I mentioned in the beginning I already verified that.

[dhanarajramesh]

in another way of thinking, can you check this post Deleting the undeletable datastore: The resource is in use; Call “HostDatastoreSystem.RemoveDatastor...

[Sam30]

Nope, like I mentioned the datastore only has .vSphere-HA folder & no other logs as mentioned in the post you sent.

And I dont have any VM's hardcoded to any particular datastore.

[Alistar]

Hi there,

I've had a similar issue once - try stopping the StorageRM service via SSH command /etc/init.d/storageRM stop and then deleting the datastore - see if it helps

Oh, and also try lsof | grep "/vmfs/volumes/your_datastore" and see what PID keeps it open and if safe, kill that process.

[Sam30]

Dee, they have updated the KB with resolution as well but that didn't help either coz my Neither am I using Qlogic iSCSI nor was my ramdisk space full

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M        1M       30M   3% --

etc                        28M      296K       27M   1% --

tmp                       192M        8K      191M   0% --

hostdstats               1053M       31M     1021M   3% --

snmptraps                   1M        0B        1M   0% --

[Sam30]

Hi Alistar

Stopping /etc/init.d/storageRM stop didn't help either.

and for lsof that works on ESX not ESXi.

I tried using this but that didn't Dump anything to check either:-

/vmfs/volumes/4f68b74e-7ad5d6f7-a763-782bcb507614 # vmkfstools -D /vmfs/volumes/4f68b74e-7ad5d6f7-a763-782bcb507614

Could not get the dump information for '/vmfs/volumes/4f68b74e-7ad5d6f7-a763-782bcb507614' (rv -1)

Could not dump metadata for '/vmfs/volumes/4f68b74e-7ad5d6f7-a763-782bcb507614': Inappropriate ioctl for device

Error: Inappropriate ioctl for device

[WilCont]

Como eliminar un datastore que fue despresentado y aun se ve en el ESXi

1 Conectarse al ESXi por SSH

2 Lista los datastore presentados para observar el UUID del Datastore con problema

# esxcli storage filesystem list

3 .# /etc/init.d/storageRM stop

4. lista los archivos abiertos en el datastore, la primera columna muestra el PID 

# lsof | grep '/vmfs/volumes/58580b99-a196935b-fa23-fc15b41d8300'

Kill a cada uno de los PID del datastore con falla, los cuales son los que no permiten eliminar el datstore

5 ejecutar el comando para desmontar el datastore

# esxcli storage filesystem unmount -u 58580b99-a196935b-fa23-fc15b41d8300

6 # /etc/init.d/storageRM start

7 #esxcli storage filesystem list 

Listo!!

[Dee006]

Hi Sam,

I've came across similar incident few days back,but the only way is rebooting the host.

[cscsbb72]

Had a similar issue, unmount from vCenter didn't work.  The tmp folder had loads of space, using the unmount command in SSH worked for me!