VMware Cloud Community
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

HELP!! Can't remove datastore - "in use"

I've tried every suggestion out there, and still can't remove one datastore in my vCenter.

(I've not been able to reboot the hosts, or even put them in maintnenace - that's not an option at the moment).

2 -

I've done the following:

- migrated all VM's.

- checked all vmx files for references to the datastore

   - found 2 VM's not in inventory or powered on, which referenced the datastore - changed those

- deleted the .dvsData folder from the datastore

- Not used for HA (the 2 hosts are not in a cluster)

- made sure I/O Control is disabled on all hosts

    - even followed this: VMware Front Experience: How to disable Storage I/O Control for an unavailable datastore

- no snapshots exist on any of the vm's that *were* on this datastore

   - also checked for orphaned *-00000x.vmdk on migrated vm's - none exist

- unmounted the datastore

    - this works from one host, but not the other.

    - unmounting from host1 gives the red "X" on "no virtual machines exist (all others are green)

    - unmounting from host2 unmounts the datastore on host2 but it still shows as mounted on host1

- deleting the datastore (whether mounted or not) fails with the "resource in use" error

Reply
0 Kudos
1 Solution

Accepted Solutions
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

SUCCESS!!!

I meant to say that I did services.sh restart from an SSH session.

That did not work.

Then, I noticed I could choose Unmount from a direct session to host2, but still not successfully unmount.

I went back to vCenter, and got the red X on "active VM's" as soon as I tried to unmount via the same host (yes...aha...the host will let me choose Unmount, but vCenter will not...hmmm).

I rebooted the vCenter server, and Unmounted the datastore before the vCenter services could start.

Not sure if the reboot of vCenter did the trick, or Unmounting and deleting *before* vCenter services could start did the trick.

THANK YOU SO MUCH FOR YOUR HELP!!!!!

ANSWER:

from OP, do all of those things.

- Check for open processes using this:

    vsish -e ls /storage/scsifw/devices/<"naa. id of your datastore">/worlds/ |sed 's:/::' |while read i;do ps |grep $i;done

- Restart management agents on the host(s): services.sh restart

- Reboot vCenter server (or at the least, restart vCenter services).

At some point in this list, you should be able to unmount and delete the datastore.

View solution in original post

Reply
0 Kudos
17 Replies
tomtom901
Commander
Commander
Jump to solution

Can you do ls -allh via SSH in the datastore? So, in steps

1) Connect via SSH to the host that will not unmount

2) do cd /vmfs/volumes/<name of your datastore>

3) do ls -allh

And share the output.

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

~ # ls -allh /vmfs/volumes/san_lun3/

drwxr-xr-t    1 root     root        1.1K Oct 27 17:10 .

drwxr-xr-x    1 root     root         512 Oct 27 18:15 ..

-r--------    1 root     root        5.2M Jun 14  2012 .fbb.sf

-r--------    1 root     root      254.7M Jun 14  2012 .fdc.sf

-r--------    1 root     root        1.1M Jun 14  2012 .pb2.sf

-r--------    1 root     root      256.0M Jun 14  2012 .pbc.sf

-r--------    1 root     root      250.6M Jun 14  2012 .sbc.sf

-r--------    1 root     root        4.0M Jun 14  2012 .vh.sf

~ #

tomtom901
Commander
Commander
Jump to solution

Those would be VMFS meta files so quite frankly there shouldn't be a reason not to remove this datastore. Is it possible that the the datastore on the other host is used as a swap datastore? Configuration -> Software -> Virtual Machine Swapfile location? Or perhaps a log directory (although there should be log files present then) Configuration -> Software -> Advanced Settings -> Syslog?

You could also try the unmount while tailing /var/log/vmkernel.log (tail -f /var/log/vmkernel.log)

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

Your response led me to this post:

Re: Can't unmount datastore(s) - file system is busy error even though it's not in use

Which leads me to  few processes which seem to have files locked on san_lun3 - even though the VM's have been motioned off the datastore.

One of the issues is an open KMS session from a system that is not active (someone else has a console window open to one VM - I can't find the session or the user, I'd like to just kill all console sessions to this VM if I can.

13393497 13392045 vmx-mks:SEPM-Win7 /bin/vmx

13393498 13392045 vmx-svga:SEPM-Win7 /bin/vmx

13393499 13392045 vmx-vcpu-0:SEPM-Win7 /bin/vmx

The other issue I see are some files related to another VM which has been motioned off the datastore:

6458291 6458291 vmx                  /bin/vmx

6458297 6458291 vmx-vthread-5:DC1    /bin/vmx

6462836 6458291 vmx-vthread-6:DC1    /bin/vmx

6462837 6458291 vmx-mks:DC1          /bin/vmx

6462838 6458291 vmx-svga:DC1         /bin/vmx

6462841 6458291 vmx-vcpu-0:DC1       /bin/vmx

6462842 6458291 vmx-vcpu-1:DC1       /bin/vmx

tomtom901
Commander
Commander
Jump to solution

If you are certain no VM's are running on the host in question, you could do a kill -9 6458291 && kill -9 13392045 to kill the proces. Then (if the VM didn't die :smileysilly:) try unmounting again.

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

I found a less intrusive method:

for the 2 "sessions" I noted above, I vMotioned the HOST (not datastore) to another host, then back again.

As soon as I motioned to a different host, the KMS files disappeared from the list of "locked" files/processes.

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

ARGH!!

I thought I had it nailed - the host which has *NO* kms sessions or anything looking like the previous post - I am still unable to unmount or delete the datastore.

From host2, the unmount still gives me the red X on "no VM files".

Both hosts have a *bunch* of these (I can provide the whole list if that may help):

918368 9135 hostd-worker         hostd

918369 9135 hostd-worker         hostd

918370 9135 hostd-worker         hostd

918371 9135 hostd-worker         hostd

9135 9135 hostd-worker         hostd

9136 9135 hostd-poll           hostd

9137 9135 hostd-worker         hostd

9138 9135 hostd-worker         hostd

9921 9135 hostd-worker         hostd

9923 9135 hostd-worker         hostd

9924 9135 hostd-worker         hostd

9988 9135 hostd-worker         hostd

9989 9135 hostd-worker         hostd

10026 9135 hostd-vix-worke      hostd

10027 9135 hostd-vix-worke      hostd

Reply
0 Kudos
tomtom901
Commander
Commander
Jump to solution

What you could try is tailing the vmkernel.log located in /var/log when you're doing the unmount. Perhaps that gives some pointers.

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

tailing vmkernel.log yields *nothing* when I hit "Dismount".

The box with red X for "no VM files" pops up immediately.

I also tried to go straight to Delete datastore, and nothing was logged in vmkernel.log.

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

I've restarted the management agents as well, thinking there might be something "locked" via the management agent.

Still no love on deleting this datastore.

Reply
0 Kudos
tomtom901
Commander
Commander
Jump to solution

You could try running services.sh restart to ensure every daemon is restarted, or even reboot the entire host. I think there is a process locked on this datastore, but it's hard to verify remotely.

Reply
0 Kudos
jonathanp
Expert
Expert
Jump to solution

we had some issues with that in the past...

One thing....

Are you using a backup proxy that could have the datastore mounted, or locks not released...

In the past we have had to reboot the proxy or do some other stuff to do that...

Regards

Reply
0 Kudos
mikejroberts
Enthusiast
Enthusiast
Jump to solution

I had this exact same issue on 5.5 U1 and I worked with support and did not get an answer.  I had to reboot all 10 hosts in one of my clusters to be able to unmount the datastores.  Support was not helpful.  They could not find what was locking the LUNs.

Reply
0 Kudos
jonathanp
Expert
Expert
Jump to solution

Reply
0 Kudos
snowdog_2112
Enthusiast
Enthusiast
Jump to solution

SUCCESS!!!

I meant to say that I did services.sh restart from an SSH session.

That did not work.

Then, I noticed I could choose Unmount from a direct session to host2, but still not successfully unmount.

I went back to vCenter, and got the red X on "active VM's" as soon as I tried to unmount via the same host (yes...aha...the host will let me choose Unmount, but vCenter will not...hmmm).

I rebooted the vCenter server, and Unmounted the datastore before the vCenter services could start.

Not sure if the reboot of vCenter did the trick, or Unmounting and deleting *before* vCenter services could start did the trick.

THANK YOU SO MUCH FOR YOUR HELP!!!!!

ANSWER:

from OP, do all of those things.

- Check for open processes using this:

    vsish -e ls /storage/scsifw/devices/<"naa. id of your datastore">/worlds/ |sed 's:/::' |while read i;do ps |grep $i;done

- Restart management agents on the host(s): services.sh restart

- Reboot vCenter server (or at the least, restart vCenter services).

At some point in this list, you should be able to unmount and delete the datastore.

Reply
0 Kudos
francsza
Contributor
Contributor
Jump to solution

One other usefull command is "lsof" .It lists all open files on linux

You can then search for the datastore you're trying to unmount and deal with whatever has an open file on it.

In my case it was vsantrace files being written to the datastore

Below are the steps and output from the commands:

/vmfs/volumes/# lsof |grep 55363c53-3d5bb128-f945-3440b59e9294

289619      vsantracedUrgen       FILE                        5   /vmfs/volumes/55363c53-3d5bb128-f945-3440b59e9294/vsantraces/vsantracesUrgent--2016-11-10T13h41m14s206.gz

289590      vsantraced            FILE                        5   /vmfs/volumes/55363c53-3d5bb128-f945-3440b59e9294/vsantraces/vsantraces--2016-11-10T13h41m13s175.gz

/vmfs/volumes/ # /etc/init.d/vsantraced stop

watchdog-vsantraced: Terminating watchdog process with PID 289567

vsantraced stopped

watchdog-vsantracedUrgen: Terminating watchdog process with PID 289596

vsantracedUrgen stopped

/vmfs/volumes # esxcli storage filesystem unmount -u 55363c53-3d5bb128-f945-3440b59e9294

/vmfs/volumes # esxcli storage filesystem list

Mount Point                                        Volume Name  UUID                                 Mounted  Type                       Size      Free

-------------------------------------------------  -----------  -----------------------------------  -------  --------------------  ---------  --------

                                                   VMLUN9FC     55363c53-3d5bb128-f945-3440b59e9294    false  VMFS-unknown version          0         0

/vmfs/volumes/1e40e8e6-2145a0e7-9273-3c08c1db46ab               1e40e8e6-2145a0e7-9273-3c08c1db46ab     true  vfat                  261853184  48779264

/vmfs/volumes/15b08ce4-7333041b-4a3f-4739103e28a5               15b08ce4-7333041b-4a3f-4739103e28a5     true  vfat                  261853184  48783360

/vmfs/volumes/506da079-0a6bdc0c-e974-3440b59c3c5a               506da079-0a6bdc0c-e974-3440b59c3c5a     true  vfat                  299712512  96813056

/vmfs/volumes # esxcfg-scsidevs -m

naa.6006016038a0280032d8054ed48ae111:1                           /vmfs/devices/disks/naa.6006016038a0280032d8054ed48ae111:1 55363c53-3d5bb128-f945-3440b59e9294  0  VMLUN9FC

/vmfs/volumes # esxcli storage core  device set --state=off -d naa.6006016038a0280032d8054ed48ae111

Reply
0 Kudos
vmkfix-SSA
Contributor
Contributor
Jump to solution

I followed the below article to resolve the issue. May be it help to you.

https://vmkfix.blogspot.com/2023/04/vmware-unmount-datastorevolume-from.html 

Reply
0 Kudos