I've tried every suggestion out there, and still can't remove one datastore in my vCenter.
(I've not been able to reboot the hosts, or even put them in maintnenace - that's not an option at the moment).
2 -
I've done the following:
- migrated all VM's.
- checked all vmx files for references to the datastore
- found 2 VM's not in inventory or powered on, which referenced the datastore - changed those
- deleted the .dvsData folder from the datastore
- Not used for HA (the 2 hosts are not in a cluster)
- made sure I/O Control is disabled on all hosts
- even followed this: VMware Front Experience: How to disable Storage I/O Control for an unavailable datastore
- no snapshots exist on any of the vm's that *were* on this datastore
- also checked for orphaned *-00000x.vmdk on migrated vm's - none exist
- unmounted the datastore
- this works from one host, but not the other.
- unmounting from host1 gives the red "X" on "no virtual machines exist (all others are green)
- unmounting from host2 unmounts the datastore on host2 but it still shows as mounted on host1
- deleting the datastore (whether mounted or not) fails with the "resource in use" error
SUCCESS!!!
I meant to say that I did services.sh restart from an SSH session.
That did not work.
Then, I noticed I could choose Unmount from a direct session to host2, but still not successfully unmount.
I went back to vCenter, and got the red X on "active VM's" as soon as I tried to unmount via the same host (yes...aha...the host will let me choose Unmount, but vCenter will not...hmmm).
I rebooted the vCenter server, and Unmounted the datastore before the vCenter services could start.
Not sure if the reboot of vCenter did the trick, or Unmounting and deleting *before* vCenter services could start did the trick.
THANK YOU SO MUCH FOR YOUR HELP!!!!!
ANSWER:
from OP, do all of those things.
- Check for open processes using this:
vsish -e ls /storage/scsifw/devices/<"naa. id of your datastore">/worlds/ |sed 's:/::' |while read i;do ps |grep $i;done
- Restart management agents on the host(s): services.sh restart
- Reboot vCenter server (or at the least, restart vCenter services).
At some point in this list, you should be able to unmount and delete the datastore.
Can you do ls -allh via SSH in the datastore? So, in steps
1) Connect via SSH to the host that will not unmount
2) do cd /vmfs/volumes/<name of your datastore>
3) do ls -allh
And share the output.
~ # ls -allh /vmfs/volumes/san_lun3/
drwxr-xr-t 1 root root 1.1K Oct 27 17:10 .
drwxr-xr-x 1 root root 512 Oct 27 18:15 ..
-r-------- 1 root root 5.2M Jun 14 2012 .fbb.sf
-r-------- 1 root root 254.7M Jun 14 2012 .fdc.sf
-r-------- 1 root root 1.1M Jun 14 2012 .pb2.sf
-r-------- 1 root root 256.0M Jun 14 2012 .pbc.sf
-r-------- 1 root root 250.6M Jun 14 2012 .sbc.sf
-r-------- 1 root root 4.0M Jun 14 2012 .vh.sf
~ #
Those would be VMFS meta files so quite frankly there shouldn't be a reason not to remove this datastore. Is it possible that the the datastore on the other host is used as a swap datastore? Configuration -> Software -> Virtual Machine Swapfile location? Or perhaps a log directory (although there should be log files present then) Configuration -> Software -> Advanced Settings -> Syslog?
You could also try the unmount while tailing /var/log/vmkernel.log (tail -f /var/log/vmkernel.log)
Your response led me to this post:
Re: Can't unmount datastore(s) - file system is busy error even though it's not in use
Which leads me to few processes which seem to have files locked on san_lun3 - even though the VM's have been motioned off the datastore.
One of the issues is an open KMS session from a system that is not active (someone else has a console window open to one VM - I can't find the session or the user, I'd like to just kill all console sessions to this VM if I can.
13393497 13392045 vmx-mks:SEPM-Win7 /bin/vmx
13393498 13392045 vmx-svga:SEPM-Win7 /bin/vmx
13393499 13392045 vmx-vcpu-0:SEPM-Win7 /bin/vmx
The other issue I see are some files related to another VM which has been motioned off the datastore:
6458291 6458291 vmx /bin/vmx
6458297 6458291 vmx-vthread-5:DC1 /bin/vmx
6462836 6458291 vmx-vthread-6:DC1 /bin/vmx
6462837 6458291 vmx-mks:DC1 /bin/vmx
6462838 6458291 vmx-svga:DC1 /bin/vmx
6462841 6458291 vmx-vcpu-0:DC1 /bin/vmx
6462842 6458291 vmx-vcpu-1:DC1 /bin/vmx
If you are certain no VM's are running on the host in question, you could do a kill -9 6458291 && kill -9 13392045 to kill the proces. Then (if the VM didn't die :smileysilly:) try unmounting again.
I found a less intrusive method:
for the 2 "sessions" I noted above, I vMotioned the HOST (not datastore) to another host, then back again.
As soon as I motioned to a different host, the KMS files disappeared from the list of "locked" files/processes.
ARGH!!
I thought I had it nailed - the host which has *NO* kms sessions or anything looking like the previous post - I am still unable to unmount or delete the datastore.
From host2, the unmount still gives me the red X on "no VM files".
Both hosts have a *bunch* of these (I can provide the whole list if that may help):
918368 9135 hostd-worker hostd
918369 9135 hostd-worker hostd
918370 9135 hostd-worker hostd
918371 9135 hostd-worker hostd
9135 9135 hostd-worker hostd
9136 9135 hostd-poll hostd
9137 9135 hostd-worker hostd
9138 9135 hostd-worker hostd
9921 9135 hostd-worker hostd
9923 9135 hostd-worker hostd
9924 9135 hostd-worker hostd
9988 9135 hostd-worker hostd
9989 9135 hostd-worker hostd
10026 9135 hostd-vix-worke hostd
10027 9135 hostd-vix-worke hostd
What you could try is tailing the vmkernel.log located in /var/log when you're doing the unmount. Perhaps that gives some pointers.
tailing vmkernel.log yields *nothing* when I hit "Dismount".
The box with red X for "no VM files" pops up immediately.
I also tried to go straight to Delete datastore, and nothing was logged in vmkernel.log.
I've restarted the management agents as well, thinking there might be something "locked" via the management agent.
Still no love on deleting this datastore.
You could try running services.sh restart to ensure every daemon is restarted, or even reboot the entire host. I think there is a process locked on this datastore, but it's hard to verify remotely.
we had some issues with that in the past...
One thing....
Are you using a backup proxy that could have the datastore mounted, or locks not released...
In the past we have had to reboot the proxy or do some other stuff to do that...
Regards
I had this exact same issue on 5.5 U1 and I worked with support and did not get an answer. I had to reboot all 10 hosts in one of my clusters to be able to unmount the datastores. Support was not helpful. They could not find what was locking the LUNs.
Also to see information on the lun ( about what is using it )
SUCCESS!!!
I meant to say that I did services.sh restart from an SSH session.
That did not work.
Then, I noticed I could choose Unmount from a direct session to host2, but still not successfully unmount.
I went back to vCenter, and got the red X on "active VM's" as soon as I tried to unmount via the same host (yes...aha...the host will let me choose Unmount, but vCenter will not...hmmm).
I rebooted the vCenter server, and Unmounted the datastore before the vCenter services could start.
Not sure if the reboot of vCenter did the trick, or Unmounting and deleting *before* vCenter services could start did the trick.
THANK YOU SO MUCH FOR YOUR HELP!!!!!
ANSWER:
from OP, do all of those things.
- Check for open processes using this:
vsish -e ls /storage/scsifw/devices/<"naa. id of your datastore">/worlds/ |sed 's:/::' |while read i;do ps |grep $i;done
- Restart management agents on the host(s): services.sh restart
- Reboot vCenter server (or at the least, restart vCenter services).
At some point in this list, you should be able to unmount and delete the datastore.
One other usefull command is "lsof" .It lists all open files on linux
You can then search for the datastore you're trying to unmount and deal with whatever has an open file on it.
In my case it was vsantrace files being written to the datastore
Below are the steps and output from the commands:
/vmfs/volumes/# lsof |grep 55363c53-3d5bb128-f945-3440b59e9294
289619 vsantracedUrgen FILE 5 /vmfs/volumes/55363c53-3d5bb128-f945-3440b59e9294/vsantraces/vsantracesUrgent--2016-11-10T13h41m14s206.gz
289590 vsantraced FILE 5 /vmfs/volumes/55363c53-3d5bb128-f945-3440b59e9294/vsantraces/vsantraces--2016-11-10T13h41m13s175.gz
/vmfs/volumes/ # /etc/init.d/vsantraced stop
watchdog-vsantraced: Terminating watchdog process with PID 289567
vsantraced stopped
watchdog-vsantracedUrgen: Terminating watchdog process with PID 289596
vsantracedUrgen stopped
/vmfs/volumes # esxcli storage filesystem unmount -u 55363c53-3d5bb128-f945-3440b59e9294
/vmfs/volumes # esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ----------- ----------------------------------- ------- -------------------- --------- --------
VMLUN9FC 55363c53-3d5bb128-f945-3440b59e9294 false VMFS-unknown version 0 0
/vmfs/volumes/1e40e8e6-2145a0e7-9273-3c08c1db46ab 1e40e8e6-2145a0e7-9273-3c08c1db46ab true vfat 261853184 48779264
/vmfs/volumes/15b08ce4-7333041b-4a3f-4739103e28a5 15b08ce4-7333041b-4a3f-4739103e28a5 true vfat 261853184 48783360
/vmfs/volumes/506da079-0a6bdc0c-e974-3440b59c3c5a 506da079-0a6bdc0c-e974-3440b59c3c5a true vfat 299712512 96813056
/vmfs/volumes # esxcfg-scsidevs -m
naa.6006016038a0280032d8054ed48ae111:1 /vmfs/devices/disks/naa.6006016038a0280032d8054ed48ae111:1 55363c53-3d5bb128-f945-3440b59e9294 0 VMLUN9FC
/vmfs/volumes # esxcli storage core device set --state=off -d naa.6006016038a0280032d8054ed48ae111
I followed the below article to resolve the issue. May be it help to you.
https://vmkfix.blogspot.com/2023/04/vmware-unmount-datastorevolume-from.html