Cannot remove file from VMFS volume on shared SCSI storage

[om8000]

Cannot remove file fromS volume on shared SCSI storage,

VMware ESX Server 3.5.0 build-153875.

Have tried:

- changing permissions on file = OK

- rm, rm -f = NOT WORKING

- unlink = NOT WORKING

- rename = OK

- mv = NOT WORKING

- cp = OK

- vmkfstools reset LUN seems to work, but still the same

Reanme is working, less/more is working, cp is working,

but moving or deleting is not working. Move can copy file

to another location but is not able to delete an old file.

So I have tried to shut down storage only - no virtual machines

are running on it, rescan and refresh LUN/SAN on both ESX1,ESX2

turning on SCSI storage, rescan again , and I still cannot delete the file.

I would like to delete it without restarting ESX1 and ESX2 servers.

Can anyone please help?

Performed tasks are listed below.

# cd /vmfs/volumes/SCSI-EXT1/BROKEN/PORTAL-backup

#

# pwd

/vmfs/volumes/SCSI-EXT1/BROKEN/PORTAL-backup

# ls

PORTAL.vmdk

# mv PORTAL.vmdk PORTAL.vmdk.txt

# ls

PORTAL.vmdk.txt

# rm PORTAL.vmdk.txt

rm: remove regular file `PORTAL.vmdk.txt'? y

rm: cannot remove `PORTAL.vmdk.txt': No such file or directory

# ls

PORTAL.vmdk.txt

# vdf -h

/vmfs/volumes/4a9bcc84-072e4e5e-af8a-00137252e117

683G 375G 307G 54% /vmfs/volumes/SCSI-EXT1

# esxcfg-vmhbadevs -m

vmhba1:0:0:1 /dev/sdd1 4a9bcc84-072e4e5e-af8a-00137252e117

# vmkfstools -L lunreset /vmfs/devices/disks/vmhba1:0:0:1

# ls

PORTAL.vmdk.txt

# rm PORTAL.vmdk.txt

rm: remove regular file `PORTAL.vmdk.txt'? y

rm: cannot remove `PORTAL.vmdk.txt': No such file or directory

# ls

PORTAL.vmdk.txt

# ll

total 18874368

-rwxrwxrwx 1 root root 19327352832 Jun 22 12:36 PORTAL.vmdk.txt

[FranckRookie]

Hi Om8000,

Maybe not of a great help, but could you run an "ls -alhs" to see if there is any hidden file and to have more information about the file...

Regards

Franck

[Basheer09]

I have observed this issue as well with same symptoms. please get in touch with storage vendor and VMware support to identify if there is any LUN corruption.

[om8000]

No hidden files

[om8000]

no LUN corruption

[FranckRookie]

Next try ... two threads from the knowledge base where you could find information on how to find locks: 1008728 and 10051.

[om8000]

# cd /vmfs/volumes/SCSI-EXT1/BROKEN/PORTAL-backup

# ls

PORTAL.vmdk.txt

# touch PORTAL.vmdk.txt

# stat PORTAL.vmdk.txt

File: `PORTAL.vmdk.txt'

Size: 23622320128 Blocks: 46137344 IO Block: 131072 Regular File

Device: 9h/9d Inode: 2786 Links: 1

Access: (0777/-rwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)

Access: 2010-07-01 15:34:44.000000000 +0200

Modify: 2010-07-01 15:34:44.000000000 +0200

Change: 2010-07-01 15:34:44.000000000 +0200

# vmkfstools -D /vmfs/volumes/SCSI-EXT1/BROKEN/PORTAL-backup/PORTAL.vmdk.txt

# tail /var/log/vmkernel

Jun 23 10:10:25 esx1 vmkernel: 189:02:42:17.379 cpu5:1040)megaraid: reset--2096928222 cmd=0 <c=0 t=0 l=0>

Jun 23 10:14:34 esx1 vmkernel: 189:02:46:26.483 cpu4:1041)WARNING: Res3: 1053: resource 167 (cluster 64) already freed by another host: This may be a non-issue

Jun 23 10:15:13 esx1 vmkernel: 189:02:47:05.971 cpu7:1040)WARNING: Res3: 1053: resource 186 (cluster 51) already freed by another host: This may be a non-issue

Jun 23 10:15:22 esx1 vmkernel: 189:02:47:14.578 cpu4:1040)WARNING: Res3: 1053: resource 186 (cluster 51) already freed by another host: This may be a non-issue

Jul 1 15:35:21 esx1 vmkernel: 197:08:07:11.758 cpu4:1040)FS3: 130:

[om8000]

Any new ideas?

[FranckRookie]

... http://communities.vmware.com/thread/202089 ...