Can't delete vmxf, hlog and Nvram

[ygao]

Hi, I have a problem. One of our guys reboot the shared storage without proper shutting down the VMs. All the VMs freaked out, they are not powering on. After I removed the VM from inventory, and try to add it back in with .vmx file, the vm come back saying unknow (invalid).

I found the following post and it says delete the vmxf, hlog and Nvram.

But when I'm trying to delete the file, it says

rm: remove regular file `SCF-ENTFV-001.nvram'? y

rm: cannot remove `SCF-ENTFV-001.nvram': Permission denied

I tried the chattr command as well, it didn't work.

Any ideas? thanks.

[Chamon]

Does the storage show up as normal? if you ls -la in the vm directory what do you get can you post it here?

[ygao]

yeah, the storage show up and the filer was rebooted.

total 42025700

drwxr-xr-x 1 root root 4096 May 4 17:37 .

drwxrwxrwx 1 root root 4096 Jun 4 21:33 ..

-rwxrwxrwx 1 root root 84 Jun 4 20:21 .lck-b0b20c0000000000

-rw------- 1 root root 2013265920 Feb 27 01:19 SCF-ENTFV-001-3db9cf44 .vswp

-rw------- 1 root root 42949672960 Jun 4 20:03 SCF-ENTFV-001-flat.vm dk

-rw------- 1 root root 8684 May 4 17:37 SCF-ENTFV-001.nvram

-rw------- 1 root root 406 May 4 17:41 SCF-ENTFV-001.vmdk

-rw------- 1 root root 0 Feb 5 12:44 SCF-ENTFV-001.vmsd

-rwxr-xr-x 1 root root 2219 May 4 17:37 SCF-ENTFV-001.vmx

-rw------- 1 root root 268 Feb 5 12:44 SCF-ENTFV-001.vmxf

-rw-rr 1 root root 29034 Feb 5 11:24 vmware-10.log

-rw-rr 1 root root 29830 Feb 5 11:24 vmware-11.log

-rw-rr 1 root root 30160 Feb 5 11:24 vmware-12.log

-rw-rr 1 root root 32174 Feb 26 17:54 vmware-13.log

-rw-rr 1 root root 25236 Apr 2 14:17 vmware-14.log

-rw-rr 1 root root 29039 Feb 5 11:24 vmware-9.log

-rw-rr 1 root root 24168 May 4 17:41 vmware.log

[java_cat33]

If you can't power on any of your VM's - reboot your ESX hosts, my guess is that you'll have some file locks in place on your ESX hosts due to the SAN going down. It'll be quicker to reboot them then to find each lock.

[Chamon]

if you run

ps aux |grep <vm-name> does it return the PID for the VM?

If so you may need to kill it first

kill -9 <PID returned from the above command>

and then try to add it back

[ygao]

I already removed the NFS share connection, rebooted the ESX host, rebooted the filer and re-add the NFS share, sitll no luck

[Chamon]

If they can't reboot the host what abour masking the LUN and rescanning the host and then presenting the LUN back and rescanning again?

[ygao]

Nope, it doesn't return the PID for the VM

anyone know what this file is?

-rwxrwxrwx 1 root root 84 Jun 4 20:49 .lck-66748a0000000000

looks like a lock file

[ygao]

This is NFS on netapp, so there is no masking? I thought masking is only on FC filers.

[Chamon]

After ou remove access to the NFS share can you verify on the ESX side that it knows that the share is gone. We had a problem where the host still had a lock on the share even though it was no longer there and would not let us do anything with it

[Chamon]

Just meant to remove the ESX access to the share

[ygao]

Yup, the share is called NFS_Share, the first line is before I remove the share, the 2nd line is after I remove the NFS_Share through VI

# ls

48f974ea-dfcc81d8-b977-002264c34ed0 49879988-c6af7381-8fc2-002264c34ed0 77cdc99c-d2cd307a NFS_Share scf-esxvm-003:storage1 scf-esxvm-003:storage2

# ls

48f974ea-dfcc81d8-b977-002264c34ed0 49879988-c6af7381-8fc2-002264c34ed0 scf-esxvm-003:storage1 scf-esxvm-003:storage2

[Chamon]

How many hosts do you have that have access to this share? Could another host have this lock? If you need to delete these files can you do a chmod 755 <file> and then delete it? Are you logged in as root? Or su - to it?

[ygao]

I have two hosts, both hosts were rebooted and all teh vms were removed from inventory.

When I do 755 on the file this is what I get, permisison denied.

# chmod 755 scf-entps-001.vmxf

chmod: changing permissions of `scf-entps-001.vmxf': Operation not permitted

#

I'm log in as root

[Chamon]

Do you have any recent back ups? The files may have been curupted. If

you have support you may want to log a call with vmware they can take

some time to get back with you.

[Chamon]

you can also clone the vmdk files and add them to a new Vm.

vmkfstools -i /path/to/vmdk /new/location/to/vmdk

and then add the cloned vmdk files to your new empty VM. How many VMs were on the share?