VMware Cloud Community
dave76
Contributor
Contributor

Unable to access files on vmfs

Hi all - long time reader first time poster Smiley Happy

After reading countless articles and trying to find a fix, thought I'd try posting and see if anyone can help.

I had my ESXI 3.5 server crash the other day, now on one of the RAID Arrays (SATA running on Adaptec card) I'm unable to use any of the VMDK files - I can see them all there in the datastore and command line, but can't do anything with them - have the 'invalid argument' error which seems quite common.

I can't cp, mv, rm etc the files.

Now from all my reading - doing the vmkfstools -D of the file gives me the system id of 00042388de7b

Mar 27 13:35:34 vmkernel: 0:00:45:51.235 cpu2:11808)FS3: 130: <START Phone system-flat.vmdk>
Mar 27 13:35:34 vmkernel: 0:00:45:51.235 cpu2:11808)Lock [type 10c00001 offset 15677440 v 167, hb offset 3368960
Mar 27 13:35:34 vmkernel: gen 6, mode 1, owner 4d81f4fa-dcf1efec-32c3-00042388de7b mtime 3548]
Mar 27 13:35:34 vmkernel: 0:00:45:51.235 cpu2:11808)Addr <4, 17, 111>, gen 112, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
Mar 27 13:35:34 vmkernel: 0:00:45:51.235 cpu2:11808)len 42949672960, nb 40960 tbz 33660, zla 3, bs 1048576
Mar 27 13:35:34 vmkernel: 0:00:45:51.236 cpu2:11808)FS3: 132: <END Phone system-flat.vmdk>

esxcfg-info | grep -i 'system uuid' | awk -F '-' '{print $NF}' gives me

00042388de7b
1f212b599d8a

but doing a ps in any form I can't find the VM mentioned anywhere ?

Here is an ls -la of one of the VMs

/vmfs/volumes/4a35a7a1-d9f6b1a6-0d17-00042388de7b/Phone system # ls -la
drwxrwxrwx    1 root     root         1400 Mar 25 14:24 .
drwxr-xr-t    1 root     root         1820 Mar 25 14:36 ..
-rw-------    1 root     root    1073741824 Mar 21 03:26 Phone system-0748290f.vswp
-rw-------    1 root     root    42949672960 Mar 21 04:17 Phone system-flat.vmdk
-rw-------    1 root     root         8684 Mar 17 12:47 Phone system.nvram
-rwxrwxrwx    1 root     root          402 Mar 25 14:49 Phone system.vmdk
-rwxrwxrwx    1 root     root            0 Mar 25 14:49 Phone system.vmsd
-rwxrwxrwx    1 root     root         2295 Mar 25 14:49 Phone system.vmx
-rwxrwxrwx    1 root     root          267 Mar 25 14:49 Phone system.vmxf
-rw-r--r--    1 root     root        20644 Mar 17 12:57 vmware-1.log

Trying to run any command results in the following

/vmfs/volumes/4a35a7a1-d9f6b1a6-0d17-00042388de7b/Phone system # touch *
touch: Phone system-0748290f.vswp: Invalid argument
touch: Phone system-flat.vmdk: Invalid argument
touch: Phone system.nvram: Invalid argument
touch: vmware-1.log: Invalid argument

This is a RAID array only on the local machine and not shared on any other machines.

I've removed the machines from inventory and tried re creating, pointing to the vmdks but still no joy.

Any help to get the VMs back workable would be greatly appreciated.

Dave

Reply
0 Kudos
6 Replies
DSTAVERT
Immortal
Immortal

I wouldn't work on the disks directly. Try a Linux Live CD, Ubuntu works well, and add VMFS-tools http://glandium.org/projects/vmfs-tools/ and try copying off the VMDKs to another location.

-- David -- VMware Communities Moderator
dave76
Contributor
Contributor

Thanks for the reply, I'll give it a go and post an update....

Reply
0 Kudos
idle-jam
Immortal
Immortal

another option from the above is that if you have valid support i would suggest your getting help from there.

Reply
0 Kudos
wila
Immortal
Immortal

Hi,

The article I wrote here might help:

http://planetvm.net/blog/?p=1592

Although it seems you are quite able to figure it out yourself from your text above.

If vmfs-tools doesn't help to copy the data, I strongly suggest to clone the volume to a spare disk, before trying to do anything potentially destructive to the data.

--
Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
dave76
Contributor
Contributor

Thanks for the tips

I'll be playing with the box on the weekend

Re the live cd option and vmfs tools I came across this

http://clonezilla.org/

which is a live cd with vmfs tools preinstalled which will hopefully do the trick....

Will let you know how I go...

Reply
0 Kudos
dave76
Contributor
Contributor

It took a bit longer than expected to be able to work on this again, but I'm happy to report SUCCESS :smileygrin:

Here is what I did in case anyone else experiences similar issues.

  1. Stopped and copied the working VMs to another machine so that could continue operation whilst I played
  2. Booted the server using clonezilla live cd
  3. Plugged in and mounted a USB drive
  4. Mounted the volume with problems using FMFS-tools
  5. Copied the faulty VM directories to the USB drive
  6. Removed the USB drive and plugged into a standard linux machine (yay no native USB support on esxi)
  7. Copied the faulty VM directories to another esxi server datastore and recreated the VMs in vSphere using the old vmdk files as the hard drives

Next thing I'm going to do is recreate the faulty datastore and copy the VMs back to the original machine and see if they work there as well.

Big thanks for the tips guys Smiley Happy and hope this helps someone else later.

Dave

Reply
0 Kudos