VMware Cloud Community
ArrowSIVAC
Enthusiast
Enthusiast
Jump to solution

Repair / Recover VMFS Volume

I have a situation where someone had an older vSphere 4.x cluster. The VMs were hosted off of a iSCSI SAN controller to the nodes in the cluster.  The volume was 260GB originally when I made it a few years back.   Subsequently they have added enough VMs that they needed to expand.  They expanded the LUN capacity to 3TB, but was never able to expand the partition.   With VMFS3 this is an issue.

Seems if you epand the LUN, EVEN IF you NEVER expand the partition, your systems, if they unmount the volume can never mount it again.

This apparently went on for some time where over time more systems would not "access" the LUN.... and after a power outage, no system could then access the dozens of VMs.

Now I need to find a way to mount that 3TB iSCSI LUN, even if read only, long enough to mount the 260 of VMs off of it to NFS share i have ready.

Goal: map iSCSI LUN to Windows / Linux system.  Mount the volume RO.  Copy data to NFS export. Destroy and rebuild the data on new iSCI VMFS5 lun.

I can mount the LUN to CENTOS system and parted shows the 3TB LUN but primary partition only 260GB.  So I see the data as expected.   Now I tried to compile 'vmfs-tools-0.2.5"  but get compiler errors.  Nothing of any help or note. I tried ubuntu and CENTOS48 also and errors not much help.

Is their another, more simplistic way to get this accomplished?

Thanks,

1 Solution

Accepted Solutions
continuum
Immortal
Immortal
Jump to solution

If ESXi can not see any content of the VMFS-volume - try Linux vmfs-tools - it often sees more.
Try the LiveCD I make - it has latest vmfs-tools buildin.

Carving out flat.vmdks is possible but only is worth a try if:

1. you have good reason to assume that the flat.vmdk was written in one fragment
2. the vmdk was thick provsioned (eager zeroed works best)
3. you can provide exact details on the disk layout of the lost flat.vmdk


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

View solution in original post

Reply
0 Kudos
37 Replies
Josh26
Virtuoso
Virtuoso
Jump to solution

Was this by chance a LUN that was expanded to over the 2Tb limit on an ESXi 4.x host?

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

Hi there
I made a LiveCD with vmfs-tools 0.2.5

You can download it from here:

http://sanbarrow.com/mcs-esxi5-recovery-X-001.iso

Open-iscsi is installed as well - so you should be able to use it without needing to add anything after boot.

Let me hear about your results please

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

ArrowSIVAC
Enthusiast
Enthusiast
Jump to solution

I was up late last night and fubmled my way through getting the data off the volume.  Below is my notes.

I am not an ubuntu guru.... so my comments may reflect ignorance vs actual issues and has been a few years sense I tried that distro... but my fond memories of that distro no longer exist.  The below notes are based on the mount of a local ext3, because my goal of export via NFS was not possible due to various apt-get faults and issues....

I will try the supplied ISO boot image that previous poster provided once it finishes download such that I can let them know how that works.  Goal was simple, mount the vmfs volume,  export via nfs.. copy via NFS mount to new iSCSI volume.

*******************************

login as: ibm

ibm@172.20.13.28's password:

Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)

* Documentation:  https://help.ubuntu.com/

  System information as of Tue Jul 31 01:09:23 EDT 2012

  System load:  0.16              Processes:           77

  Usage of /:   8.5% of 14.69GB   Users logged in:     0

  Memory usage: 3%                IP address for eth0: 172.20.13.28

  Swap usage:   0%

  Graph this data and manage this system at https://landscape.canonical.com/

Last login: Mon Jul 30 23:39:47 2012 from 192.168.59.31

su ibm@ubuntu12:~$ su -

Password:

su: Authentication failure

ibm@ubuntu12:~$ su -

Password:

su: Authentication failure

ibm@ubuntu12:~$ vmfs-fuse

VMFS: Unable to read FS information

Unable to open volume.

ibm@ubuntu12:~$

ibm@ubuntu12:~$

ibm@ubuntu12:~$ mount

/dev/mapper/ubuntu12-root on / type ext4 (rw,errors=remount-ro)

proc on /proc type proc (rw,noexec,nosuid,nodev)

sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)

none on /sys/fs/fuse/connections type fusectl (rw)

none on /sys/kernel/debug type debugfs (rw)

none on /sys/kernel/security type securityfs (rw)

udev on /dev type devtmpfs (rw,mode=0755)

devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)

tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)

none on /run/shm type tmpfs (rw,nosuid,nodev)

/dev/sda1 on /boot type ext2 (rw)

ibm@ubuntu12:~$ sudo iscsiadm  -m discovery -t st -p 172.20.31.40

[sudo] password for ibm:

172.20.31.40:3260,1 iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2

ibm@ubuntu12:~$ sudo iscsiadm --m node --targetname iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2 --portal 172.20.31.40 --login

Logging in to [iface: default, target: iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2, portal: 172.20.31.40,3260]

Login to [iface: default, target: iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2, portal: 172.20.31.40,3260]: successful

ibm@ubuntu12:~$ sudo iscsiadm -m session

tcp: [1] 172.20.31.40:3260,1 iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2

ibm@ubuntu12:~$ sudo  fdisk -l

Disk /dev/sda: 17.2 GB, 17179869184 bytes

255 heads, 63 sectors/track, 2088 cylinders, total 33554432 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00062394

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048      499711      248832   83  Linux

/dev/sda2          501758    33552383    16525313    5  Extended

/dev/sda5          501760    33552383    16525312   8e  Linux LVM

Disk /dev/mapper/ubuntu12-root: 15.8 GB, 15825108992 bytes

255 heads, 63 sectors/track, 1923 cylinders, total 30908416 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

Disk /dev/mapper/ubuntu12-root doesn't contain a valid partition table

Disk /dev/mapper/ubuntu12-swap_1: 1069 MB, 1069547520 bytes

255 heads, 63 sectors/track, 130 cylinders, total 2088960 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

Disk /dev/mapper/ubuntu12-swap_1 doesn't contain a valid partition table

Disk /dev/sdb: 279.2 GB, 279172874240 bytes

255 heads, 63 sectors/track, 33940 cylinders, total 545259520 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000927cf

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1             128   524281274   262140573+  fb  VMware VMFS

ibm@ubuntu12:~$ sudo vmfs-fuse /dev/sdb /media/cdrom/

VMFS VolInfo: invalid magic number 0x00000000

VMFS: Unable to read volume information

Trying to find partitions

ibm@ubuntu12:~$

ibm@ubuntu12:~$ sudo parted /dev/sdb

GNU Parted 2.3

Using /dev/sdb

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) p

Model: LEFTHAND iSCSIDisk (scsi)

Disk /dev/sdb: 279GB

Sector size (logical/physical): 512B/512B

Partition Table: msdos

Number  Start   End    Size   Type     File system  Flags

1      65.5kB  268GB  268GB  primary

ibm@ubuntu12:~$ sudo vmfs-fuse /dev/sdb1 /media/vmfs/

ibm@ubuntu12:~$ sudo ls /media/vmfs -lah

total 4.0K

drwxr-xr-t 36 root root 5.7K Jun 29 11:10 .

drwxr-xr-x  5 root root 4.0K Jul 31 01:30 ..

……

-r--------  1 root root 4.0M Jun 26  2010 .vh.sf

drwxr-xr-x  2 root root 1.6K Nov 19  2010 w2003r2entx86

drwxr-xr-x  2 root root 1.6K Oct 25  2011 w2008r2ent64bit

ibm@ubuntu12:~$ sudo cp -a /media/vmfs/hpdc01 /media/hp

*********************

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

which version of vmfs-tools do you use ?
AFAIK only the latest has experimental support for extends

do you see all .*.sf files or just the .vh.sf ?

before you give up ... I am testing a procedure to cut vmdks out of raw filesystems - if you want to serve as a guinea pig let me know 😉


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jedsta
Contributor
Contributor
Jump to solution

I was wondering if you ever had any luck with a procedure to extract vmdks out of a raw system.

I've got some VM's which were mounted via iSCSI. Something got messed up during a reboot of the NAS, I can still see all of the iSCSI mounts, and partitions. However ESXi does not see anything on them.

I'm hoping to extract the VMDK's if possible. If their is anything I can try, would be happy to give it a spin.

Thanks.

4. Jul 31, 2012 8:40 AM in response to: ArrowSIVAC

before you give up ... I am testing a procedure to cut vmdks out of raw filesystems - if you want to serve as a guinea pig let me know 😉

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

If ESXi can not see any content of the VMFS-volume - try Linux vmfs-tools - it often sees more.
Try the LiveCD I make - it has latest vmfs-tools buildin.

Carving out flat.vmdks is possible but only is worth a try if:

1. you have good reason to assume that the flat.vmdk was written in one fragment
2. the vmdk was thick provsioned (eager zeroed works best)
3. you can provide exact details on the disk layout of the lost flat.vmdk


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
ArrowSIVAC
Enthusiast
Enthusiast
Jump to solution

I have been working on other projects the last few months but the volume I did save.  I did a DD of the volume over to another controller and just rebuilt the environment from scratch.  I would be glad to try this for a few reasons.   1)  Because... what good nerd would not want to aspire to the status of guinee pig...  2) Their are two VMs which if we could get back would save us lots of hours rebuilding demos.

Can you point me to the URL for the live DVD.  I fully realize this is "beta" and I have copies of copies of copies of this data set so it is not a big deal if it mucks things up.

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

Hi
here is the link
http://sanbarrow.com/mcs-esxi5-recovery-X-001.iso


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
adabbas
Contributor
Contributor
Jump to solution

@continuum

Thanks for the ISO, but I am not seeing anything on my local VMFS datastore. I've installed this fresh ESXi 5.0 server to test your ISO, but cannot see anything of the disk.

fdisk -l only list /dev/sda1 and the system is GPT and the Id is ee, no VMFS, not FAT16, no Linux partitions.

It seems that the total area of the physical disk is in that /dev/sda1.

I am also getting warning that GPT is on /dev/sda.

Am I missing something?

BTW: I tried installing ESXi on the same disk or on a USB flash then create a datastore on the disk, but still nothing.

I've tested it on both 5.0 and 5.1. In all my tests, I've created a number of VMs on the disk.

This is a local SATA disk.

I have also tried installing ESXi as a VM inside Workstation, did not see any of the VMFS5 datastores, but saw a VMFS3 datastore that I have added to make sure. I am sure, I downloaded the correct ISO mcs-esxi5-recovery-X-001.iso

Thanks again.

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

if the vmfs volume is lets say /dev/sdc1 run
vmfs-fuse /dev/sdc1 /mnt


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
adabbas
Contributor
Contributor
Jump to solution

For some odd reason this works on /dev/sdc1 but fails on /dev/sda1 !!

It gives

VMFS VolINFO: Invalid magic number 0x00000000

VMFS: Unable to read volume information

Trying to find partitions

Unable to open device/file "/dev/sda1"

Unable to open filesystem

It seems to me that this is because this is the disk where ESXi is also installed.

BTW: I saw posts for you saying that you have included some recovery tools with the ISO, is there a list of those tools?

Thanks again for this great ISO Smiley Happy

Tomorrow, I should be trying it to recover a missing server, whish me luck Smiley Wink

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

Run gparted first - there you will see which partitions are used for VMFS.

/dev/sd*3 = thats typical for a LUN with an ESXi installation and the default VMFS datastore

/dev/sd*1 = thats typical for a VMFS only LUN

I included tools that I often use for remote recovery jobs: ddrescue , scalpel , open-iscsi , testdisk .... some more mount-tools for vmdks.


In case you get "Invalid magic number" with vmfs-fuse for a partition that is listed as VMFS call me - this is serious but can be fixed in lucky conditions


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

adabbas
Contributor
Contributor
Jump to solution

mounting /dev/sda3 worked without issue :smileyblush:

Tomorow is the real issues.

We have an iSCSI LUN that had a number of VMs, the LUN was having issues and suddenly the folder containing one VM is no longer accessible.

If you CD to it, you get an I/O error message after a while. But the data should be still there somewhere/somehow as the free space on the VMFS datastore did not change. Does your ISO has tools that can help retrieve the data??

I was hopping that we will be lucky, and be able to see it using your ISO and copy it off. Maybe we will be, but in the likely case that we will not be that lucky, I may need to use some recovery tools.

To make my life simpler I will be attaching the LUN to a VM directly using RDM (this way, I avoid using iSCSI and other remote tools). Yet, I probably will need to use other tools to recover the data Smiley Sad

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

To recover vmdks with I/O errors I often use the LiveCD with vmfs-fuse and then use ddrescue to copy out the files


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
GaryGhotra
Contributor
Contributor
Jump to solution

This link is not working, can you please send me new link if it's still available...

ballaflex
Contributor
Contributor
Jump to solution

hello

can i have the link of the iso file because i have a similar problem.

Like that :

Running in esxi 3.0.2 Raid5 (4 disc) and the first disc is not in the member raid5 because it's damage.

One of my virtual machine stop working and i was trying to extend virtual disk to creating another disk bigger than the first one to link the vmdk file on it when i saw that not handle  my problem i delete the disk & the virtual disk and the big problem start at this moment because the vmdk file goes .

Now i'm in the situation where i have lost one vmdk file

I have try to recover using diskinternal VMFS recover 1.5 , UFS explorer professional nothing !!!!!
You have did a best work

Thanks & Enjoy the rest of the day

Reply
0 Kudos
ballaflex
Contributor
Contributor
Jump to solution

Hello

I think that i found where you can download the iso live CD & hope it's the good one , keep me post if not

link: ftp://recovery:recovery@ftp.mightycare.de/mcs-esxi5-recovery-X-001.iso

Best regard

Ballaflex

Reply
0 Kudos
dcdtrasgu
Contributor
Contributor
Jump to solution

thanks for the link but this Fallen

you can upload it again

thanks

Reply
0 Kudos
jojokid
Contributor
Contributor
Jump to solution

Hello,

I have the same issue and i'm unable to connect on your ftp to download the iso file.

Please Help :smileycry:

Reply
0 Kudos