I have a situation where someone had an older vSphere 4.x cluster. The VMs were hosted off of a iSCSI SAN controller to the nodes in the cluster. The volume was 260GB originally when I made it a few years back. Subsequently they have added enough VMs that they needed to expand. They expanded the LUN capacity to 3TB, but was never able to expand the partition. With VMFS3 this is an issue.
Seems if you epand the LUN, EVEN IF you NEVER expand the partition, your systems, if they unmount the volume can never mount it again.
This apparently went on for some time where over time more systems would not "access" the LUN.... and after a power outage, no system could then access the dozens of VMs.
Now I need to find a way to mount that 3TB iSCSI LUN, even if read only, long enough to mount the 260 of VMs off of it to NFS share i have ready.
Goal: map iSCSI LUN to Windows / Linux system. Mount the volume RO. Copy data to NFS export. Destroy and rebuild the data on new iSCI VMFS5 lun.
I can mount the LUN to CENTOS system and parted shows the 3TB LUN but primary partition only 260GB. So I see the data as expected. Now I tried to compile 'vmfs-tools-0.2.5" but get compiler errors. Nothing of any help or note. I tried ubuntu and CENTOS48 also and errors not much help.
Is their another, more simplistic way to get this accomplished?
Thanks,
If ESXi can not see any content of the VMFS-volume - try Linux vmfs-tools - it often sees more.
Try the LiveCD I make - it has latest vmfs-tools buildin.
Carving out flat.vmdks is possible but only is worth a try if:
1. you have good reason to assume that the flat.vmdk was written in one fragment
2. the vmdk was thick provsioned (eager zeroed works best)
3. you can provide exact details on the disk layout of the lost flat.vmdk
Was this by chance a LUN that was expanded to over the 2Tb limit on an ESXi 4.x host?
Hi there
I made a LiveCD with vmfs-tools 0.2.5
You can download it from here:
http://sanbarrow.com/mcs-esxi5-recovery-X-001.iso
Open-iscsi is installed as well - so you should be able to use it without needing to add anything after boot.
Let me hear about your results please
Ulli
I was up late last night and fubmled my way through getting the data off the volume. Below is my notes.
I am not an ubuntu guru.... so my comments may reflect ignorance vs actual issues and has been a few years sense I tried that distro... but my fond memories of that distro no longer exist. The below notes are based on the mount of a local ext3, because my goal of export via NFS was not possible due to various apt-get faults and issues....
I will try the supplied ISO boot image that previous poster provided once it finishes download such that I can let them know how that works. Goal was simple, mount the vmfs volume, export via nfs.. copy via NFS mount to new iSCSI volume.
*******************************
login as: ibm
ibm@172.20.13.28's password:
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Tue Jul 31 01:09:23 EDT 2012
System load: 0.16 Processes: 77
Usage of /: 8.5% of 14.69GB Users logged in: 0
Memory usage: 3% IP address for eth0: 172.20.13.28
Swap usage: 0%
Graph this data and manage this system at https://landscape.canonical.com/
Last login: Mon Jul 30 23:39:47 2012 from 192.168.59.31
su ibm@ubuntu12:~$ su -
Password:
su: Authentication failure
ibm@ubuntu12:~$ su -
Password:
su: Authentication failure
ibm@ubuntu12:~$ vmfs-fuse
VMFS: Unable to read FS information
Unable to open volume.
ibm@ubuntu12:~$
ibm@ubuntu12:~$
ibm@ubuntu12:~$ mount
/dev/mapper/ubuntu12-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
ibm@ubuntu12:~$ sudo iscsiadm -m discovery -t st -p 172.20.31.40
[sudo] password for ibm:
172.20.31.40:3260,1 iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2
ibm@ubuntu12:~$ sudo iscsiadm --m node --targetname iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2 --portal 172.20.31.40 --login
Logging in to [iface: default, target: iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2, portal: 172.20.31.40,3260]
Login to [iface: default, target: iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2, portal: 172.20.31.40,3260]: successful
ibm@ubuntu12:~$ sudo iscsiadm -m session
tcp: [1] 172.20.31.40:3260,1 iqn.2003-10.com.lefthandnetworks:atllhn1:2686:bl490-ss-2
ibm@ubuntu12:~$ sudo fdisk -l
Disk /dev/sda: 17.2 GB, 17179869184 bytes
255 heads, 63 sectors/track, 2088 cylinders, total 33554432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00062394
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 33552383 16525313 5 Extended
/dev/sda5 501760 33552383 16525312 8e Linux LVM
Disk /dev/mapper/ubuntu12-root: 15.8 GB, 15825108992 bytes
255 heads, 63 sectors/track, 1923 cylinders, total 30908416 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/ubuntu12-root doesn't contain a valid partition table
Disk /dev/mapper/ubuntu12-swap_1: 1069 MB, 1069547520 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2088960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/ubuntu12-swap_1 doesn't contain a valid partition table
Disk /dev/sdb: 279.2 GB, 279172874240 bytes
255 heads, 63 sectors/track, 33940 cylinders, total 545259520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000927cf
Device Boot Start End Blocks Id System
/dev/sdb1 128 524281274 262140573+ fb VMware VMFS
ibm@ubuntu12:~$ sudo vmfs-fuse /dev/sdb /media/cdrom/
VMFS VolInfo: invalid magic number 0x00000000
VMFS: Unable to read volume information
Trying to find partitions
ibm@ubuntu12:~$
ibm@ubuntu12:~$ sudo parted /dev/sdb
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: LEFTHAND iSCSIDisk (scsi)
Disk /dev/sdb: 279GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 65.5kB 268GB 268GB primary
ibm@ubuntu12:~$ sudo vmfs-fuse /dev/sdb1 /media/vmfs/
ibm@ubuntu12:~$ sudo ls /media/vmfs -lah
total 4.0K
drwxr-xr-t 36 root root 5.7K Jun 29 11:10 .
drwxr-xr-x 5 root root 4.0K Jul 31 01:30 ..
……
-r-------- 1 root root 4.0M Jun 26 2010 .vh.sf
drwxr-xr-x 2 root root 1.6K Nov 19 2010 w2003r2entx86
drwxr-xr-x 2 root root 1.6K Oct 25 2011 w2008r2ent64bit
ibm@ubuntu12:~$ sudo cp -a /media/vmfs/hpdc01 /media/hp
*********************
which version of vmfs-tools do you use ?
AFAIK only the latest has experimental support for extends
do you see all .*.sf files or just the .vh.sf ?
before you give up ... I am testing a procedure to cut vmdks out of raw filesystems - if you want to serve as a guinea pig let me know 😉
I was wondering if you ever had any luck with a procedure to extract vmdks out of a raw system.
I've got some VM's which were mounted via iSCSI. Something got messed up during a reboot of the NAS, I can still see all of the iSCSI mounts, and partitions. However ESXi does not see anything on them.
I'm hoping to extract the VMDK's if possible. If their is anything I can try, would be happy to give it a spin.
Thanks.
before you give up ... I am testing a procedure to cut vmdks out of raw filesystems - if you want to serve as a guinea pig let me know 😉
If ESXi can not see any content of the VMFS-volume - try Linux vmfs-tools - it often sees more.
Try the LiveCD I make - it has latest vmfs-tools buildin.
Carving out flat.vmdks is possible but only is worth a try if:
1. you have good reason to assume that the flat.vmdk was written in one fragment
2. the vmdk was thick provsioned (eager zeroed works best)
3. you can provide exact details on the disk layout of the lost flat.vmdk
I have been working on other projects the last few months but the volume I did save. I did a DD of the volume over to another controller and just rebuilt the environment from scratch. I would be glad to try this for a few reasons. 1) Because... what good nerd would not want to aspire to the status of guinee pig... 2) Their are two VMs which if we could get back would save us lots of hours rebuilding demos.
Can you point me to the URL for the live DVD. I fully realize this is "beta" and I have copies of copies of copies of this data set so it is not a big deal if it mucks things up.
Hi
here is the link
http://sanbarrow.com/mcs-esxi5-recovery-X-001.iso
Thanks for the ISO, but I am not seeing anything on my local VMFS datastore. I've installed this fresh ESXi 5.0 server to test your ISO, but cannot see anything of the disk.
fdisk -l only list /dev/sda1 and the system is GPT and the Id is ee, no VMFS, not FAT16, no Linux partitions.
It seems that the total area of the physical disk is in that /dev/sda1.
I am also getting warning that GPT is on /dev/sda.
Am I missing something?
BTW: I tried installing ESXi on the same disk or on a USB flash then create a datastore on the disk, but still nothing.
I've tested it on both 5.0 and 5.1. In all my tests, I've created a number of VMs on the disk.
This is a local SATA disk.
I have also tried installing ESXi as a VM inside Workstation, did not see any of the VMFS5 datastores, but saw a VMFS3 datastore that I have added to make sure. I am sure, I downloaded the correct ISO mcs-esxi5-recovery-X-001.iso
Thanks again.
if the vmfs volume is lets say /dev/sdc1 run
vmfs-fuse /dev/sdc1 /mnt
For some odd reason this works on /dev/sdc1 but fails on /dev/sda1 !!
It gives
VMFS VolINFO: Invalid magic number 0x00000000
VMFS: Unable to read volume information
Trying to find partitions
Unable to open device/file "/dev/sda1"
Unable to open filesystem
It seems to me that this is because this is the disk where ESXi is also installed.
BTW: I saw posts for you saying that you have included some recovery tools with the ISO, is there a list of those tools?
Thanks again for this great ISO
Tomorrow, I should be trying it to recover a missing server, whish me luck
Run gparted first - there you will see which partitions are used for VMFS.
/dev/sd*3 = thats typical for a LUN with an ESXi installation and the default VMFS datastore
/dev/sd*1 = thats typical for a VMFS only LUN
I included tools that I often use for remote recovery jobs: ddrescue , scalpel , open-iscsi , testdisk .... some more mount-tools for vmdks.
In case you get "Invalid magic number" with vmfs-fuse for a partition that is listed as VMFS call me - this is serious but can be fixed in lucky conditions
mounting /dev/sda3 worked without issue :smileyblush:
Tomorow is the real issues.
We have an iSCSI LUN that had a number of VMs, the LUN was having issues and suddenly the folder containing one VM is no longer accessible.
If you CD to it, you get an I/O error message after a while. But the data should be still there somewhere/somehow as the free space on the VMFS datastore did not change. Does your ISO has tools that can help retrieve the data??
I was hopping that we will be lucky, and be able to see it using your ISO and copy it off. Maybe we will be, but in the likely case that we will not be that lucky, I may need to use some recovery tools.
To make my life simpler I will be attaching the LUN to a VM directly using RDM (this way, I avoid using iSCSI and other remote tools). Yet, I probably will need to use other tools to recover the data
To recover vmdks with I/O errors I often use the LiveCD with vmfs-fuse and then use ddrescue to copy out the files
This link is not working, can you please send me new link if it's still available...
hello
can i have the link of the iso file because i have a similar problem.
Like that :
Running in esxi 3.0.2 Raid5 (4 disc) and the first disc is not in the member raid5 because it's damage.
One of my virtual machine stop working and i was trying to extend virtual disk to creating another disk bigger than the first one to link the vmdk file on it when i saw that not handle my problem i delete the disk & the virtual disk and the big problem start at this moment because the vmdk file goes .
Now i'm in the situation where i have lost one vmdk file
I have try to recover using diskinternal VMFS recover 1.5 , UFS explorer professional nothing !!!!!
You have did a best work
Thanks & Enjoy the rest of the day
Hello
I think that i found where you can download the iso live CD & hope it's the good one , keep me post if not
link: ftp://recovery:recovery@ftp.mightycare.de/mcs-esxi5-recovery-X-001.iso
Best regard
Ballaflex
thanks for the link but this Fallen
you can upload it again
thanks
Hello,
I have the same issue and i'm unable to connect on your ftp to download the iso file.
Please Help :smileycry: