How to recover VMFS volume after a RAID controller failure

[ithilieth]

Dear community,

I have a very critical issue regarding my ESXi installation and VMFS datastore.

Here is the story:

1. ESXi 5.0 (free) installed on IBM 3650 with adaptec 8k RAID controller and 6xHDD (1 was hotspare).
2. Two RAID-5 arrais - first was 5GB size, where was my ESX installation, second - ~1,8T where was my VMDK files.
3. I have a 1 production virtual machine, which is very important for me to recover.
4. Yesterday my motherboard or RAID controller has failed which forced me to shut down the ESX host.
5. I have moved my HDD to other working server. RAID-5 array rebuilds w/o problems (I had a hotspare disk).
6. After RAID rebuilds, i have tried to boot to my hypervisor, but error message appears: BANK5: not a VMware boot bank. No hypervisor found.
7. Then I have installed ESXi 5.1U1 on USB disk, booted from it and found that the ESX doesnt recognize the datastore.

So, i think that VMFS is corrupted in some way, hopefully the partition table is broken. I have found this thread (https://communities.vmware.com/thread/391750?start=15&tstart=0) but I am afraid to execute the guidance there, since it concern the specific case.

Currently I am in the ESX CLI (via installer) and have the following information:

1. partedUtil getptbl /vmfs/devices/disks/mpx.vmhba1 :C0 :T1 :L0

msdos

242480 255 63 3895451648

Can you assist me how to proceed further so i can normally access/recover my VMFS volume and run my important virtual machine again?

[a_p_]

Has the datastore initially been created with ESXi 5/VMFS5 (i.e. not an upgrade from ESXi 4/VMFS3)?

What's the ouptut of:

partedUtil getUsableSectors "/vmfs/devices/disks/mpx.vmhba1:C0:T1:L0"

André

[ithilieth]

Hello,

Thanks for your quick reply.

Initially i have installed ESXi 5.0 and I am almost 100% sure that VMFS version was 5 (I have installed it more than 6 months ago and of course it wasn't documented)

Output of the partedUtil getUsableSectors "/vmfs/devices/disks/mpx.vmhba1:C0:T1:L0" command is:

1 3895451647

EDIT: It was fresh ESX5 installation so there was not an update of VMFS.

[a_p_]

The reason I asked was to understand whether the original partition type was MBR or GPT and the start sector was 128 or 2048.

With the information you provided, take a look at Recovering a lost partition table with a VMFS datastore | vInfrastructure Blog for details about how to "Recover a lost VMFS partition on a GPT disk". If you have question, pease don't hesitate to ask before you run the command to create the partition.

André

[ithilieth]

Ye, i think I have understand why do you ask.

In the link you have provided, there are two types or partitions - MBR and GPT (which maybe is from gParted).

They say that for legacy MBR disks, fdisk have to be used.

I have tried to execute fdisk -l, but the output was

"Found valid GPT with protective MBR; using GPT"

fdisk: Sorry, cant't handle GPT partitions, use partedUtil

Regarding this output and the arcticles i have read, i assume that i have to use 2048.

So, i assume that I have to execute the following command. Can you please check it as well, so i can be more assertive before executing it:

partedUtil setptbl /vmfs/devices/disks/mpx.vmhba1\:C0\:T1\:L0  gpt "1 2048 3895451647 AA31E02A400F11DB9590000C2911D1B8 0"

Few things i still don't understand:

1. after the gpt, there is a "1". What is it for and should it be 1 in my case?
2. After GUID there is a "0". What is it for and should it be 0 in my case?
3. Will this command destroy my data (VMDK files)?

[a_p_]

1.) The 1 represents the partition number and since it is the only partition on the disk it's 1.

2.) The 0 is ok. The type is important in case you would want ort e.g. boot from the partition.

3.) No risk, no fun. I can't guarantee anything, but creating the partition table should not harm the data.

The command looks ok as far as I can see.

André

[ithilieth]

"Great". I have executed the command and here is the result:

Error: Unable to satisfy all constraints on the partition.

Any ideas?

EDIT:

Shall i perform the command on /vmfs/devices/disks/etc.... or on /dev/disks/etc...?

EDIT2: fdisk -l returns 255 heads, 63 sectors, 242480 cyl, which -1 is 3895441199.

This number differs from output of getUsableSectors. Could it be the issue?

[a_p_]

I don't think it's the size or end sector which causes this error. What I'm concerned about is the output you provided in your initial post (msdos rather than gpt). I think this could be an issue with a partition table corruption, but I'm not sure how to safely fix this and I really don't mess with your data. At this point I'd suggest you contact continuum to see whether he can help.

André

[ithilieth]

Any chance that Recover Data from VMFS, ESX, ESXi, vSphere Disks have rewritten or modifying the VMFS partition?

I have tried to run it, but somehow this application hangs on my PC.

Also I have send a PM tocontinuum, lets hope he would be able to take a look.

[continuum]

Diskinternals VMFS recovery is not that good - also try to read the volume with "vmfs-fuse" from a Linux LiveCD.
If you want to try another commercial product  - try UFS-explorer.

If all that fails I can assist with recovering thick provisioned vmdks - call me via skype if that is necessary

Ulli

[ithilieth]

Hello,

Thanks for your reply. As you probably noticed, i am new to the community.

Can you please send me some link where I can download proper Linux Live CD and eventually some instructions how to use vmfs-fuse?

I have tried with some live CD, downloaded from your website but my server didn't boot from it so i will have to try another liveCD.

Regards