Lost all VMs -- Console shows all as Unknown (Inaccessible)

[mundus]

We have an ESX 3.0.2 server running and the VI client was hanging when we tried to look at the console of the running VMs (a mix of Windows and Linux). We put the server in maintenance mode, shutdown everything, and rebooted.

When the server came back up, the entire VMFS partition was empty. All of the folders with our VMs were gone. The partition shows 0% in use. Is there any way for us to get our VMs back?

Thanks.

[IRQ2006]

Very strange .... just to eliminate a possible fault with your ESX host, are you able to attach the VMFS volume to another host to see if you can see your lost VMs, and yes i would try to rescan first assuming it is a SAN storage

[jamieorth]

Is this VMFS partition SAN or local storage? If SAN, have you tried to rescan HBA's and VMFS volumes?

[mundus]

It's local storage, not a SAN...

[IRQ2006]

My last option will be reinstalling ESX on the box, the installation wizard should give you the option to keep existing VMs.

As this is a strange behaviour I would rebuild my ESX anyway and double check storage configuration to eliminate any possible configuration issues.

[jamieorth]

Are you verifying that the volume is empty through the VI datastore browser? I would use the console or scp to have a second look. Maintenance mode and rebooting the host should not affect the vmfs volume or the data contained therein. Has there only been the one reboot or have you rebooted again, and also what do the logs show?

[mundus]

The entire /vmfs/volumes folder is empty. If I go to the Configuration -- Storage section of the ESX client, I can see SCSI Target 1 has what looks like the correct path. Is there any way to recreate the storage volume and have it find the original VMs?

I tried rebooting again, but nothing changed.

Thx.

[opbz]

Well assuming you have not lost data try the following.

If you have an old vm-support or anu record of your old existing partiton disk for that drive it will be a BIG help.

Use fdisk /dev/sdX(where X is the drive where you had the VMFS partition)

n (new partition)

p (primary partition if possible)

select the old values for beginning and ending of the partition

t to change type select fb it will come up as unknown that is normal...

x for expert mode

b to specify a block size select 128

w to write and quit

This will create a partition with all features of vmfs you can delete or recreate it... we are not alterring data on that partition

then rescan hbas if you are lucky your vmfs will appear if it does not try rebooting...If it does not come up you are out of luck you migh have luck with some of the data recovery companies but chances are data is gone

[mundus]

I tried to follow those directions, but i came up against the following issues:

after selecting p it asks for the partition number (1-4)? i'm not sure how to answer

after entering "x" there is no "b" option. I broke out of fdisk and went to expert mode and then "m" for help. There is no "b" option for block.

Here's the output from df -k and fdisk:

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/sda3 20153172 3281620 15847812 18% /

/dev/sda1 101089 30548 65322 32% /boot

none 134116 0 134116 0% /dev/shm

/dev/sda6 521748 57124 438120 12% /var/log

Disk /dev/sda: 146.8 GB, 146814976000 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1 * 1 13 104391 83 Linux

/dev/sda2 14 15076 120993547+ fb Unknown

/dev/sda3 15077 17625 20474842+ 83 Linux

/dev/sda4 17626 17834 1678792+ f Win95 Ext'd (LBA)

/dev/sda5 17626 17755 1044193+ 82 Linux swap

/dev/sda6 17756 17821 530113+ 83 Linux

/dev/sda7 17822 17834 104391 fc Unknown

The volume is local disk and consists of 2 146G drives that are mirrored in hardware.\

To answer the question from the middle of the page, the messages log and vmkernal logs look pretty normal.

It looks like the data is still there. Any ideas?

Thanks in advance....

M-

[opbz]

hi your VMFS partition was your sda2 it is still showug up as FC unknown... THis does not look good. WE can try to ensure its set properly it and hope it comes back up

here is the steps needed.

1: fdisk /dev/sda

2: p to print the file table choose the partition that is Fb unknown from your fdisk it is your partition 2

3: d delete select 2 make sure you select 2!!

4: n new to create a new partition

5: select p for primary

6: 2 for partition number

(press enter twice)

7: x (extended menu) if you do not press this you will not get the b option...

8: b (beginning of partition)

9 :128 (move the beginning to 128 for alignment)

10: r (return to main menu)

11: t (partition type)

12: (type fb for VMFS and press enter)

13: w (write changes to disk and exit)

then try rescaning your hbas with esxcfg-rescan vmhba1 or from the gui try and rescan from confgiuration -> storage controller or try rebooting...

[mundus]

I think we're close here, but the output is different from expected.

Here's the sequence so far:

# fdisk /dev/sda

The number of cylinders for this disk is set to 17849.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 146.8 GB, 146814976000 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1 * 1 13 104391 83 Linux

/dev/sda2 14 15076 120993547+ fb Unknown

/dev/sda3 15077 17625 20474842+ 83 Linux

/dev/sda4 17626 17834 1678792+ f Win95 Ext'd (LBA)

/dev/sda5 17626 17755 1044193+ 82 Linux swap

/dev/sda6 17756 17821 530113+ 83 Linux

/dev/sda7 17822 17834 104391 fc Unknown

Command (m for help): d

Partition number (1-7): 2

Command (m for help): n

Command action

l logical (5 or over)

p primary partition (1-4)

p

Selected partition 2

First cylinder (14-17849, default 14):

Using default value 14

Last cylinder or +size or +sizeM or +sizeK (14-15076, default 15076):

Using default value 15076

Command (m for help): x

Expert command (m for help): b

Partition number (1-7): 2

New beginning of data (208845-242195939, default 208845): 128

Value out of range.

New beginning of data (208845-242195939, default 208845):

#

#

# fdisk /dev/sda

The number of cylinders for this disk is set to 17849.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 146.8 GB, 146814976000 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1 * 1 13 104391 83 Linux

/dev/sda2 14 15076 120993547+ fb Unknown

/dev/sda3 15077 17625 20474842+ 83 Linux

/dev/sda4 17626 17834 1678792+ f Win95 Ext'd (LBA)

/dev/sda5 17626 17755 1044193+ 82 Linux swap

/dev/sda6 17756 17821 530113+ 83 Linux

/dev/sda7 17822 17834 104391 fc Unknown

Command (m for help): d

Partition number (1-7): 2

Command (m for help): n

Command action

l logical (5 or over)

p primary partition (1-4)

p

Selected partition 2

First cylinder (14-17849, default 14):

Using default value 14

Last cylinder or +size or +sizeM or +sizeK (14-15076, default 15076):

Using default value 15076

Command (m for help): x

Expert command (m for help): b

Partition number (1-7): 2

New beginning of data (208845-242195939, default 208845): 128

Value out of range.

New beginning of data (208845-242195939, default 208845):

stuck again.