weird outage lost inventory

[goof1427]

Hi-

thank you for your time.  So I have this weird issue... this morning one of my hosts went into maintenance mode and I lost the cluster and the VMs in it.  I finally was able to reconnect the LUN.  The LUN shows that it has data on it... but nothing shows up in the host Inventory.  how do I get the vmdk files off the LUN? Is there a way to restore it?

[a_p_]

If nothing shows up in the inventory but the VM's show up in the datastore browser, you can add the VM's back to the inventory by right clicking the vmx file and select "Add to Inventory". However since you are talking about a cluster, make sure the VM's have not been moved to another host by e.g. HA.

André

[goof1427]

lets go back a step.  I can see the LUN being picked up in the Storage Adapters...however I don't see it in the Storage...so if I try to add the LUN it asks that the drive be formatted, I dont think I want that otherwise I have to recreate like 10VMs and lose all the data in it... I tried to create the LUN as an RDM to a virtual machine but it did not see the drive and wanted to format it as well.

[goof1427]

here is a pic of it..

[a_p_]

From your screen shot it looks like the partition type has changed from VMFS (FB) to SFS (42).

Please read http://kb.vmware.com/kb/1002168 very carefully. Even though the KB article refers to older versions of ESX(i) it should still be valid.

André

[goof1427]

Thanks Andre.  Getting closer... I got the following error. partition table entries are not in disk order... and when I fdisk the SAN LUN fdisk cannot open it.

I looked into this article as well.  kind of the same as you said.

http://www.virtualizationteam.com/virtualization-vmware/vmware-vi3-virtualization-vmware/vmware-esx-...

[goof1427]

I got it back to VMFS....however do I access the data now?  If I add it to storage is saying that 'this configuraiton will delete the current disk layout.  All file systems and data will be permanently lost.'

I used a mixture of the article you posted and this one as well.. KB Article: 1009829

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1009829&sl...

[a_p_]

Can you please post the output of fdisk -lu

Btw. don't worry about "partition table entries are not in disk order". This refers to the disk on which ESXi is installed. Even though the VMFS partition is located at the end of the disk, it is partition 3.

André

[a_p_]

If the command esxcfg-volume -l (lower case "L") returns something, please attach this output too.

André

[goof1427]

attached is the results from fdisk -lu.  and there were no results from esxcfg -l.

[a_p_]

It looks like you fixed the partition type, but did not fix the alignment. The partition still starts at sector 63. Please take a look at the KB I posted earlier for how to align the partition to sector 128.

André

[goof1427]

wow..!  Andre you are the man!  the volume finally shows up!  i give it up to you for your knowledge.  and I especially thank you for your time.  I was able to re-add the VMs back to the host and just got to recreate the pools..no biggie.

I just wish I knew why it went into maintenance mode in the first place...

Also I will be writing up what I did in this thread ..for my sake and maybe it can help someone else.  Thanks again.

[a_p_]

You are absolutely welcome. Good to hear you have the datastore back.

Usually the partition type gets modified by e.g. a Windows host, which has access to the LUN and tries to mount it. I think this is what you should look for to eliminate this to happen again in the future.

Also I will be writing up what I did in this thread ..for my sake and maybe it can help someone else.

That would really be great. Btw. you can mark answers which helped you to solve the issue as correct or helpful. This will also help other users with similar issues.

André

[goof1427]

Here is a summary of what worked for me.  Thanks again Andre.

KB Article:  1002168

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100216...

KB Articel: 1009829

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1009829&sl...

enable ssh for the host

follow this article to enable it via vsphere client

http://www.thomasmaurer.ch/2010/09/activate-ssh-on-esxi-4-1-via-vsphere-client/

-configuration tab

-security profile>properties

-remote tech support (ssh)>Options>start automatically.

I used putty to login.

Steps:

1. fdisk -lu

Check under System to see it whether it is SFS or VMFS

**You can ignore the partition table entries are not in disk order (this is where you ESXi host is)

**Notice the two things we will fix here SFS file system and 63 Sectors.

2. esxcfg-rescan vmhbax ((where X is the # for your hba)

3. esxcfg-scsidevs -c

4. fdisk /vmfs/devices/disks/naa. (this is the Device UID)

5. for the command (m for help): p

6. for the command (m for help): t

7. select partition: 1

8. Hex code: fb  (this changes System type from SFS to VMFS)

9. for command (m for help): w

10. esxcfg-rescan vmhbax (where X is the # for your hba)

Change from 63 sectors to 128:

11.  fdisk /vmfs/devices/disks/naa. (this is the Device UID)

12. for command (m for help): x

13. expert command (m for help): b

14. partition number (1-4): 1

15. new beginning of data (63-1048573951, default 63): 128

16. expert command (m for help): w

17. fdisk -lu (to confirm sector changed from 63 to 128)

18. Rescan all in VSphere for Storage...

You should now see the LUN and can now browse the datastore and import the VMs back in.

Command Actions:

Command Action
a       toggle a bootable flag
b       edit bsd disklabel
c       toggle the dos compatibility flag
d       delete a partition
l       list known partition types
n       add a new partition
o       create a new empty DOS partition table
p       print the partition table
q       quit without saving changes
s       create a new empty Sun disklabel
t       change a partition's system id
u       change display/entry units
v       verify the partition table
w       write table to disk and exit
x       extra functionality (experts only)