Powercut... Luns on EMC San are blank - unformatted

[mordzy]

Probably not the right forum however i wonder if anyone else has seen this before and has any suggestions other than restore from backup...

I have a EMC Clariion san (FC)

2x esx 5.5 connected via FC

We had a power cut in the early hours. the ups shut down the ESX servers and guests and the san powered down its self (as they do to save the cache to the disks etc)

When power fault was fixed and power restored this morning i booted everything up and logged into my vcenter server. I could see all the paths to the luns from each ESX server but no volumes. When i add a volume it allows me to choose a LUN to format. EEeeeeek. thats literally 7 blank luns...

Now i have backups which is fine but some of the servers failed to backup via Veeam because it was mid power failure. I have only lost a days work on these servers but i would like to try and get back the volumes if possible.

So firstly - has anyone seen this before and if so did you manage to recover.

Has anyone any suggestions to try recover?

how the hell can it happen?

[Cyberfed27]

Did you check Unisphere to make sure the SAN came up cleanly? We have a CX4-120.

I would start with that first check all the communication path's between the SAN, your FC switches and the ESXi hosts.

If all looks good I would do a reboot of each ESXi host and see if maybe it was an issue because both the SAN, switches and ESXi hosts were powering up at the same time.

Sounds scary. Good luck.

[Titanomachia]

are you able to SSH to the server and check the /var/log/kernel.log for "warning" or anything related to LVM? you can also access it via https://hostnameorIP/host

[williambishop]

Try this...

VMware KB: Recovering a lost partition table on a VMFS volume

[williambishop]

Also, you didn't say, but I have to ask, are the luns visible to every host they are supposed to be on? All 7?

[a_p_]

Did you already check whether the partition table on the LUNs still exists and shows a VMFS partition (http://kb.vmware.com/kb/1036609)?

In case the partition is still visible, but discovered as snapshot LUNs (http://kb.vmware.com/kb/1011387) you may either try to persistently (force) mount the LUNs on each of the hosts, or consider to resignature them.

André

[mordzy]

Thanks guys,

I will try these this morning -

Cyberfed27, I have already done the power down, back up in the right order.

I was unable to get into unisphere due to an engineer reloading a laptop that had all the tools on. We have tried a number of different java installs but i think i may need a 32bit version of windows to get it working. Have a support call with EMC this morning so should hopefully get into it to see whats happened.

Cheers

Alex

[vNEX]

Hello Alex,

for partition inspection and recovery follow this fresh article which aplly to your release:

VMware KB: Recreating a missing VMFS datastore partition in VMware vSphere 5.0/5.1/5.5

If you will succeed I would continue with VMFS metadata analyzer:

VMware KB: Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

And if everything fails (hope you will succeed!) you can try data recovery services see:

VMware KB: Data recovery services for data not recoverable by VMware Technical Support

Good Luck!

_________________________________________________________________________________________

If you found this or any other answer helpful, please consider to award points. (use Helpful or Correct buttons)

Regards,

P.