VMware Cloud Community
Groombro
Contributor
Contributor

Recovered ESXi Lost it's VM's

Hi All

I hope you can help because I've got an issue that's a killer.

we run about 8 ESXi 5.5 and 6 boxes out in Offices with limited internet to provide local DC, FNP and Applications they need in country.

these boxes are a standard build of HP DL380 G8p with ESXi on a onboard SD card with a 6 disk raid 5 datastore.

one of these servers hosting 3 VM's developed a issue (fan Failure) and as such an engineer was called to replace the fan. we shut the machine down remotely and the engineer did the replacement when the system came back-up there where no VM's registered on the datastore and it appeared blank.

we checked the ilo and vkernal logs and it seems on the reboot the SD card failed and the system recovered to a recovery partition on the raid datastore and had reset the partitions and returned to it's original install configuration (i.e. the wrong i.p. address and a 60 day eval license).

we couldn't find a way to restore the VM's ourselves so have had to send the disks to data recovery specialists to get the data restored.

the issues are

1. we didn't know that the recovery partition was there and that this was a possibility.

2.was user input required to set the system to restore or was this an automatic recovery?

3. how do we prevent this happening on our other ESXi hosts ?

ideally we would like to know can we get a Raid 1 pair of disks and move the recovery partition to them (or even to a USB stick in each server) to preserve the Data store.

has anyone seen this behavior before?

we have had SD cards fail before and normally just get a Purple screen of Doom until we replace the Card and reconfigure. but those are in our Vcentre and attached to the SAN.

Thanks in Advance

Michael Groombridge

Reply
0 Kudos
1 Reply
continuum
Immortal
Immortal

Hi
a datastore that appears blank with a esxi 5.5 host may still have all the data.
If this ever happens again please keep calm and create a vmfs-meta-data dump before you try anything else.
Please see my instructions here:
Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

> has anyone seen this behavior before?
Yes - I see this quite regularly - ( datastores appear to be blank after an unexpected hard shutdown) especially with RAID5 arrays and excessive use of thin provisioned vmdks.

If you are still trying to recover VMs feel free to call me via skype "sanbarrow"
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos