VMware Cloud Community
continuum
Immortal
Immortal

ESXi 4.0 half dead - but still running ...

A running ESXi 4.0 lost access to 2 of its 3 local datastores - the first of them is also used for the ESXi 4.0 boot drive.
Only one of the datastores still appears in datastorebrowser or ls /vmfs/volumes - it still has running VMs

~ # ls /dev/disks/
t10.ATA_____ST31500341AS________________________________________9VS1HH6K
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:1
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:2
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:3
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:4
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:5
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:6
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:7
t10.ATA_____ST31500341AS________________________________________9VS1HH6K:8
t10.ATA_____ST31500341AS________________________________________9VS1LV13
t10.ATA_____ST31500341AS________________________________________9VS1LV13:1
t10.ATA_____ST31500341AS________________________________________9VS1R2T7
t10.ATA_____ST31500341AS________________________________________9VS1R2T7:1
vml.0100000000202020202020202020202020395653314848364b535433313530
vml.0100000000202020202020202020202020395653314848364b535433313530:1
vml.0100000000202020202020202020202020395653314848364b535433313530:2
vml.0100000000202020202020202020202020395653314848364b535433313530:3
vml.0100000000202020202020202020202020395653314848364b535433313530:4
vml.0100000000202020202020202020202020395653314848364b535433313530:5
vml.0100000000202020202020202020202020395653314848364b535433313530:6
vml.0100000000202020202020202020202020395653314848364b535433313530:7
vml.0100000000202020202020202020202020395653314848364b535433313530:8
vml.0100000000202020202020202020202020395653314c563133535433313530
vml.0100000000202020202020202020202020395653314c563133535433313530:1
vml.01000000002020202020202020202020203956533152325437535433313530
vml.01000000002020202020202020202020203956533152325437535433313530:1

~ # fdisk /dev/disks/t10.ATA_____ST31500341AS________________________________________9VS1R2T7

still reacts - the same command for the first two disks hangs forever.

Looks like a reboot of that server would fail ...
Whats the best way to procede in such a case ?


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
10 Replies
JohnADCO
Expert
Expert

Get the VM's running on another host ASAP?     I'd be torn as the the most important to do first, complete back up or move to another host.

Reply
0 Kudos
continuum
Immortal
Immortal

sure - I would do that if it were possible

this is a root-server running at some hosting company so an easy way like vmotion all VMs to a safe other host is out of question

I guess that ESXi 4.0   - like so often - has wiped the partition table
only difference - this time I see the case before a reboot


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
hitman16980
Contributor
Contributor

If the VM's are running, image them... I'm sure they can reboot and not the host, correct?

Reply
0 Kudos
hitman16980
Contributor
Contributor

Can you offload vm's via "Export OVF"?

Reply
0 Kudos
a_p_
Leadership
Leadership

I guess that ESXi 4.0   - like so often - has wiped the partition table

According to the listing you provided, the disks still contain partitions. You may want ot check the partition types using fdisk -lu to see whether the VMFS partitions still show up with the correct type "FB".

André

Reply
0 Kudos
cblomart
Enthusiast
Enthusiast

Connect a external storage and copy your VM there (ie: iSCSI or NFS nas). If no vcenter i would suppose no svmotion possible... even so you'll have downtime with only one ESXi. The rest is not much complicated... get or rebuild your hardware to run the VMs.

Reply
0 Kudos
continuum
Immortal
Immortal

I tried to run fdisk against the first two local disks with
fdisk <disk-name> but only the acessible 3rd one reacts to this command

for the first two disks fdisk commands dont work at all at the moment

I guess I try a reboot tomorrow - I already extracted the mission critical data out of two  - now orphaned snapshots
so the critical stuff is already rescued.

I had good luck - I recovered 2 x  100Mb MS databasefiles from a snapshot and they are completely healthy


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
a_p_
Leadership
Leadership

What if you run fdisk -lu without specifying a disk. This should show the specs of all 3 disks.

André

Reply
0 Kudos
continuum
Immortal
Immortal

I try that tomorrow when the customer is back online - wonder if the ESXi will survive until then Smiley Wink


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

fdisk -lu only displays one disk with VMFS
the first two do noty react at all

Case closed - customer will continue to run this ESXi until he found a better solution than using a root server at Hetzner


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos