Can't see datastore after reinstall of the Host 6.0

[UKGCearon]

Hello Everyone;

I have this seriously weird issue. A sister company of ours is running a 6.0 host but yesterday when 1 of the support tried to access the host he couldnt connect with the root. They assumed it was lockout and decided to log in from the console to no avail. as they were discussing what to do VMs started to crash and they have decided the fastest thing to do was to reinstall the host and re-mount the VMs. They have 1 RAID 0 for the host and a RAID 5 for the VMs. Even thou they have successfully reinstalled 6.0 for some reason the system doesnt see the datastore but does see the disk as VMFS. We have run partedUtil to see if there was any corruption and there was none. GPT has no Errors but for some reason we cant seem to get the system to see the datastore on the disk. Since they, for some reason, dont have a back-up since december and 1 of the VMs have some pretty important data they need it.

Whatever I tried didnt helped and I got invested cause of weirdness. If any1 had any similar issues that can shed some light I appreciate it.

Thank You,

[a_p_]

Welcome to the Community,

what you may do is to check whether the VMFS datatstore has been detected as a Snapshot LUN, and can be mounted manually..

You'll find the commands in the "Command line" section of https://kb.vmware.com/s/article/1011387​

André

[UKGCearon]

Thank You a.p. for your response but the issue is deep. If you look at the SSs I have attached, the problem is weird cause although Client sees the formatting as GPT and the partition as VMFS.

I have verified this with voma as well ( voma.jpg). However since I cant find UUID I cant do anything and this problem although not mine is killing me as I have never seen anything like this.

[a_p_]

Just a guess.

The installed ESXi build is a 3 years old one. What I could think of is that maybe the datastore has been formatted using a newer build, which now causes the LVM mismatch.

As mentioned it's only a though, but it shouldn't take long to update/patch the host to find out whether this helps.

So far I've only seen LVM major version mismatch errors when trying to access datastores with a previous ESXi version.

Does grep -i mismatch /var/log/vmkernel.log show any LVM errors?

André

[UKGCearon]

It returns nothing. I shall try updating the host but I have been told this was always 6.0. But I shall update and try.

[a_p_]

Please run the following command, and post the output.

hexdump -C /dev/disks/mpx.vmhba1\:C0\:T0\:L0:3 -n 16636 -s 0x100200 | grep -C 2 '[a-zA-Z0-9]\{4\}-[a-zA-Z0-9]'

Note: You'll have to replace the partition "mpx.vmhba1\:C0\:T0\:L0:3" with yours, i.e. "naa.6d4ae5.....feb4:1"

André

[UKGCearon]

Idk if this was expected but it returned nothing as well.

[a_p_]

I actually expected an output which includes the datastore's UUID, which shows up for mounted filesystems using:

esxcli storage filesystem list

There should be at least something in the vmkernel.log regarding the issue.

André

[UKGCearon]

I ran the command and although not 100% sure am thinking all of these are UUIDs of ds1 which is the disk that host is installed.

And I seriously appreciate all of your help here.

[a_p_]

In the output "ds1" shows up as "VMFS-unknown version", which most likely means that the datastore has been created using a newer ESXi/VMFS version than what ESXi 6.0 supports.

This bringa us back to the recommendation for patching, or upgrading the host. In case this is a standalone host, which supports ESXi 6.5, or newer, I'd consider upgrading to the newest supported version/build. In case it's managed by vCenter Server, or requires support for any 4rd-party applications, aou'll check the compatibility before upgrading the host.

André

[UKGCearon]

I have just updated the system to 6.5.0 and the disk that I want to mount still showing no signs of wanting to be mounted.  Idk why ds1 is showing unknown version as from the client it shows it is VMFS 5.8.1. I shall now patch this to 6.5 u1 and see if that will make any difference

[a_p_]

Version 5.81 has been introduced with ESXi 6.5, so this should actually work. However, updating to v6.5 U1 is worth a try.

If this still doesn't help, and voma still shows an LVM mismatch, then try to contact VMTN user continuum​. He's a pro when it comes to data recovery, and may be able to help.

André

[UKGCearon]

no dice with this as well. However if 5.81 was developed with 6.5 how come that worked and not the other disk. Thou I shall take your advice and try to contact continuum. Thank you for your help

[continuum]

hello
please read Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay
create a dump like that and call me via skype
Ulli

[UKGCearon]

Downloading the dump right now, but it will take a while as the server is in a very remote location.

[continuum]

Please check that dump file - it is empty !!!
This means either you created the dump for the wrong object - or this datastore has been wiped with zeroes. ( not very likely ??? )