VMware Cloud Community
vmhare
Contributor
Contributor
Jump to solution

Missing Datastore after Reboot (EXSi 6.7)

After restarting the ESXi host that housed my VCSA the datastore that contained the vCenter VM was missing from the web UI. The disk still shows up but rescanning and refreshing does not mount the datastore.

The datastore is backed by a VMFS partition on the same disk as the ESXi installation. Was not able to find any snapshots of the VMFS. Running:

ls -lisa /vmfs/devices/disks/

I can see both the disk and a mapping for the partition.

partedUtil getptbl /dev/disks/naa.5000c5003b6c54af

Shows no issues with the partition table.

Only clue I can find is in the vmkernel.log.

2020-09-21T17:12:18.876Z cpu3:2097198)NMP: nmp_ThrottleLogForDevice:3802: Cmd 0x28 (0x459b7fbd93c0, 2476806) to dev "naa.5000c5003b6c54af" on path "vmhba1:C0:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0. Act:NONE

Somewhat of a VMWare newbie, any suggestions for where I should go from here?

1 Solution

Accepted Solutions
ashilkrishnan
VMware Employee
VMware Employee
Jump to solution

Hi ,

Please run these commands to check if the datastore is listed:

esxcfg-scsidevs -m

esxcli <conn_options> storage filesystem list


Then you can run following command to mount datastore manually:  esxcfg-volume –M  DatastoreUUID

Hope that helps

View solution in original post

8 Replies
ashilkrishnan
VMware Employee
VMware Employee
Jump to solution

Hi ,

Please run these commands to check if the datastore is listed:

esxcfg-scsidevs -m

esxcli <conn_options> storage filesystem list


Then you can run following command to mount datastore manually:  esxcfg-volume –M  DatastoreUUID

Hope that helps

vmhare
Contributor
Contributor
Jump to solution

Thank you for the suggestion ashilkrishnan​! Running the commands you suggested yielded the following result:

Screenshot 2020-09-21 21.25.01.png

I am correct in assuming that all those errors indicate some sort of corruption on the volume?

Reply
0 Kudos
ashilkrishnan
VMware Employee
VMware Employee
Jump to solution

Yes, it does seem like some issue with that volume.

Please check the vmkernel, hostd and vobd logs (/var/run/log) to see if it provides more information about this volume.

vmhare
Contributor
Contributor
Jump to solution

Could not find anything in the hostd or vobd logs. The vmkernel log had many instances of this message though:

Screenshot 2020-09-21 22.06.18.png

Reply
0 Kudos
ashilkrishnan
VMware Employee
VMware Employee
Jump to solution

It does report I/O errors. You can try running lunreset and try to mount again. This will release any existing SCSI locks.

vmkfstools -L lunreset /vmfs/devices/disks/NAA_ID

khiregange
VMware Employee
VMware Employee
Jump to solution

The SCSI sense code returns "HARDWARE ERROR" for that particular LUN  >

https://www.virten.net/vmware/esxi-scsi-sense-code-decoder/?host=&device=2&plugin=&sensekey=4&asc=32...

Looks like there are on disks error which has corrupted the VMFS meta data regions

Take the snapshot of the LUN first and then map the snapshot LUN to the esxi host and run the below command in order - >

voma -m vmfs -f check -d /dev/disks/naa.5000c5003b6c54af

voma -m vmfs -f fix -d /dev/disks/naa.5000c5003b6c54af

if you are running the esxi 6.5 u3 and above you may try advfix

voma -m vmfs -f advfix -d /dev/disks/naa.5000c5003b6c54af -p /temp

If you are able to mount the snapshot LUN by either resignature / force mount  -

you can they can svmotion of the VMs off the snapshot datastore and decommsion the base and snapshot LUN

CAUTION - > if there are severe corruption on the LUN and if the advfix doesnt fix the corruption then you have to restore the data from the backup

vmhare
Contributor
Contributor
Jump to solution

Looks like the datastore is hosed, all the voma commands result in an IO error:

Running VMFS Checker version 2.1 in check mode

Initializing LVM metadata, Basic Checks will be done

Initializing LVM metadata..-

LVM magic not found at expected Offset,

It might take long time to search in rest of the disk.

VMware ESX Question:

Do you want to continue (Y/N)?

0) _Yes

1) _No

Select a number from 0-1: 0

         ERROR: IO failed: Input/output error

         ERROR: Failed to Initialize LVM Metadata

   VOMA failed to check device : IO error

Total Errors Found:           0

   Kindly Consult VMware Support for further assistance

Would have been great to get the VCSA back to at least figure out what happened, oh well. Here's to hoping the backups are viable.

Thanks again for all your help khiregangeashilkrishnan

Reply
0 Kudos
baduba
Contributor
Contributor
Jump to solution

for find node down

- esxcfg-colume -l

Mounting :

- esxcfg-colume -M 54xxxxdb-ccxd3dss-sxxdsas

mount points

- df

Reply
0 Kudos