After restarting the ESXi host that housed my VCSA the datastore that contained the vCenter VM was missing from the web UI. The disk still shows up but rescanning and refreshing does not mount the datastore.
The datastore is backed by a VMFS partition on the same disk as the ESXi installation. Was not able to find any snapshots of the VMFS. Running:
ls -lisa /vmfs/devices/disks/
I can see both the disk and a mapping for the partition.
partedUtil getptbl /dev/disks/naa.5000c5003b6c54af
Shows no issues with the partition table.
Only clue I can find is in the vmkernel.log.
2020-09-21T17:12:18.876Z cpu3:2097198)NMP: nmp_ThrottleLogForDevice:3802: Cmd 0x28 (0x459b7fbd93c0, 2476806) to dev "naa.5000c5003b6c54af" on path "vmhba1:C0:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0. Act:NONE
Somewhat of a VMWare newbie, any suggestions for where I should go from here?
Hi ,
Please run these commands to check if the datastore is listed:
esxcfg-scsidevs -m
esxcli <conn_options> storage filesystem list
Then you can run following command to mount datastore manually: esxcfg-volume –M DatastoreUUID
Hope that helps
Hi ,
Please run these commands to check if the datastore is listed:
esxcfg-scsidevs -m
esxcli <conn_options> storage filesystem list
Then you can run following command to mount datastore manually: esxcfg-volume –M DatastoreUUID
Hope that helps
Thank you for the suggestion ashilkrishnan! Running the commands you suggested yielded the following result:
I am correct in assuming that all those errors indicate some sort of corruption on the volume?
Yes, it does seem like some issue with that volume.
Please check the vmkernel, hostd and vobd logs (/var/run/log) to see if it provides more information about this volume.
Could not find anything in the hostd or vobd logs. The vmkernel log had many instances of this message though:
It does report I/O errors. You can try running lunreset and try to mount again. This will release any existing SCSI locks.
vmkfstools -L lunreset /vmfs/devices/disks/NAA_ID
The SCSI sense code returns "HARDWARE ERROR" for that particular LUN >
Looks like there are on disks error which has corrupted the VMFS meta data regions
Take the snapshot of the LUN first and then map the snapshot LUN to the esxi host and run the below command in order - >
voma -m vmfs -f check -d /dev/disks/naa.5000c5003b6c54af
voma -m vmfs -f fix -d /dev/disks/naa.5000c5003b6c54af
if you are running the esxi 6.5 u3 and above you may try advfix
voma -m vmfs -f advfix -d /dev/disks/naa.5000c5003b6c54af -p /temp
If you are able to mount the snapshot LUN by either resignature / force mount -
you can they can svmotion of the VMs off the snapshot datastore and decommsion the base and snapshot LUN
CAUTION - > if there are severe corruption on the LUN and if the advfix doesnt fix the corruption then you have to restore the data from the backup
Looks like the datastore is hosed, all the voma commands result in an IO error:
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done
Initializing LVM metadata..-
LVM magic not found at expected Offset,
It might take long time to search in rest of the disk.
VMware ESX Question:
Do you want to continue (Y/N)?
0) _Yes
1) _No
Select a number from 0-1: 0
ERROR: IO failed: Input/output error
ERROR: Failed to Initialize LVM Metadata
VOMA failed to check device : IO error
Total Errors Found: 0
Kindly Consult VMware Support for further assistance
Would have been great to get the VCSA back to at least figure out what happened, oh well. Here's to hoping the backups are viable.
Thanks again for all your help khiregange ashilkrishnan
for find node down
- esxcfg-colume -l
Mounting :
- esxcfg-colume -M 54xxxxdb-ccxd3dss-sxxdsas
mount points
- df