Guys,
Had to shut down the host to put more memory in. The new memory is detected fine, but both my datastores disappeared, so I can't bring the VMs back online.
Can't figure out why this happened or how to fix it.
ESXi v5.5. Running off USB (mpx.vmhba32:C0:T0:L0)
Terminal outputs:
# esxcfg-scsidevs -c
Device UID Device Type Console Device Size Multipath PluginDisplay Name
mpx.vmhba32:C0:T0:L0 Direct-Access /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0 1911MB NMP Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
naa.600605b009617b801cfe4186271b2743 Direct-Access /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743 1907200MB NMP Local LSI Disk (naa.600605b009617b801cfe4186271b2743)
naa.600605b009617b801cfe4186271b7c33 Direct-Access /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33 2861056MB NMP Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)
# partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743
gpt
243133 255 63 3905945600
1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
# partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33
gpt
364733 255 63 5859442688
1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
Edit:
Things attempted so far:
VMware Knowledge Base - Recreating a missing VMFS datastore partition in VMware vSphere 5.x and 6.x (2046610)
VMware Knowledge Base - LUN suddenly becomes unavailable to one or more ESXi/ESX hosts and LUN needs to be reset (1000044)
Please see whether the LUN has been detected as a snapshot LUN, and if mounting the datastore manually will work.
see the "Command line" section at https://kb.vmware.com/kb/1011387
André
Nope.
# esxcli storage vmfs snapshot list
#
When looking at existing partitions, only the USB one is listed:
# esxcli system coredump partition list
Name Path Active Configured
---------------------- ------------------------------------------ ------ ----------
mpx.vmhba32:C0:T0:L0:7 /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7 false true
# offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((
0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done
/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
gpt
243 255 63 3913728
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
---------------------
/vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743
gpt
243133 255 63 3905945600
1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
Checking offset found at 2048:
0200000 d00d c001
0200004
1400000 f15e 2fab
1400004
0140001d 64 69 73 6b 32 00 00 00 00 00 00 00 00 00 00 00 |disk2...........|
0140002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
---------------------
/vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33
gpt
364733 255 63 5859442688
1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
Checking offset found at 2048:
0200000 d00d c001
0200004
1400000 f15e 2fab
1400004
0140001d 64 69 73 6b 31 00 00 00 00 00 00 00 00 00 00 00 |disk1...........|
0140002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
---------------------
# fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b2743
***
*** The fdisk command is deprecated: fdisk does not handle GPT partitions. Please use partedUtil
***
Found valid GPT with protective MBR; using GPT
Disk /dev/disks/naa.600605b009617b801cfe4186271b2743: 3905945600 sectors, 3725M
Logical sector size: 512
Disk identifier (GUID): ffa051fd-46a5-4151-a6d3-b62a3a5b9ec0
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3905945566
Number Start (sector) End (sector) Size Code Name
1 2048 3905945566 3724M 0700
# fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b7c33
***
*** The fdisk command is deprecated: fdisk does not handle GPT partitions. Please use partedUtil
***
fdisk: device has more than 2^32 sectors, can't use all of them
Found valid GPT with protective MBR; using GPT
Disk /dev/disks/naa.600605b009617b801cfe4186271b7c33: 4294967295 sectors, 4095M
Logical sector size: 512
Disk identifier (GUID): cd7dd043-9548-4f87-bba7-9b900622d9e4
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5859442654
Number Start (sector) End (sector) Size Code Name
1 2048 5859442654 5587M 0700
# esxcli storage core path list
usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0
UID: usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0
Runtime Name: vmhba32:C0:T0:L0
Device: mpx.vmhba32:C0:T0:L0
Device Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
Adapter: vmhba32
Channel: 0
Target: 0
LUN: 0
Plugin: NMP
State: active
Transport: usb
Adapter Identifier: usb.vmhba32
Target Identifier: usb.0:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 122880
unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33
UID: unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33
Runtime Name: vmhba2:C2:T1:L0
Device: naa.600605b009617b801cfe4186271b7c33
Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)
Adapter: vmhba2
Channel: 2
Target: 1
LUN: 0
Plugin: NMP
State: active
Transport: parallel
Adapter Identifier: unknown.vmhba2
Target Identifier: unknown.2:1
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 131072
unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743
UID: unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743
Runtime Name: vmhba2:C2:T0:L0
Device: naa.600605b009617b801cfe4186271b2743
Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)
Adapter: vmhba2
Channel: 2
Target: 0
LUN: 0
Plugin: NMP
State: active
Transport: parallel
Adapter Identifier: unknown.vmhba2
Target Identifier: unknown.2:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 131072
# esxcli storage core device list
naa.600605b009617b801cfe4186271b7c33
Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)
Has Settable Display Name: true
Size: 2861056
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33
Vendor: LSI
Model: MR9240-4i
Revision: 2.13
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Is SSD: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: unsupported
Other UIDs: vml.0200000000600605b009617b801cfe4186271b7c334d5239323430
Is Local SAS Device: false
Is USB: false
Is Boot USB Device: false
No of outstanding IOs with competing worlds: 32
mpx.vmhba32:C0:T0:L0
Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
Has Settable Display Name: false
Size: 1911
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
Vendor: General
Model: USB Flash Disk
Revision: 1.00
SCSI Level: 2
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: true
Is SSD: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: unsupported
Other UIDs: vml.0000000000766d68626133323a303a30
Is Local SAS Device: false
Is USB: true
Is Boot USB Device: true
No of outstanding IOs with competing worlds: 32
naa.600605b009617b801cfe4186271b2743
Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)
Has Settable Display Name: true
Size: 1907200
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743
Vendor: LSI
Model: MR9240-4i
Revision: 2.13
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Is SSD: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: unsupported
Other UIDs: vml.0200000000600605b009617b801cfe4186271b27434d5239323430
Is Local SAS Device: false
Is USB: false
Is Boot USB Device: false
No of outstanding IOs with competing worlds: 32
Hi
Check if the following VMware KB article helps:VMware Knowledge Base
One of the first things I tried.
vmkernel.log output when rescanning:
2019-04-07T19:11:25.957Z cpu3:38976)VC: 2059: Device rescan time 16 msec (total number of devices 6)
2019-04-07T19:11:25.957Z cpu3:38976)VC: 2062: Filesystem probe time 59 msec (devices probed 6 of 6)
2019-04-07T19:11:25.957Z cpu3:38976)VC: 2064: Refresh open volume time 0 msec
...
2019-04-07T19:15:06.493Z cpu2:32791)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8081b380, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2019-04-07T19:15:06.493Z cpu2:32791)ScsiDeviceIO: 2338: Cmd(0x412e8081b380) 0x1a, CmdSN 0x418 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
...
2019-04-07T19:16:27.080Z cpu6:34890)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b7c33" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x6 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0x7 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2019-04-07T19:16:27.081Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0x8 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2019-04-07T19:16:27.083Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b2743" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x9 from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0xa from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0xb from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Do you can sharing vmkernel.log and hosts.log with the date next to reboot?
From the first one of the two KB articles you mentioned in your initial post, did you run vmkfstools -V (step 5), and then check the vmkernel.log for related entries?
André