VMware Cloud Community
UnS3eN
Enthusiast
Enthusiast

Datastores disappeared after a reboot

Guys,

Had to shut down the host to put more memory in. The new memory is detected fine, but both my datastores disappeared, so I can't bring the VMs back online.

Can't figure out why this happened or how to fix it.


ESXi v5.5. Running off USB (mpx.vmhba32:C0:T0:L0)

Terminal outputs:

# esxcfg-scsidevs -c

Device UID                            Device Type      Console Device                                            Size      Multipath PluginDisplay Name

mpx.vmhba32:C0:T0:L0                  Direct-Access    /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0                  1911MB    NMP     Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

naa.600605b009617b801cfe4186271b2743  Direct-Access    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743  1907200MB NMP     Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

naa.600605b009617b801cfe4186271b7c33  Direct-Access    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33  2861056MB NMP     Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

# partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

gpt

243133 255 63 3905945600

1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

# partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

gpt

364733 255 63 5859442688

1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Edit:
Things attempted so far:
VMware Knowledge Base - Recreating a missing VMFS datastore partition in VMware vSphere 5.x and 6.x (2046610)

VMware Knowledge Base - LUN suddenly becomes unavailable to one or more ESXi/ESX hosts and LUN needs to be reset (1000044)

Reply
0 Kudos
7 Replies
a_p_
Leadership
Leadership

Please see whether the LUN has been detected as a snapshot LUN, and if mounting the datastore manually will work.

see the "Command line" section at https://kb.vmware.com/kb/1011387

André

Reply
0 Kudos
UnS3eN
Enthusiast
Enthusiast

Nope.

# esxcli storage vmfs snapshot list

#

When looking at existing partitions, only the USB one is listed:

# esxcli system coredump partition list

Name                    Path                                        Active  Configured

----------------------  ------------------------------------------  ------  ----------

mpx.vmhba32:C0:T0:L0:7  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7   false        true

Reply
0 Kudos
UnS3eN
Enthusiast
Enthusiast

# offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((

0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done

/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

gpt

243 255 63 3913728

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

---------------------

/vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

gpt

243133 255 63 3905945600

1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Checking offset found at 2048:

0200000 d00d c001

0200004

1400000 f15e 2fab

1400004

0140001d  64 69 73 6b 32 00 00 00  00 00 00 00 00 00 00 00  |disk2...........|

0140002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

---------------------

/vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

gpt

364733 255 63 5859442688

1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Checking offset found at 2048:

0200000 d00d c001

0200004

1400000 f15e 2fab

1400004

0140001d  64 69 73 6b 31 00 00 00  00 00 00 00 00 00 00 00  |disk1...........|

0140002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

---------------------

# fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b2743

***

*** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

***

Found valid GPT with protective MBR; using GPT

Disk /dev/disks/naa.600605b009617b801cfe4186271b2743: 3905945600 sectors, 3725M

Logical sector size: 512

Disk identifier (GUID): ffa051fd-46a5-4151-a6d3-b62a3a5b9ec0

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 3905945566

Number  Start (sector)    End (sector)  Size       Code  Name

   1            2048      3905945566       3724M   0700

# fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b7c33

***

*** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

***

fdisk: device has more than 2^32 sectors, can't use all of them

Found valid GPT with protective MBR; using GPT

Disk /dev/disks/naa.600605b009617b801cfe4186271b7c33: 4294967295 sectors, 4095M

Logical sector size: 512

Disk identifier (GUID): cd7dd043-9548-4f87-bba7-9b900622d9e4

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 5859442654

Number  Start (sector)    End (sector)  Size       Code  Name

   1            2048      5859442654       5587M   0700

# esxcli storage core path list

usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0

   UID: usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0

   Runtime Name: vmhba32:C0:T0:L0

   Device: mpx.vmhba32:C0:T0:L0

   Device Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Adapter: vmhba32

   Channel: 0

   Target: 0

   LUN: 0

   Plugin: NMP

   State: active

   Transport: usb

   Adapter Identifier: usb.vmhba32

   Target Identifier: usb.0:0

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 122880

unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33

   UID: unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33

   Runtime Name: vmhba2:C2:T1:L0

   Device: naa.600605b009617b801cfe4186271b7c33

   Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

   Adapter: vmhba2

   Channel: 2

   Target: 1

   LUN: 0

   Plugin: NMP

   State: active

   Transport: parallel

   Adapter Identifier: unknown.vmhba2

   Target Identifier: unknown.2:1

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 131072

unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743

   UID: unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743

   Runtime Name: vmhba2:C2:T0:L0

   Device: naa.600605b009617b801cfe4186271b2743

   Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

   Adapter: vmhba2

   Channel: 2

   Target: 0

   LUN: 0

   Plugin: NMP

   State: active

   Transport: parallel

   Adapter Identifier: unknown.vmhba2

   Target Identifier: unknown.2:0

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 131072

# esxcli storage core device list

naa.600605b009617b801cfe4186271b7c33

   Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

   Has Settable Display Name: true

   Size: 2861056

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

   Vendor: LSI

   Model: MR9240-4i

   Revision: 2.13

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0200000000600605b009617b801cfe4186271b7c334d5239323430

   Is Local SAS Device: false

   Is USB: false

   Is Boot USB Device: false

   No of outstanding IOs with competing worlds: 32

mpx.vmhba32:C0:T0:L0

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Has Settable Display Name: false

   Size: 1911

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Vendor: General

   Model: USB Flash Disk

   Revision: 1.00

   SCSI Level: 2

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: true

   Is SSD: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0000000000766d68626133323a303a30

   Is Local SAS Device: false

   Is USB: true

   Is Boot USB Device: true

   No of outstanding IOs with competing worlds: 32

naa.600605b009617b801cfe4186271b2743

   Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

   Has Settable Display Name: true

   Size: 1907200

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

   Vendor: LSI

   Model: MR9240-4i

   Revision: 2.13

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0200000000600605b009617b801cfe4186271b27434d5239323430

   Is Local SAS Device: false

   Is USB: false

   Is Boot USB Device: false

   No of outstanding IOs with competing worlds: 32

Reply
0 Kudos
diegodco31
Leadership
Leadership

Hi

Check if the following VMware KB article helps:VMware Knowledge Base

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
UnS3eN
Enthusiast
Enthusiast

One of the first things I tried.

vmkernel.log output when rescanning:

2019-04-07T19:11:25.957Z cpu3:38976)VC: 2059: Device rescan time 16 msec (total number of devices 6)

2019-04-07T19:11:25.957Z cpu3:38976)VC: 2062: Filesystem probe time 59 msec (devices probed 6 of 6)

2019-04-07T19:11:25.957Z cpu3:38976)VC: 2064: Refresh open volume time 0 msec

...

2019-04-07T19:15:06.493Z cpu2:32791)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8081b380, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2019-04-07T19:15:06.493Z cpu2:32791)ScsiDeviceIO: 2338: Cmd(0x412e8081b380) 0x1a, CmdSN 0x418 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

...

2019-04-07T19:16:27.080Z cpu6:34890)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b7c33" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x6 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0x7 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2019-04-07T19:16:27.081Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0x8 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2019-04-07T19:16:27.083Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b2743" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x9 from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0xa from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0xb from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

Reply
0 Kudos
dsilvatriples
Contributor
Contributor

Do you can sharing vmkernel.log and hosts.log with the date next to reboot?

Reply
0 Kudos
a_p_
Leadership
Leadership

From the first one of the two KB articles you mentioned in your initial post, did you run vmkfstools -V (step 5), and then check the vmkernel.log for related entries?

André

Reply
0 Kudos