chrisb81
Contributor
Contributor

Datastore disappeared in ESXi 6.7

Hi all,

I had an issue where I lost connectivity to my ESXi machine, and it looked like the vSphere itself was the culprit. After being unable to connect to it or operate it directly from the terminal, I hard reset the machine.

It turned out that a faulty switch was to blame for a lack of connectivity, and I'd plugged the terminal output into the main VGA out (which was giving a BIOS message as an output), not realising there was a DVI graphics card in the machine as well (where the VMware hypervisor was outputting).

So, the hard reset was totally pointless. Unfortunatley, when the machine came back up, one of my datastores had gone.

The ESXi web console shows the storage as still being there, and will also show the partition. partedUtil states that the partition table is fine, but if I try to mount it, I'm told that the volume doesn't exist.

I then ran

voma -m vmfs -f check -d [diskname]

And received the following message:

Running VMFS Checker version 2.1 in check mode              

Initializing LVM metadata, Basic Checks will be done        

         ERROR: IO failed: Input/output error               

   VOMA failed to check device : IO error                   

                                                            

Total Errors Found:           0

   Kindly Consult VMware Support for further assistance

Is this drive toast? It has some data on it that I would like to retrieve.

Any advice greatly appreciated!

Chris.

0 Kudos
6 Replies
nachogonzalez
Expert
Expert

Hey, hope you are doing fine
Quick question:

What is the backing device for this datastore? (NFS, iSCSI, SAN, etc)
In case it is SAN, is it possible that the zoning is failing? (since you mentioned a failing switch)

Has the datastore dissapeared to all the hosts in the cluster?

Can you see it on the storage array?

0 Kudos
chrisb81
Contributor
Contributor

Hi, sorry I should have mentioned - it's just a basic SATA setup. This is my personal dev machine at home, so it's not particularly high-end.

0 Kudos
nachogonzalez
Expert
Expert

can you please run the following commands and paste the outputs?

esxcli storage core device list

esxcli storage filesystem list

Also, can you rescan the esxi host?

0 Kudos
chrisb81
Contributor
Contributor

Hi,

Thanks for the help on this - sorry for my slow reply. I've pasted the results below. The disk that's disappeared is t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52514______ and when typing esxcli storage filesystem list it no longer appears as a volume

[root@localhost:~] esxcli storage core device list          

t10.ATA_____ATP_IG_SATA_DOM_________________________99001130523112000021

   Display Name: Local ATA Disk (t10.ATA_____ATP_IG_SATA_DOM_________________________99001130523112000021)

   Has Settable Display Name: true

   Size: 0

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ATP_IG_SATA_DOM_________________________99001130523112000021

   Vendor: ATA    

   Model: ATP IG SATA DOM

   Revision: 1212

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: true

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unknown

   Other UIDs: vml.01000000003939303031313330353233313132303030303231415450204947

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 16

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

mpx.vmhba0:C0:T3:L0

   Display Name: Local TSSTcorp CD-ROM (mpx.vmhba0:C0:T3:L0)

   Has Settable Display Name: false

   Size: 0

   Device Type: CD-ROM

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/cdrom/mpx.vmhba0:C0:T3:L0

   Vendor: TSSTcorp

   Model: CDDVDW SN-208DB

   Revision: SB01

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: true

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0005000000766d686261303a333a30

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52509______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52509______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52509______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433532353039202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51877______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51877______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51877______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433531383737202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

mpx.vmhba32:C0:T0:L0

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Has Settable Display Name: false

   Size: 29327

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Vendor: SanDisk

   Model: Ultra          

   Revision: 0100

   SCSI Level: 2

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: true

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.01000000003443353330303031333131303139313138333133556c74726120

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: true

   Is Boot Device: true

   Device Max Queue Depth: 1

   No of outstanding IOs with competing worlds: 1

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51883______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51883______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51883______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433531383833202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52514______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52514______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52514______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433532353134202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51882______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51882______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC51882______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433531383832202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52511______

   Display Name: Local ATA Disk (t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52511______)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/t10.ATA_____ST500DM005_HD502HJ______________________S20BJ90CC52511______

   Vendor: ATA    

   Model: ST500DM005 HD502

   Revision: 0001

   SCSI Level: 5

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0100000000533230424a393043433532353131202020202020535435303044

   Is Shared Clusterwide: false

   Is SAS: false

   Is USB: false

   Is Boot Device: false

   Device Max Queue Depth: 31

   No of outstanding IOs with competing worlds: 31

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

[root@localhost:~] esxcli storage filesystem list

Mount Point                                        Volume Name  UUID                                 Mounted  Type            Size          Free

-------------------------------------------------  -----------  -----------------------------------  -------  ------  ------------  ------------

/vmfs/volumes/5db5e3bc-20c43806-4d33-001e677b3802  disk00       5db5e3bc-20c43806-4d33-001e677b3802     true  VMFS-6  499826819072  480281362432

/vmfs/volumes/5db644b8-59967b90-d1a8-001e677b3802  disk01       5db644b8-59967b90-d1a8-001e677b3802     true  VMFS-6  499826819072  209363927040

/vmfs/volumes/5db85a91-c0eb4a3e-c1ad-001e677b3802  disk02       5db85a91-c0eb4a3e-c1ad-001e677b3802     true  VMFS-6  499826819072  214658187264

/vmfs/volumes/5dba04aa-7b478990-8ac3-001e677b3802  disk03       5dba04aa-7b478990-8ac3-001e677b3802     true  VMFS-6  499826819072   51540656128

/vmfs/volumes/5dbace2a-39f84e9a-697f-001e677b3802  disk04       5dbace2a-39f84e9a-697f-001e677b3802     true  VMFS-6  499826819072   68715282432

/vmfs/volumes/00f1b8e3-987867bb-4eaf-c8460b5a2851               00f1b8e3-987867bb-4eaf-c8460b5a2851     true  vfat       261853184     106242048

/vmfs/volumes/056ebbec-cca57717-1eff-d956352b4f93               056ebbec-cca57717-1eff-d956352b4f93     true  vfat       261853184     261849088

/vmfs/volumes/5db5d2a1-62103664-f055-001e677b3802               5db5d2a1-62103664-f055-001e677b3802     true  vfat       299712512     117448704

0 Kudos
nachogonzalez
Expert
Expert

Hey hope you are doing fine, can you do a rescan of the disks?

0 Kudos
scott28tt
VMware Employee
VMware Employee

Moderator: Please consider using the Attach function in the bottom right of the post creator rather than doing a copy/paste text dump into a post.

0 Kudos