Unable to List Datastores with ESXCLI

[TopHatProductio]

I powered on an ESXi host this morning, and one of the VMs didn't boot successfully. After checking the web UI, I found that one of my datastores had gone missing. I checked the front of the server, and saw that one of my disks were red, instead of green. It may have come a bit loose. I took the disk that the datastore was on, and reinserted it. It was green again. I restarted the ESXi host, but the datastore did not re-appear. I then SSH'd into the server, and encountered the following:

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
The time and date of this login have been sent to the system logs.

WARNING:
   All commands run on the ESXi shell are logged and may be included in
   support bundles. Do not provide passwords directly on the command line.
   Most tools can prompt for secrets or accept them from standard input.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@esxi:~] clear
[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~] esxcli storage vmfs list
Error: Unknown command or namespace storage vmfs list

[root@esxi:~] clear
[root@esxi:~] esxcli storage vmfs snapshot
Usage: esxcli storage vmfs snapshot {cmd} [cmd options]

Available Namespaces:
  extent                Manage VMFS snapshot extents.

Available Commands:
  list                  List unresolved snapshots/replicas of VMFS volume.
  mount                 Mount a snapshot/replica of a VMFS volume.
  resignature           Resignature a snapshot/replica of a VMFS volume.
[root@esxi:~] clear
[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~] ls
altbootbank      bootpart.gz      lib              locker           proc             scratch          tardisks.noauto  var              vmupgrade
bin              dev              lib64            mbr              productLocker    store            tmp              vmfs
bootbank         etc              local.tgz        opt              sbin             tardisks         usr              vmimages
[root@esxi:~] cd vmfs/
[root@esxi:/vmfs] ls
devices  volumes
[root@esxi:/vmfs] cd volumes/
[root@esxi:/vmfs/volumes] ls
0642927b-2ba73473-7b07-e92e2b1d65a3        5f493ce7-cbc1516b-04b7-78e7d158470c        5fd3efc5-5a997654-fb16-78e7d1584710        Remote_Dev
5f47faa9-29b93591-b178-78e7d158470c        5f493d19-af596b3e-cec0-78e7d158470c        Arch_Based                                 Windows_Enterprise
5f47fab7-a4a33f3f-7ce2-78e7d158470c        5f493d38-009a0f4e-265f-78e7d158470c        ISO_Store                                  ccc6b470-3dd0acae-26f1-ee8f663fcba9
5f493ca1-c3f4d372-0e69-78e7d158470c        5f49a89e-112da914-a583-78e7d158470c        MacOS_X                                    vffs-52e3cf30-ea7c-2973-60c0-9fb88c799819
[root@esxi:/vmfs/volumes] sudo
-sh: sudo: not found
[root@esxi:/vmfs/volumes] su
-sh: su: not found
[root@esxi:/vmfs/volumes] clear
[root@esxi:/vmfs/volumes] ls
0642927b-2ba73473-7b07-e92e2b1d65a3        5f493d38-009a0f4e-265f-78e7d158470c        Remote_Dev
5f47faa9-29b93591-b178-78e7d158470c        5f49a89e-112da914-a583-78e7d158470c        Windows_Enterprise
5f47fab7-a4a33f3f-7ce2-78e7d158470c        5fd3efc5-5a997654-fb16-78e7d1584710        ccc6b470-3dd0acae-26f1-ee8f663fcba9
5f493ca1-c3f4d372-0e69-78e7d158470c        Arch_Based                                 vffs-52e3cf30-ea7c-2973-60c0-9fb88c799819
5f493ce7-cbc1516b-04b7-78e7d158470c        ISO_Store
5f493d19-af596b3e-cec0-78e7d158470c        MacOS_X
[root@esxi:/vmfs/volumes] esxcli storage vmfs snapshot list
[root@esxi:/vmfs/volumes] esxcli storage vmfs snapshot mount 0642927b-2ba73473-7b07-e92e2b1d65a3
Error: Unknown command or namespace storage vmfs snapshot mount 0642927b-2ba73473-7b07-e92e2b1d65a3

[root@esxi:/vmfs/volumes] esxcli storage vmfs snapshot list
[root@esxi:/vmfs/volumes] esxcli
Usage: esxcli [options] {namespace}+ {cmd} [cmd options]

Options:
  --formatter=FORMATTER
                        Override the formatter to use for a given command. Available formatter: keyvalue, xml, csv
  --debug               Enable debug or internal use options
  --version             Display version information for the script
  -?, --help            Display usage information for the script

Available Namespaces:
  device                Device manager commands
  elxnet                elxnet esxcli functionality
  esxcli                Commands that operate on the esxcli system itself allowing users to get additional information.
  fcoe                  VMware FCOE commands.
  graphics              VMware graphics commands.
  hardware              VMKernel hardware properties and commands for configuring hardware.
  iscsi                 VMware iSCSI commands.
  network               Operations that pertain to the maintenance of networking on an ESX host. This includes a wide variety of commands to manipulate virtual networking
                        components (vswitch, portgroup, etc) as well as local host IP, DNS and general host networking settings.
  nvme                  VMware NVMe driver esxcli extensions
  rdma                  Operations that pertain to remote direct memory access (RDMA) protocol stack on an ESX host.
  sched                 VMKernel system properties and commands for configuring scheduling related functionality.
  software              Manage the ESXi software image and packages
  storage               VMware storage commands.
  system                VMKernel system properties and commands for configuring properties of the kernel core system and related system services.
  vm                    A small number of operations that allow a user to Control Virtual Machine operations.
  vsan                  VMware vSAN commands

[root@esxi:/vmfs/volumes] esxcli storage
Usage: esxcli storage {cmd} [cmd options]

Available Namespaces:
  core                  VMware core storage commands.
  nfs                   Operations to create, manage, and remove Network Attached Storage filesystems.
  nfs41                 Operations to create, manage, and remove NFS v4.1 filesystems.
  nmp                   VMware Native Multipath Plugin (NMP). This is the VMware default implementation of the Pluggable Storage Architecture.
  san                   IO device management operations to the SAN devices on the system.
  vflash                virtual flash Management Operations on the system.
  vmfs                  VMFS operations.
  vvol                  Operations pertaining to Virtual Volumes
  filesystem            Operations pertaining to filesystems, also known as datastores, on the ESX host.
  iofilter              IOFilter related commands.

[root@esxi:/vmfs/volumes] esxcli storage vmfs
Usage: esxcli storage vmfs {cmd} [cmd options]

Available Namespaces:
  reclaim               Manage VMFS Space Reclamation
  snapshot              Manage VMFS snapshots.
  extent                Manage VMFS extents.
  host                  Manage hosts accessing a VMFS volume.
  lockmode              Manage VMFS array locking mode.
  pbcache               VMFS Pointer Block Cache statistics.

Available Commands:
  unmap                 Reclaim the space by unmapping free blocks from VMFS Volume
  upgrade               Upgrade a VMFS3 volume to VMFS5.
[root@esxi:/vmfs/volumes] esxcli storage vmfs snapshot
Usage: esxcli storage vmfs snapshot {cmd} [cmd options]

Available Namespaces:
  extent                Manage VMFS snapshot extents.

Available Commands:
  list                  List unresolved snapshots/replicas of VMFS volume.
  mount                 Mount a snapshot/replica of a VMFS volume.
  resignature           Resignature a snapshot/replica of a VMFS volume.
[root@esxi:/vmfs/volumes] esxcli storage vmfs snapshot list
[root@esxi:/vmfs/volumes] _

I'm unable to list datastores, for some reason. I've used this same command before, but now I can't see anything. Is there something I could to do regain this capability? Did I mess up the syntax?

[e_espinel]

Hello.
Run from an SSH connection on the ESXi host, the following commands:

List out all the devices being detected by the ESXi host.

# esxcfg-scsidevs -c

List the volumes available to the host. This includes VMFS, NAS and VFAT partitions

# esxcli storage filesystem list

with the following command you can list the snapshots present, but if they do not exist the result of the command will be to return the promt of the ESXi.

# esxcli storage vmfs snapshot list

which version and Buid you have installed on the ESXi.

Do I use a standard VMware image or a custom image from the server manufacturer to install VMware vSphere?

Please attach the result of the commands indicated in this post

[TopHatProductio]

Thank you for getting back to me quickly. The server is running standard (I sometimes refer to that as vanilla/stock) ESXi 6.5u3. Here are the results of the commands (screenshot, then copy/paste):

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
The time and date of this login have been sent to the system logs.

WARNING:
   All commands run on the ESXi shell are logged and may be included in
   support bundles. Do not provide passwords directly on the command line.
   Most tools can prompt for secrets or accept them from standard input.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@esxi:~] esxcfg-scsidevs -c
Device UID                            Device Type      Console Device                                            Size      Multipath PluginDisplay Name
mpx.vmhba0:C0:T0:L0                   CD-ROM           /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0                   0MB       NMP     Local Optiarc CD-ROM (mpx.vmhba0:C0:T0:L0)
naa.5000c50057ff26eb                  Direct-Access    /vmfs/devices/disks/naa.5000c50057ff26eb                  3815447MB NMP     Local IBM-XIV Disk (naa.5000c50057ff26eb)
naa.5000c50057ff4263                  Direct-Access    /vmfs/devices/disks/naa.5000c50057ff4263                  3815447MB NMP     Local IBM-XIV Disk (naa.5000c50057ff4263)
naa.5000c50057ff882b                  Direct-Access    /vmfs/devices/disks/naa.5000c50057ff882b                  3815447MB NMP     Local IBM-XIV Disk (naa.5000c50057ff882b)
naa.5000c50057ff969f                  Direct-Access    /vmfs/devices/disks/naa.5000c50057ff969f                  3815447MB NMP     Local IBM-XIV Disk (naa.5000c50057ff969f)
naa.5000cca2520716e4                  Direct-Access    /vmfs/devices/disks/naa.5000cca2520716e4                  7630885MB NMP     Local HGST Disk (naa.5000cca2520716e4)
naa.5000cca2520769d8                  Direct-Access    /vmfs/devices/disks/naa.5000cca2520769d8                  7501160MB NMP     Local HGST Disk (naa.5000cca2520769d8)
naa.5000cca25208546c                  Direct-Access    /vmfs/devices/disks/naa.5000cca25208546c                  7501160MB NMP     Local HGST Disk (naa.5000cca25208546c)
naa.5000cca25208549c                  Direct-Access    /vmfs/devices/disks/naa.5000cca25208549c                  7501160MB NMP     Local HGST Disk (naa.5000cca25208549c)
naa.50060480b80f473e                  Enclosure Svc Dev/vmfs/devices/genscsi/naa.50060480b80f473e                0MB       NMP     Local EMC Enclosure Svc Dev (naa.50060480b80f473e)
naa.600508b1001c14f5be87c23d7d7cb769  Direct-Access    /vmfs/devices/disks/naa.600508b1001c14f5be87c23d7d7cb769  476908MB  NMP     Local HP Disk (naa.600508b1001c14f5be87c23d7d7cb769)
naa.600508b1001c2724e7e687ff1124d4d8  Direct-Access    /vmfs/devices/disks/naa.600508b1001c2724e7e687ff1124d4d8  381437MB  NMP     Local HP Disk (naa.600508b1001c2724e7e687ff1124d4d8)
naa.600508b1001c40d842f0eee882c9ba80  Direct-Access    /vmfs/devices/disks/naa.600508b1001c40d842f0eee882c9ba80  286070MB  NMP     Local HP Disk (naa.600508b1001c40d842f0eee882c9ba80)
naa.600508b1001c4b01fe6fbb44ec91ec3f  Direct-Access    /vmfs/devices/disks/naa.600508b1001c4b01fe6fbb44ec91ec3f  572293MB  NMP     Local HP Disk (naa.600508b1001c4b01fe6fbb44ec91ec3f)
naa.600508b1001c9177a6f267115eebac20  Direct-Access    /vmfs/devices/disks/naa.600508b1001c9177a6f267115eebac20  476908MB  NMP     Local HP Disk (naa.600508b1001c9177a6f267115eebac20)
naa.600508b1001cdb8c73de40eb88711135  Direct-Access    /vmfs/devices/disks/naa.600508b1001cdb8c73de40eb88711135  286070MB  NMP     Local HP Disk (naa.600508b1001cdb8c73de40eb88711135)
naa.600508b1001cfaa8ebac55088acb2286  Direct-Access    /vmfs/devices/disks/naa.600508b1001cfaa8ebac55088acb2286  286070MB  NMP     Local HP Disk (naa.600508b1001cfaa8ebac55088acb2286)
[root@esxi:~] esxcli storage filesystem list
Mount Point                                        Volume Name                                UUID                                 Mounted  Type             Size           Free
-------------------------------------------------  -----------------------------------------  -----------------------------------  -------  ------  -------------  -------------
/vmfs/volumes/5f493ca1-c3f4d372-0e69-78e7d158470c  Arch_Based                                 5f493ca1-c3f4d372-0e69-78e7d158470c     true  VMFS-6   299842404352     3778019328
/vmfs/volumes/5f493ce7-cbc1516b-04b7-78e7d158470c  MacOS_X                                    5f493ce7-cbc1516b-04b7-78e7d158470c     true  VMFS-6   299842404352      211812352
/vmfs/volumes/5f493d19-af596b3e-cec0-78e7d158470c  Windows_Enterprise                         5f493d19-af596b3e-cec0-78e7d158470c     true  VMFS-6   299842404352     7107248128
/vmfs/volumes/5f493d38-009a0f4e-265f-78e7d158470c  Remote_Dev                                 5f493d38-009a0f4e-265f-78e7d158470c     true  VMFS-6   499826819072   494350106624
/vmfs/volumes/5f49a89e-112da914-a583-78e7d158470c  ISO_Store                                  5f49a89e-112da914-a583-78e7d158470c     true  VMFS-6   138781130752    57018417152
/vmfs/volumes/5fd3efc5-5a997654-fb16-78e7d1584710  vffs-52e3cf30-ea7c-2973-60c0-9fb88c799819  5fd3efc5-5a997654-fb16-78e7d1584710     true  VFFS    1099780063232  1098755604480
/vmfs/volumes/0642927b-2ba73473-7b07-e92e2b1d65a3                                             0642927b-2ba73473-7b07-e92e2b1d65a3     true  vfat        261853184      103759872
/vmfs/volumes/5f47fab7-a4a33f3f-7ce2-78e7d158470c                                             5f47fab7-a4a33f3f-7ce2-78e7d158470c     true  vfat       4293591040     4242931712
/vmfs/volumes/ccc6b470-3dd0acae-26f1-ee8f663fcba9                                             ccc6b470-3dd0acae-26f1-ee8f663fcba9     true  vfat        261853184      261840896
/vmfs/volumes/5f47faa9-29b93591-b178-78e7d158470c                                             5f47faa9-29b93591-b178-78e7d158470c     true  vfat        299712512      117432320
[root@esxi:~] esxcli storage vmfs snapshot list
[root@esxi:~]

Please let me know if there is anything else that you need.

[TopHatProductio]

In addition to this, the disk has the following properties:

• Capacity: 300GB
• Type: SAS HDD
• Vendor: HP

In the web UI, this is what storage devices page looks like:

The datastore in question is located on a disk that's directly connected to the server's SAS backplane. From the limited analysis I could do, these three items seem to closely match the description. However, I can't tell which one is mounted and in use:

naa.600508b1001c40d842f0eee882c9ba80  Direct-Access    /vmfs/devices/disks/naa.600508b1001c40d842f0eee882c9ba80  286070MB  NMP     Local HP Disk (naa.600508b1001c40d842f0eee882c9ba80)
naa.600508b1001cdb8c73de40eb88711135  Direct-Access    /vmfs/devices/disks/naa.600508b1001cdb8c73de40eb88711135  286070MB  NMP     Local HP Disk (naa.600508b1001cdb8c73de40eb88711135)
naa.600508b1001cfaa8ebac55088acb2286  Direct-Access    /vmfs/devices/disks/naa.600508b1001cfaa8ebac55088acb2286  286070MB  NMP     Local HP Disk (naa.600508b1001cfaa8ebac55088acb2286)

[TopHatProductio]

Rebooted the server and took another look at the screen for some clues, got this:

Re-enabled the logical drive and let ESXi go at it. I think the issue may be resolved, but am going to be cautious for the next few days.

I'll let you know if something changes. So far, so good.

[e_espinel]

Hello.
From the information sent we have:
The recognized filesystems are all mounted, i.e. you have access to them.

There are no snapshots on the ESXi host.

I see an IBM XIV external storage of which you have several LUNs assigned to the ESXi host.

You have several HP disks connected and configured on the ESXi host, so your controller is HP?

Run the following command to validate the controllers that the ESXi host has.
esxcfg-scsidevs -a

If your server is HPE or HP, or if disk controllers are HP you should install the following utility to manage and configure HP controllers and disks

HPE Smart Storage Administrator (HPE SSA) CLI for VMware 6.5

https://support.hpe.com/hpesc/public/swd/detail?swItemId=MTX_5fa5b0d965c84875b9b7410471

The installation of the utility is online and does not require restarting the ESXi host.

In the following link there is more information about HPE SSACLI
The important thing is to look at the status of the physical disks, the status of the Arrays and the status of the Volumes.

https://www.computermasters.it/hpe-smart-storage-administrator-hpe-ssa-cli/

[TopHatProductio]

1. I wish I could have snapshots, but all of my VMs have either PCIe devices passed to them, raw disks attached, or both 😞 
2. The IBM drives are high-capacity devices, reserved primarily for backups and large files.
3. The server is an HP ProLiant. The controller that the previously-troubled datastore is attached to is the built-in one provided by HP.
4. Here are the results of the command:

[root@esxi:~] esxcfg-scsidevs -a
vmhba0  vmkata            link-n/a  sata.vmhba0                             (0000:00:1f.2) Intel Corporation ICH10 4 port SATA IDE Controller
vmhba1  nhpsa             link-n/a  sas.500143800922ab50                    (0000:03:00.0) Hewlett-Packard Company Smart Array P410i
vmhba2  mpt2sas           link-n/a  sas.5000d31000482e2d                    (0000:08:00.0) LSI Logic / Symbios Logic LSI2116_1
vmhba64 vmkata            link-n/a  sata.vmhba64                            (0000:00:1f.2) Intel Corporation ICH10 4 port SATA IDE Controller
[root@esxi:~]
​

Thank you for providing the link to "HPE Smart Storage Administrator (HPE SSA) CLI for VMware 6.5". I took a look around, and found this:

• https://support.hpe.com/hpesc/public/swd/detail?swItemId=MTX-dafe3325c6dc4488b53bb3f3c7

Is this a newer version?

I will be sure to install this utility when I have some time.