ESXi 6.5 - HP P420i - HBA mode - Failed to create VMFS datastore

[GeertCRW]

Hi

I have a HP D360p with a P420i controller

P420i firmware = 8.32
HBA mode
All connected SSD disks are visible in ESXi 6.5
When i try to create a new Datastore, i get the following message

Failed to create VMFS datastore yyy - Operation failed, diagnostics report: Unable to create Filesystem, please see VMkernel log for more details: Failed to create VMFS on device naa.5002538d4079ba3c:3

VMkernel logging

2018-02-01T20:52:01.363Z cpu6:67831 opID=a0767170)LVM: 12525: LVMProbeDevice failed with status "Device does not contain a logical volume".

2018-02-01T20:52:01.399Z cpu13:66202)nhpsa: hpsa_vmkScsiCmdDone:6384: Sense data: error code: 0x70, key: 0x5, info:00 00 00 00 , cmdInfo:00 00 00 00 , CmdSN: 0x1c89, worldId: 0x0, Cmd: 0x16, ASC: 0x20, ASCQ: 0x0

2018-02-01T20:52:01.399Z cpu13:66202)ScsiDeviceIO: 2962: Cmd(0x4395013f0e00) 0x16, CmdSN 0x1c89 from world 0 to dev "naa.5002538d4079ba3c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2018-02-01T20:52:01.401Z cpu13:67831 opID=a0767170)LVM: 10306: LVMProbeDevice failed on (3593002560, naa.5002538d4079ba3c:4): Device does not contain a logical volume

  2018-02-01T20:52:01.401Z cpu13:67831 opID=a0767170)FSS: 2295: Failed to create FS on dev [naa.5002538d4079ba3c:4] fs [lll] type [vmfs6] fbSize 1048576 => Not supported

Thanks for any help

Geert

[daphnissov]

Show the output of esxcli storage core device list please.

[GeertCRW]

[root@esxi2:~] esxcli storage core device list

mpx.vmhba32:C0:T0:L0

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Has Settable Display Name: false

   Size: 3830

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Vendor: Generic

   Model: Ultra Fast Media

   Revision: 0210

   SCSI Level: 2

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: true

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0000000000766d68626133323a303a30

   Is Shared Clusterwide: false

   Is Local SAS Device: false

   Is SAS: false

   Is USB: true

   Is Boot USB Device: true

   Is Boot Device: true

   Device Max Queue Depth: 1

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

naa.500253885006d47f

   Display Name: Local ATA Disk (naa.500253885006d47f)

   Has Settable Display Name: true

   Size: 114473

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.500253885006d47f

   Vendor: ATA

   Model: Samsung SSD 840

   Revision: DB6Q

   SCSI Level: 6

   Is Pseudo: false

   Status: on

   Is RDM Capable: true

   Is Local: true

   Is Removable: false

   Is SSD: true

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0200000000500253885006d47f53616d73756e

   Is Shared Clusterwide: false

   Is Local SAS Device: true

   Is SAS: true

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 14

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

naa.5002538d4079ba3c

   Display Name: Local ATA Disk (naa.5002538d4079ba3c)

   Has Settable Display Name: true

   Size: 476940

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.5002538d4079ba3c

   Vendor: ATA

   Model: Samsung SSD 850

   Revision: 2B6Q

   SCSI Level: 6

   Is Pseudo: false

   Status: on

   Is RDM Capable: true

   Is Local: true

   Is Removable: false

   Is SSD: true

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.02000000005002538d4079ba3c53616d73756e

   Is Shared Clusterwide: false

   Is Local SAS Device: true

   Is SAS: true

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 14

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

[root@esxi2:~]

[daphnissov]

Now show output of the following commands:

partedUtil getptbl /vmfs/devices/disks/naa.500253885006d47f

and

partedUtil getptbl /vmfs/devices/disks/naa.5002538d4079ba3c

[GeertCRW]

Before trying to create the datastore

[root@esxi2:~] partedUtil getptbl /vmfs/devices/disks/naa.500253885006d47f

gpt

14593 255 63 234441648

[root@esxi2:~] partedUtil getptbl /vmfs/devices/disks/naa.5002538d4079ba3c

gpt

60801 255 63 976773168

After trying to create the datestore (when getting the error message)

[root@esxi2:~] partedUtil getptbl /vmfs/devices/disks/naa.500253885006d47f

gpt

14593 255 63 234441648

1 2048 234438656 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[root@esxi2:~] partedUtil getptbl /vmfs/devices/disks/naa.5002538d4079ba3c

gpt

60801 255 63 976773168

1 2048 976771072 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[daphnissov]

Try to clear the partition table and create a VMFS volume once again, but this time choose VMFS-5. Does it still fail?

[GeertCRW]

When selecting VMFS5  , i'm getting the same error

[daphnissov]

Was this after you cleared the partition table on the drives?

[GeertCRW]

Yes, I first clear the partition table

[daphnissov]

Did you ever, at any point, have any other data on these SSDs before putting them inside this ESXi host?

[GeertCRW]

Yes,

i have also tried clear the drive in a other system (even black the disk with zeros)
But still creating a datastore fails

[daphnissov]

What build of ESXi is this (vmware -v) and what driver and driver version is in use for this storage controller to which these SSDs are attached?

[GeertCRW]

Hi

To answer your question

VMware ESXi 6.5.0 build-7388607

nhpsa                          2.0.16-1OEM.650.0.0.4598673          Microsemi        VMwareCertified

Not sure if this is important but the controller i'm using: (HP P420i) has no memory or battery installed

     All i could find in the documentation, memory is only required if you want to use RAID 5 or use more then 2 disks

After several hours of reboots and testing 😞

Before i start playing with enabling HBA mode , I was successfully using the P420i controller in RAID mode (2x disks each in RAID-0) and this for several months

At this moment HBA mode is still not working, but I was able to get regular RAID mode working again.

It seems that each time that i enabled HBA mode, the controller was  F#*@*& up, and i needed to perform specific steps to get it working in RAID mode

The issues seem not related to

• the controller firmware, i have tested with firmware version 7.02 , 8.00 (current that i'm using) , 8.32
• ESXi drivers, i have tested with nhpa version 2.0.14 and 2.0.16
  VMware Compatibility Guide - I/O Device Search
• Different ESXi build versions don't make a difference (I only tried ESXi 6.5)

How i tried to switch to HBA mode

• using command line in ESXi

      ./opt/smartstorageadmin/ssacli/bin/ssacli controller slot=0 modify hbamode=on forced

• using HP Smart Storage Administrator (HPSSA)
  Re: HP P420i/P440i HBA Mode = on ( If you want to use embedded controller on DL series without the s...

Every time i switched to HBA mode,  I got the error message that i mentioned in the beginning of the thread

When i gave up to use HBA and tried to switch back to RAID mode, i got the same error message as when it was in HBA :smileycry:

     I tried to clear config in HPSSA / upgrade & downgrade to different firmware versions / use other ESXi drivers , nothing help to get RAID mode working again

In the end i got RAID mode working by doing these steps

1. Use HPSSA to enable RAID mode
   
   
2. Boot to ESXi,
   in the CLI delete the RAID config
          ./opt/smartstorageadmin/ssacli/bin/ssacli controller slot=0 delete forced
   (using the clear option in HPSSA did help, it really needed to be done from the ESXi CLI, why ?? no idea )
   
   
3. Reboot the server
   
   
4. enter HPSSA
   configure the disks in RAID-0 again
   
   
5. Now in ESXi the disks are displayed and usable
   
   Extra:
   for 1 of the disks, i needed to do the following steps before i could create a new datastore on it
   (clearing the partition in the HTML5 client was not working)
   VMware Fix for Call “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object “ha-datastoresy...

Grz

Geert

[yaoyuanzy]

@

I have same question,but I think i found what case this situation.By compare with  my MS-Gen8, There is different:Look at RED line.

naa.500253885006d47f

   Display Name: Local ATA Disk (naa.500253885006d47f)

   Has Settable Display Name: true

   Size: 114473

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.500253885006d47f

   Vendor: ATA

   Model: Samsung SSD 840

   Revision: DB6Q

   SCSI Level: 6

   Is Pseudo: false

   Status: on

   Is RDM Capable: true

   Is Local: true

   Is Removable: false

   Is SSD: true

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.0200000000500253885006d47f53616d73756e

   Is Shared Clusterwide: false

   Is Local SAS Device: true

   Is SAS: true

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 14

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

I think when HP P420 and other raid cards in HBA mode,ESXI recoginze disk type is SATA but set it to SAS.

It's easy when system use SAS  Protcal to communicate with SATA disk,all funcation is bad.

And I tried in ESXI to Found something can change That,but is non working,now i will also back to raid mode.

@daphnissov

[pkern]

Apparently I've just came across the same bug here, with absolutely no further ideas how I might get this solved in my lab.

The current situation

• HP DL360p Gen8
• VMware ESXi 6.7 U1
• HP Smart Array P420i with latest 8.32 firmware and in HBA mode
• Driver of nhpsa with version 2.0.30-1OEM.670.0.0.7535516 (listed as compatible at VMware HCL​)
• SSDs of different brands
• Confirmed disks working in RAID mode
• All disks are visible in "Configure - Storage Devices" just fine

What I've tried so far:

• Confirmed HP Smart Array P420i compatible with specific nhpsa driver, firmware and ESXi version
• Verified that ESXi is using the correct driver using esxcfg-scsidevs -a
• Re-flashing firmware v8.32 of Hardware RAID Controller
• Rebooting hosts a couple of times
• Tried different SSD brands and different disks -> Same errors on all disks
• Trying to add disks as vFlash -> Fails
• Trying to create a datastore on the disks -> Fails
• Trying to create a datastore using vmkfstools -> Fails
• Trying to find out any helpful information from vmkernel.log -> Similar logs as in original thread, nothing helpful unfortunately
• Formatting disk using "Actions - Erase partitions..." -> Fails
• Formatting disk using partedUtils by using "mklabel" or deleting all partitions with "delete" -> seems not to work, but same error when creating datastore/filesystem on it
• Formatting disk using dd -> seems not to work, but just like above... error when creating datastore on it
• Googling around and having no clue anymore

Output: vmkfstools

[root@esxi:~] vmkfstools -C vmfs6 -b 1M -S esxi04Store /vmfs/devices/disks/naa.50026b778248badc:1

create fs deviceName:'/vmfs/devices/disks/naa.50026b778248badc:1', fsShortName:'vmfs6', fsName:'esxiStore'

deviceFullPath:/dev/disks/naa.50026b778248badc:1 deviceFile:naa.50026b778248badc:1

ATS on device /dev/disks/naa.50026b778248badc:1: not supported

.

Checking if remote hosts are using this device as a valid file system. This may take a few seconds...

Creating vmfs6 file system on "naa.50026b778248badc:1" with blockSize 1048576, unmapGranularity 1048576, unmapPriority default and volume label "esxiStore".

Failed to create VMFS on device naa.50026b778248badc:1

Usage: vmkfstools -C [vmfs5|vmfs6|vfat] /vmfs/devices/disks/vml... or,

       vmkfstools -C [vmfs5|vmfs6|vfat] /vmfs/devices/disks/naa... or,

       vmkfstools -C [vmfs5|vmfs6|vfat] /vmfs/devices/disks/mpx.vmhbaA:T:L:P

Error: vmkfstools failed: vmkernel is not loaded or call not implemented.

I'd be really thankful for any suggestions and ideas!

[rm_bk]

I'm hoping you guys can find a fix for this.  I went through the same thing a while back and wrote off the P420 as having a broken HBA mode.  I tried everything I could get my hands on to no avail.

[pkern]

I think @yaoyuanzy was on the correct way. I've now tried following:

• P420i is still in HBA mode
• SATA SSDs as Flash devices still not working
• SATA SSDs manually marked as harddrives also not working
• I added a HP SAS disk and created a datastore on it: It worked without any issues. On the same HBA, without further changes.

Then I've checked "Configure - Storage Devices - <Any SATA SSD> - Properties - Transport" down below. I clearly states that the SATA SSD is wrongly detected as SAS and is using the "SAS Transport". In my opinion that is the cause. Apparently ESXi tries to use SAS transport on just SATA disks, and therefore fails using them.

I'm having a HPE Smart Array P421 in HBA mode with same firmware running on a bare-metal FreeNAS installation (HP machine) and all the SATA disks are correctly detected as SATA disks and working fine since years, so I assume it's more likely a driver-side issue in ESXi, using the wrong transport.

[pkern]

A new observation: I've now booted the same ESXi host with the HBA using Ubuntu 18.04.1 Desktop (Live Image) and ran "lshw -c disk" on it: Ubuntu is correctly detecting the SATA disks as SATA, and the single HP SAS Disk as SAS - see screenshot attached. So apparently the issue seems to be on ESXi-side, using the wrong transports for SATA disks. I'm going to debug this further with some specific nhpsa drivers in the next few days...

[erickbm]

Thanks for this. I am in the same boat. VSAN via passthrough will not work on the p420i even though it is listed on the HCL as able to do so. I cam only get vsan to claim the disks if they are in RAID 0 and not HBA mode. I would love the ability to use passthrough with vsan.

[RattleZ]

I tend to agree with this observation. Here is my setup:

3 x HP DL360p Hosts, all with the SmartArray P420i adapter installed (without FBWC module).

Each Host has 3 x 500GB SAS HDD's + 1 x 240GB SATA SSD.

There are no issues creating a VSAN Datastore using them in a RAID-0 config. Bear in mind that with the controller set to RAID mode, I only can use 2 disks - 1xSSD+1xHDD. This is due to the limitation of the P420i without having the FBWC module installed, but is good enough for a test.

I am running 6.7U1 and also use the latest and greatest FW 8.32 and the nhpsa driver 2.0.30-1OEM

Here are my observations:

• Controller in RAID mode works, BUT the Queue depth of the controller is showing 32, which is way below of what is actually capable of. According to the "Community VSAN Storage Controller Queue Depth List", this controller is listed to have a Queue depth of 1020 in RAID0 mode. In my opinion, there could be 4 reasons for this:
  • nhpsa Driver issue/bug
  • The mix and match use of SATA + SAS disks
  • Controller Firmware issue
  • No FBWC module installed
• Also interestingly, the Controller is listed with a Queue depth of 21 when used in HBA mode, according to the "Community VSAN Storage Controller Queue Depth List" . I have tested this, and there is no Change in the Queue depth. Mine is still showing a Queue depth of 32. Is anyone else able to check the queue depth of the controller (possibly with FBWC module)? Command for this: esxtop,d ,f,d -> The value listed for your P420i vmhba under the column AQLEN shows the Queue depth.
• Setting the controller to HBA mode -> Unable to create a VSAN, also all the disks (SATA SSD + SAS HDDs) are detected as SAS disks, when verifying this with: esxcli storage core device list -> Is SAS: true
• I have raised a ticket with VMware for this and after some troubleshooting on their end, they have advised to get in touch with the vendor (HPE) and confirm the following:
  • Does HP SmartArray P420i support SATA drives?
  • Does HP SmartArray P420i support mixing of SATA with SAS drives?
  • Are there any outstanding bugs related to HP SmartArray P420i which is not allowing you to utilize the complete queue depth since it is showing as 32 only.
  • Are there any known issues on the hardware as we are seeing an issue in claiming the disks while enabling VSAN since we this error message: "A general system error occurred: Failed to reserve disk  with exception: Failed to reserve disk naa.500a07511a52a981 with exception: Reserve failed with error code: -1 "
    ->  As we are not entitled to Vendor support anymore for the Gen8 HP Hosts, we are unable to ask HPE for assistance. But here is what I've found out so far according to the official Conrtoller+Host documentation (I have attached a few screenshots about this). Also here are the links to the official documentation: Controller Overview and Server Manual 
    • YES, HP SmartArray P420i DOES support SATA drives.
    • YES, HP SmartArray P420i DOES support mixing of SATA with SAS drives.
    • NO IDEA about outstanding bugs or hardware isues.
• One interesting thing VMware pointed me to is indeed the HCL where the Controller is listed as Device Type: SAS. Does this mean from a VMware perspective that the Controller is ONLY considered as SAS capable? HCL LINK

It would be good to know if:

1. Anyone else has the above mentioned Queue depth issue (with and/or withour FBWC module)
2. Is it possible to manually override/set the Transport protocol for a disk using esxcli etc. ?
3. Trying to set up a VSAN with a SAS only configuration (not mixed SATA+SAS disks).

I was hoping for a straightforward VSAN installation as I have verified and then double-checked the VMware HCL and bought specific Hardware according to this.

Any help on this is greatly appreciated. I will also try to get further help from VMware.

Attached a few screenshots for your reference.