ESXI6.7 nvme ssd issue

[slc4xwu]

Being a software engineer, I am disturbing by the way this software create the disk ID.

According to the following table from VMware Docs we have some case that the software will not recognize the disk or lost disks.

ID Formats Supported by Device		Device Identifier Generated by Host
EUI64 ID Format	NGUID ID Format	ESXi 6.7 and earlier	ESXi 6.7 Update 2
yes	yes	t10.xxx_EUI64	t10.xxx_EUI64
yes	no	t10.xxx_EUI64	t10.xxx_EUI64
no	yes	t10.xxx_controller_serial_number	eui.xxx (NGUID) as primary ID t10.xxx_controller_serial_number as alternative primary ID

so what if

1. The NVMe SSD from the same company may use the same EUI64 for every NVMe SSD on the same interface (say using an ASUS Hyper M.2 X16 PCIe 3.0x4 Expansion card with 4 identical NVMe SSD). It is possible because of bad design from the SSD manufacturer, the EUI64 are all the same for the 4 NVMe SSD, under this case, the ESXI will only recognized one of the 4 disks (since the disk will be t10.xxx_EUI64, and EUI64 are the same for all 4 NVMe SSD, The "storage" "adapters" tab did show there are 4 interfaces (adapters), but the "stroage" "devices" will only show 1 disks.

2. Why there is no "no, no" options?

I understand that the NVMe standard 1.3 (for ESXi 6.7) is the base of how the NVMe SSD should be designed, but I think the software should be smart enough to cover the mistakes the hardware company may make and so it can recognized most of the NVMe SSD that is available from the market.

Thanks and Regards

[slc4xwu]

I am going to post an example  as the following --

[root@ESXi2:~] esxcli storage core adapter device list

HBA     Device UID

------  --------------------------------------------------------------------------------

vmhba0  t10.NVMe____PCIe_SSD________________________________393309501FA77964

vmhba1  t10.NVMe____CT1000P1SSD8____________________________1918E2009DD9________00000001

vmhba2  t10.NVMe____PCIe_SSD________________________________2020767020A77964

vmhba4  t10.NVMe____CT1000P1SSD8____________________________1918E200A04D________00000001

vmhba5  t10.NVMe____PCIe_SSD________________________________2020767020A77964

[root@ESXi2:~] esxcli storage core device uidmap list   <-- only show 4 disks

t10.NVMe____CT1000P1SSD8____________________________1918E2009DD9________00000001

   Primary UID: t10.NVMe____CT1000P1SSD8____________________________1918E2009DD9________00000001

   Alternative Primary UIDs:

   Legacy UID: vml.01000000003139313845323030394444392020202020202020435431303030

   Alternative Legacy UIDs:

t10.NVMe____PCIe_SSD________________________________393309501FA77964

   Primary UID: t10.NVMe____PCIe_SSD________________________________393309501FA77964

   Alternative Primary UIDs:

   Legacy UID: vml.0100000000333933335f303935305f314641375f3739363400504349652053

   Alternative Legacy UIDs:

t10.NVMe____CT1000P1SSD8____________________________1918E200A04D________00000001

   Primary UID: t10.NVMe____CT1000P1SSD8____________________________1918E200A04D________00000001

   Alternative Primary UIDs:

   Legacy UID: vml.01000000003139313845323030413034442020202020202020435431303030

   Alternative Legacy UIDs:

t10.NVMe____PCIe_SSD________________________________2020767020A77964

   Primary UID: t10.NVMe____PCIe_SSD________________________________2020767020A77964

   Alternative Primary UIDs:

   Legacy UID: vml.0100000000323032305f373637305f323041375f3739363400504349652053

   Alternative Legacy UIDs:

[root@ESXi2:~] esxcli nvme device namespace get -A vmhba4 -n 1

Namespace Identify Info:

   Namespace Size: 0x74706db0 Logical Blocks

   Namespace Capacity: 0x74706db0 Logical Blocks

   Namespace Utilization: 0x74706db0 Logical Blocks

   Thin Provisioning Support: false

   Namespace Atomic Support: false

   Deallocated or Unwritten Logical Block Error Support: false

   Number of LBA Formats: 0

   LBA Format: 0

   Extended Metadata: false

   Metadata as Seperate Buffer Support: false

   Metadata as Extended Buffer Support: false

   PI Type 1 Support: false

   PI Type 2 Support: false

   PI Type 3 Support: false

   PI in First Eight Bytes of Metadata Support: false

   PI in Last Eight Bytes of Metadata Support: false

   PI Enabled Type: 0

   MetaData Location: PI Disabled

   Namespace Shared by Multiple Controllers: false

   Persist Through Power Loss Support: false

   Write Exclusive Reservation Type Support: false

   Exclusive Access Reservation Type Support: false

   Write Exclusive Registrants Only Reservation Type Support: false

   Exclusive Access Registrants Only Reservation Type Support: false

   Write Exclusive All Registrants Reservation Type Support: false

   Exclusive Access All Registrants Reservation Type Support: false

   Format Progress Indicator Support: true

   Percentage Remains to Be Formatted: 0 %

   Namespace Atomic Write Unit Normal: 0

   Namespace Atomic Write Unit Power Fail: 0

   Namespace Atomic Compare and Write Unit: 0

   Namespace Atomic Boundary Size Normal: 0

   Namespace Atomic Boundary Offset: 0

   Namespace Atomic Boundary Size Power Fail: 0

   NVM Capacity: 0x0

  Namespace Globally Unique Identifier: 0x00000000000000000000000000000000   <-- both NGUID and EUI64 are "no", this is the Crucial P1 Nvme SSD, I have to down-grade the driver

   IEEE Extended Unique Identifier: 0x0

   LBA Format Support:

         Format ID: 0

         LBAData Size: 512

         Metadata Size: 0

         Relative Performance: Best performance

[Jaleopsis]

Surprised there are no replies. I'm seeing this issue where I have two identical drives and one is not being seen because the IDs are not unique.

[infy101]

I have the same issue in V7.0! Whatever happened to using something you know is unique like serial numbers or something based off the serial number!

My two identical NVMe drives show up as one! Very odd how this one got through testing!

[Posevolution]

Having same issue with the Asus hyper x16 m. 2 pcie card. I have 4 Intel 660p in it and it's only showing 1. Even when I changed the bifurcation to x4x4x4x4 for that pcie slot. It actually show nothing. When in auto it shows 1. I'm using esxi 7 though. Trying to test out VMware horizon but esxi not detecting all the nvme in the Asus adapter. 

[jsaiko1]

I'm having the exact same issue.  Has anyone been able to figure it out?  2 identical SSDs and only 1 shows up.

[adrianofante1]

The topic started in 2019 and so far it seems to me that no one has been able to solve this problem.

I'm also having the same difficulty, look:
-If I insert 2 NVME of the same brand and with the same size, only 1 is recognized in ESXi.
-If I insert 2 NVME of the same brand but of different sizes they are recognized normally.
-If I insert 2 NVME of the same size, but of different brands, the 2 are recognized normally.

Does anyone have an idea how to solve this?

HERE I HAVE SHOWING THE 4 NVME, being 2 of the same brand and size.
[r:~] esxcli nvme controller list
Name Controller Number Adapter Transport Type Is Online
-------------------------------------------------------------------------------------------- ----------------- ------- -------------- ---------
nqn.2014-08.org.nvmexpress_126f_Netac_NVMe_SSD_256GB____________________AA093445678889673711 256 vmhba4 PCIe true
nqn.2014-08.org.nvmexpress_126f_KINGBANG_KP230__________________________AA00000000003 257 vmhba5 PCIe true
nqn.2014-08.org.nvmexpress_126f_Netac_NVMe_SSD_256GB____________________AA003645689878902179 258 vmhba3 PCIe true
nqn.2021-10.com.siliconmotion:nvm-subsystem-sn-2021102900007 259 vmhba2 PCIe true

HERE ALSO APPEARS THE 4 NVME DISKS:
[r:~] esxcli storage core path list
pcie.700-pcie.0:0-t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
UID: pcie.700-pcie.0:0-t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Runtime Name: vmhba4:C0:T0:L0
Device: t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Device Display Name: Local NVMe Disk (t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000)
Adapter: vmhba4
Channel: 0
Target: 0
LUN: 0
Plugin: HPP
State: active
Transport: pcie
Adapter Identifier: pcie.700
Target Identifier: pcie.0:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 262144

pcie.900-pcie.0:0-t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
UID: pcie.900-pcie.0:0-t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Runtime Name: vmhba3:C0:T0:L0
Device: t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Device Display Name: Local NVMe Disk (t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000)
Adapter: vmhba3
Channel: 0
Target: 0
LUN: 0
Plugin: HPP
State: active
Transport: pcie
Adapter Identifier: pcie.900
Target Identifier: pcie.0:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 262144

pcie.8200-pcie.0:0-t10.NVMe____SSD_128GB_______________________________0000000000010000
UID: pcie.8200-pcie.0:0-t10.NVMe____SSD_128GB_______________________________0000000000010000
Runtime Name: vmhba2:C0:T0:L0
Device: t10.NVMe____SSD_128GB_______________________________0000000000010000
Device Display Name: Local NVMe Disk (t10.NVMe____SSD_128GB_______________________________0000000000010000)
Adapter: vmhba2
Channel: 0
Target: 0
LUN: 0
Plugin: HPP
State: active
Transport: pcie
Adapter Identifier: pcie.8200
Target Identifier: pcie.0:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 262144

pcie.800-pcie.0:0-t10.NVMe____KINGBANG_KP230__________________________0000000000000001
UID: pcie.800-pcie.0:0-t10.NVMe____KINGBANG_KP230__________________________0000000000000001
Runtime Name: vmhba5:C0:T0:L0
Device: t10.NVMe____KINGBANG_KP230__________________________0000000000000001
Device Display Name: Local NVMe Disk (t10.NVMe____KINGBANG_KP230__________________________0000000000000001)
Adapter: vmhba5
Channel: 0
Target: 0
LUN: 0
Plugin: HPP
State: active
Transport: pcie
Adapter Identifier: pcie.800
Target Identifier: pcie.0:0
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 262144

BUT WHEN LISTING THE STORAGE I ONLY SEE 3 DISCS, THE DISCS OF THE SAME BRAND AND SAME SIZE DO NOT APPEAR AND IT IS NOT POSSIBLE TO USE.
[r:~] esxcli storage core device uidmap list

t10.NVMe____KINGBANG_KP230__________________________0000000000000001
Primary UID: t10.NVMe____KINGBANG_KP230__________________________0000000000000001
Alternative Primary UIDs:
Legacy UID: vml.057b735f8a6845dab8d1ad1844b6801de68d3b3edc4cf825d73358a6ca4d297a5d
Alternative Legacy UIDs: vml.0100000000303030305f303030305f303030305f30303031004b494e474241

t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Primary UID: t10.NVMe____Netac_NVMe_SSD_256GB____________________0000000000010000
Alternative Primary UIDs:
Legacy UID: vml.05930e5cd9c6da8a23804f4d60a14cb30e7fb72bda455d6e9bf516351db1202803
Alternative Legacy UIDs: vml.0100000000303030305f303030305f303030315f30303030004e6574616320

t10.NVMe____SSD_128GB_______________________________0000000000010000
Primary UID: t10.NVMe____SSD_128GB_______________________________0000000000010000
Alternative Primary UIDs:
Legacy UID: vml.05d00dc7568003a0e4436b830c58ad3ffb4ad7caf3f8867dec2639e69c4ff3faf8
Alternative Legacy UIDs: vml.0100000000303030305f303030305f303030315f3030303000535344203132

[IRIX201110141]

Does your Host fully support PCIe Slot Bifurcation?

Regards,
Joerg

[adrianofante1]

Hello,
It's 2 HP DL360 gen9 servers and therefore doesn't support forking.

But the fork would be if I used a single card for multiple NVME.

In my case I have 3 PCI slots and each slot with 1 single NVME hd.

[Steeve_Savard]

Hello.

here is my 2 cents.

i have tried many many many different NVMe drives and can say this.

you can use M.2 NVMe drives in a server... with a PCIe to M.2 converter... BUT you cannot use the same model.

Example: i did try to use 2 ADATA M.2 on 2 distinct PCIe to M.2 converter and it didnt work.. because they have the same model/serial/reference #.

if you do the same with 2 different M.2... it would work but... on diff PCIe motherboard "Plane".

here is what i use.

i am using 5X HP 360G9 and 1X HP 380G9

originally, on all of them, i had VMware vSphere 6.5.. .now at 7.x (following upgrade path) and using vCenter.

in those server i have 8 x2TB SSD's in the front bay. (16 for the 380)

i have installed VMware on the microSD card on all of them

In those server i have added the second and third expansion slot.

i am using Intel NVMe Intel® SSD DC P4510 Series on a PCIe converter in one (Plus 1X Samsung NVMe + 1 WD M.2 NVME in the 380)

i am using the Adata M.2 in the second.

i am using a network Fiber in the last one.

the Intel NVMe Intel® SSD DC P4510 Series is setup with the "namespace" functionality.. meaning i am "separating" it into +/- 10 drives that are seeing by VMware when i look in the adapter/drives.

YES... it is a real challenge to get a clear understanding on HOW to make this work... 

why... well because not all NVMe drives offer the namespace functionality. as an example, the Intel NVMe Intel® SSD DC P4510 Series offers the possibility to "segragate" into 128 namespaces (NOT PARTITIONS).... On 1 physical drive. meaning that when you can use those as you want. And by using the "Namespace" functionality from the Intel NVMe Intel® SSD DC P4510 Series i can use it to have Caching (example: for the disk group created for vSAN) and... local cache for each VMware Host. which improve performance "dangeroulsy". --> it could be possible (if they were more like 15TB or event 18TB... to configure them with namespaces in RAID).

BE VERY CAREFULL... not all Intel NVMe Intel® SSD DC P4510 Series ... even with the same model # can do it... anything sold and that was previously in a "bundle" sold by HP/Dell/etc... will not work since they have a "proprietary" firmware. (THAT WAS A REAL FRUSTRATING SITUATION FOR ME).

At the end, my setup is ... total 6 servers, about 520 vcpu (e5-2699v4), 3TB memory... and... 110TB of vSAN capacity all flash.

believe me... that is fast... like lightning fast. 

One little BIG problem... do NOT lose power... you could end up in an "interesting situation" ... pulling your hair. Been there... done that...

Oh yes... almost forgot... Intel NVMe Intel® SSD DC P4510 Series needs to be configured using the intelMAS software and intel VIB...

If you have installed a "Vanilla" version of vSphere... you might need to take out some unused VIB's from your server "MANUALLY"

Dont look at "Micron" drives... they are expensive for nothing... Intel NVMe Intel® SSD DC P4510 Series ... is the way to go...

hope that help

[agbiront]

Hi!

Did anyone manage to solve this without changing the NVMe drives?

I have 6x512GB Kioxia NVMe drives that was planning on using for VSAN in my homelab. Started with 3, and 3 hosts, no issues.

After adding the new drives, I found that ESXi sees the NVMe drive as only 1 drive with 2 paths, because of the naming convention:

t10.NVMe____BC711_NVMe_SK_hynix_512GB_______________FFFFFFFFFFFFFFFF

I cannot change it, so the second drive is useless (well, I can change the Path Policy to fixed and point to the new drive, breaking VSAN in the process because now the first drive is inaccessible).

Running vSphere 8 with VSAN8 ESA, BTW. But the "issue" is quite old. This is a homelab with unsupported hardware, I know...

[Golvin]

The topic started in 2019 and so far it seems to me that no one has been able to solve this problem.

two nvme ssd has the same device name

t10.Nvme___XXXX_0000000000002

[stvdnd]

I would say that to correct the situation, you would be required to change the device name which is originally given by the vendor. 

that said, feel free to try... but... you may end up with a device which wont be responsive no more.

the NVMe M.2 "stick" are (from my experience) not good with namespace which lead me to say that they would also not be good with name change... i have not tried.. .but i did try to play with the namespaces and it died on me.

again, i suggest to use 2 diff brand of NVMe M.2 in the same "server" if you need too. 

i have used (in my laptop (Dell Precision 7740) Win10) 4X 2TB Sabrent NVMe M.2 and they are all the same kind/model/capacity etc... and Windows 10 has no problem seeing them independently for the total capacity of 8TB.

i also installed Win10 on one of them... and it is fast.. like real fast.

who knows... maybe installing all NVMe in a server on which you install WinServer 2019 and from there... install WorkStation on top of it can give you the view you need about your NVMe stick. i actually never tested it. 

even better.. .install WorkStation... and Nested ESXi/vCenter in it... 

have fun 🙂

cheers.

[xeonfarmer]

Has anyone made headway with this?   I have the same issue with 4 way bifurcation.