5 Replies Latest reply on May 23, 2019 1:18 AM by depping

    ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue

    mikev80 Lurker

      I have 10 x Intel P4500 and 2 x Intel P4600 in my 4 hosts vsan cluster.  I only see 2 of them in the GUI with ESXi 6.7U2.

       

      Here's where it gets interesting:

      • esxcli storage core nmp device list = sees 1 x P4500 and 1 x P4600

      What's weird is that each is assigned a unique ID following the new disk naming convention that starts with eui.xxx, but 1 of them have 9 "Other IDs" following the older naming convention t10.NVMe_____Intel P4500___<unique serial number>

       

      • ls /dev/disks = sees 1xP4500 and 1xP4600

      Same as above.  unique eui.xxx file, but symlinks that points to the same 2 eui.xxx file

       

      • esxcfg-scsidevs -l = sees 2 of them

      same as above

       

      • esxcli nvme device list = sees all 12!

       

      • esxcfg-scsidevs -a = sees all 12!

       

      storage adapter path in the GUI also sees 12 unique paths.

       

      I've opened a ticket already with support but not making much progress.

      Is it normal for 6.7 to lump all make/model into one unique ID?  My SATA drives are showing up unique but not the NVMe drives.

        • 1. Re: ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue
          TheBobkin Virtuoso
          VMware EmployeesvExpert

          Hello mikev80

           

           

          Welcome to (posting on ) Communities.

           

          What driver and firmware are you using on these and what are the part numbers for both devices (or vSAN VCG entry URL if you have it)?

          Is there any difference in firmware between the devices that can be seen and the ones that cannot?

          Them not being seen in /dev/disks means they are not being picked up by ESXi as usable storage devices not just vSAN.

           

          Any chance some of them have not been detected/marked as Local storage? (then again that wouldn't explain them not being in /dev/disks so probably not possible).

          100% positive nothing silly like RAID configuration applied to the devices?

          Can you share the output of:

          # esxcli storage core device list

          # vdq -q

           

           

          Bob

          • 2. Re: ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue
            mikev80 Lurker

            Thanks!

            When I run esxcfg-scsidev -a to find the driver, i see it listed as just "nvme" for the 12 NVMe drives.

            Any suggestion on how to check firmware of the drives that cannot be seen?

            The system does somehow recognize the drives because I see them assigned to a vmhba. 

            I also see them as individual symlinks in /dev/disks but not as a file.  Each symlink has the naming convention with each drives' serial number in the name and maps back to one of the 2 eui.xxx files.

            Nameing convention changed with 6.7U2: NVMe Devices with NGUID Device Identifiers

             

            It feels like ESXi thinks there is 1xP4500 drive with 2 multipaths and 1xP4600 drive with 9 multipaths instead of 12 individual drives.

             

            esxcli storage core device list shows 2 drives and so does vdq -q

            • 3. Re: ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue
              TheBobkin Virtuoso
              VMware EmployeesvExpert

              Hello Mike,

               

               

              "When I run esxcfg-scsidev -a to find the driver, i see it listed as just "nvme" for the 12 NVMe drives."

              There are numerous methods of getting the driver version but if you already know the driver family name ('nvme' if that is what esxcfg-scsidevs shows) then use:

              # vmkload_mod -s nvme | grep Version

               

              "Any suggestion on how to check firmware of the drives that cannot be seen?"

              As I was alluding to above, esxcli storage core device list should show the last 4 characters of firmware under 'Revision' - if they are not showing here then you are going to have to use out-of-band-management/BIOS to determine these - some 3rd party plug-ins can also tell these but this is below ESXi so ESXi doesn't have this capability unless something else is informing it.

               

               

              Bob

              • 4. Re: ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue
                mikev80 Lurker

                Ok, I'm able to verify the driver version with the HCL of the 2 devices that the system can see. I didn't suspect driver was the problem, but confirmed any way. 

                 

                Lets take the Intel P4500 capacity drives as an example.  I think my system thinks all the 10 drives are the same drives with 10 different adapter path to them.  Can anyone verify the below results is expected?

                 

                Results of esxcfg-scsidevs -l.  Note the "Other Names" section.

                 

                eui.0100000001000000e4d25c000073e014

                Device Type: Direct-Acess

                sixe: 3815447

                Display Name: Local NVMe Disk (eui.0100000001000000e4d25c000073e014)

                Multipath Plugn: NMP

                Console Device: /vmfs/devices/disks/eui.0100000001000000e4d25c000073e014

                Devfs Path: /vmfs/devices/disks/eui.0100000001000000e4d25c000073e014

                Vendor: NVMe  Model: INTEL SSDPE2KX04  Revision: QDV1

                Is RDM Capable: false

                Is Local: true Is SSD: true

                Other Names:

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                vml.010000000.....

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                t10.NVMe______INTEL SSDPE2KX04____<serial number>__00000001

                 

                 

                Results of esxcfg-scsdevs -u. It's a table that makes me think the system is assigning multiple paths to 1 NVMe drive.

                 

                Primary UID                                                       Other UID

                eui.0100000001000000e4d25c000073e014     vml.010000000.....

                eui.0100000001000000e4d25c000073e014     vml.010000000.....

                eui.0100000001000000e4d25c000073e014     vml.010000000.....

                eui.0100000001000000e4d25c000073e014     vml.010000000.....

                .

                .

                .

                .

                • 5. Re: ESXi 6.7U2 not seeing all the NVMe, only 2 of 12 - don't think it's a driver issue
                  depping Champion
                  VMware EmployeesUser Moderators

                  Can you post the details about the drivers? Intel recommended the async intel drivers usually for the P4500 and P4600 instead of the inbox.