VMware Cloud Community
TRottig
Enthusiast
Enthusiast
‎04-16-2019 07:59 AM

PMEM not detected

Hi,

I have an Intel S2600 STB with a 16GB Micron NVDimm module installed which does not get detected as nvdimm, only as regular memory:/

I have 4x8gb + 1x16GB nvdimm installed at,

pastedImage_0.png

I see the following in dmesg:

2019-04-16T14:52:25.245Z cpu26:2097953)Activating Jumpstart plugin psa-mask-paths.

2019-04-16T14:52:25.246Z cpu26:2097953)Jumpstart plugin psa-mask-paths activated.

2019-04-16T14:52:25.246Z cpu26:2097953)Activating Jumpstart plugin network-uplink-init.

2019-04-16T14:52:25.260Z cpu39:2098147)Device: 192: Registered driver 'uplink_drv' from 0

2019-04-16T14:52:25.292Z cpu44:2098148)Loading module intelnvdimm ...

2019-04-16T14:52:25.292Z cpu44:2098148)Elf: 2101: module intelnvdimm has license VMware

2019-04-16T14:52:25.292Z cpu44:2098148)Device: 192: Registered driver 'IntelNVDimm' from 43

2019-04-16T14:52:25.292Z cpu44:2098148)Mod: 4962: Initialization of intelnvdimm succeeded with module ID 43.

2019-04-16T14:52:25.292Z cpu44:2098148)intelnvdimm loaded successfully.

2019-04-16T14:52:25.293Z cpu67:2098108)WARNING: NVD: 1518: NVDIMM (handle 0x1010, idString 802c-0f-1649-14f2a4b0): Previous ARM operation failed.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 306: NVDIMM 0x1010: _NCH method failed, status Not supported

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 1 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 1348: NVDIMM 0x1010: GetHealthInfo failed, status Not implemented

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 4 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 1628: NVDIMM 0x1010: Failed to get LSA data, status Not implemented

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 7 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 718: NVDIMM 0x1010: GetCELSize failed, status Not implemented

2019-04-16T14:52:25.293Z cpu67:2098108)IntelNVDimm: IntelNVDimmAttachDev:80: Attached to NVDIMM device 0x1010

2019-04-16T14:52:25.293Z cpu67:2098108)Device: 327: Found driver IntelNVDimm for device 0x7ca5430547ff99f7

2019-04-16T14:52:25.300Z cpu67:2098108)PCI: 91: Device 0x481b430547ff9bd1 is not a PCI vmkDevice

2019-04-16T14:52:25.300Z cpu67:2098108)Device: 2628: Module 0 did not claim device 0x481b430547ff9bd1.

2019-04-16T14:52:25.303Z cpu67:2098108)PCI: 91: Device 0x5612430547ff9d66 is not a PCI vmkDevice

2019-04-16T14:52:25.303Z cpu67:2098108)Device: 2628: Module 0 did not claim device 0x5612430547ff9d66.

2019-04-16T14:52:25.305Z cpu67:2098108)PCI: 91: Device 0x4f60430547ff9f68 is not a PCI vmkDevice

2019-04-16T14:52:25.305Z cpu0:2097412)i40en: indrv_UplinkAssociate:1027: Associate i40en driver with vmnic0

2019-04-16T14:52:25.305Z cpu67:2098108)Uplink: 11671: Device vmnic0 not yet opened

2019-04-16T14:52:25.305Z cpu67:2098108)Device: 327: Found driver uplink_drv for device 0x4f60430547ff9f68

2019-04-16T14:52:25.305Z cpu67:2098108)Uplink: 13950: Opening device vmnic0

2019-04-16T14:52:25.305Z cpu67:2098108)Uplink: 11689: enabled port 0x2 with mac a4:bf:01:05:9e:9a

2019-04-16T14:52:25.305Z cpu0:2097412)i40en: indrv_UplinkStartIo:1469: Starting I/O on vmnic0

2019-04-16T14:52:25.307Z cpu0:2097412)i40en: i40en_Up:9103: Set Up

2019-04-16T14:52:25.311Z cpu67:2098108)PCI: 91: Device 0x1e92430547ffa186 is not a PCI vmkDevice

2019-04-16T14:52:25.311Z cpu0:2097412)i40en: indrv_UplinkAssociate:1027: Associate i40en driver with vmnic1

2019-04-16T14:52:25.311Z cpu67:2098108)Uplink: 11671: Device vmnic1 not yet opened

Any idea why it might not work?

Thanks

Tags (1)
0 Kudos
5 Replies
TRottig
Enthusiast
Enthusiast
‎01-07-2020 11:34 PM

Still have not been able to resolve this, running U3 now, no change.

PMems work fine in other OSs (tested Linux, Win, FreeBSD), just ESXi has issues.

I had assumed that vendor modules might expose more functionality that might be needed to detect/ready them properly (no implemented msgs),

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 1 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 1348: NVDIMM 0x1010: GetHealthInfo failed, status Not implemented

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 4 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 1628: NVDIMM 0x1010: Failed to get LSA data, status Not implemented

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 213: ACPI device 0x430072f85960: Query for function 7 status Not implemented.

2019-04-16T14:52:25.293Z cpu67:2098108)NVD: 718: NVDIMM 0x1010: GetCELSize failed, status Not implemented

but I have since learned that Dell servers for instance use the same NVDimms (Micron's, same model#) so that leaves the BIOS of the board.

I tried searching the compatibility list but couldnt find NVDimms at all in there ...

0 Kudos
TRottig
Enthusiast
Enthusiast
‎05-03-2020 01:03 PM

Just installed 7.0 - still not working

It does seem to detect them but something seems to be off...

Still some "unsupported" features. I know these are the same modules that work fine in Dell servers - so i must assume its something that vSphere needs the BIOS does not expose on this Intel board ... but that Linux/FreeBSD/WIndows don't need...

No ideas?

Thx

dmesg  |grep -i pmem
VMB: 83: fakePMemPct = 0x0
VMB: 87: injectPMemSratEntries = FALSE
TSC: 548950 cpu0:1)BootConfig: 711: poisonPmem = TRUE (1)
TSC: 732152 cpu0:1)BootConfig: 711: fakePMemPct = 0 (0)
TSC: 737106 cpu0:1)BootConfig: 711: autoSetupLocalPMem = 1 (1)
TSC: 740474 cpu0:1)BootConfig: 711: injectPMemSratEntries = FALSE (0)
2020-05-03T20:45:02.202Z cpu0:1048576)Initializing InitVMKernel: PMem_Init ...
2020-05-03T20:45:02.202Z cpu0:1048576)FDS: 441: Registered FDS Driver PMemNamespaces
2020-05-03T20:45:02.202Z cpu0:1048576)FDS: 441: Registered FDS Driver PMemVolumes
2020-05-03T20:45:02.202Z cpu0:1048576)FDS: 441: Registered FDS Driver PMemDS
2020-05-03T20:45:02.202Z cpu0:1048576)FDS: 441: Registered FDS Driver PMemDisk
2020-05-03T20:45:02.202Z cpu0:1048576)SysInitTable: 69: Finished sysInit step: PMem_Init in 816 us.
2020-05-03T20:45:02.211Z cpu0:1048576)Initializing InitVMKernel: PMemPoison_Init ...
2020-05-03T20:45:02.212Z cpu0:1048576)SysInitTable: 69: Finished sysInit step: PMemPoison_Init in 521 us.
2020-05-03T20:45:35.873Z cpu51:1049433)Activating Jumpstart plugin auto-configure-pmem.
2020-05-03T20:45:36.352Z cpu51:1049433)Jumpstart plugin auto-configure-pmem activated.
[root@localhost:~] dmesg  |grep -i NVD
TSC: 726576 cpu0:1)BootConfig: 711: nvdArsBootWaitMaxTimeout = 80 (80)
TSC: 729528 cpu0:1)BootConfig: 711: nvdArsErrorInject = 0 (0)
TSC: 743336 cpu0:1)BootConfig: 711: injectNVDIMMState = 0 (0)
TSC: 748844 cpu0:1)BootConfig: 711: allowDsmForNVDIMMHealth = TRUE (1)
0:00:00:05.826 cpu0:1048576)VMKAcpi: 207: \_SB_.NVDR: found ACPI quirk for pnpID ACPI0012 ignore 0 flags 0x2
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N000: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N001: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N010: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N011: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N020: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N021: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N030: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N031: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N040: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N041: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N050: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N051: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N100: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N101: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N110: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N111: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N120: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N121: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N130: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N131: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N140: found ACPI PNP Device.
0:00:00:05.826 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N141: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N150: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N151: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N200: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N201: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N210: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N211: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N220: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N221: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N230: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N231: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N240: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N241: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N250: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N251: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N300: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N301: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N310: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N311: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N320: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N321: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N330: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N331: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N340: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N341: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N350: found ACPI PNP Device.
0:00:00:05.827 cpu0:1048576)VMKAcpi: 241: \_SB_.NVDR.N351: found ACPI PNP Device.
0:00:00:05.837 cpu0:1048576)Device: 1541: Registered device: 0x430a09a014c0 \_SB_.NVDR h08ACPI0012 (parent=0x4a2f430a09a0179d)
0:00:00:05.837 cpu0:1048576)VMKAcpi: 676: \_SB_.NVDR: registered ACPI PNP Device (h08ACPI0012).
0:00:00:05.845 cpu0:1048576)Device: 192: Registered driver 'nvdroot' from 0
2020-05-03T20:45:02.206Z cpu0:1048576)Initializing InitVMKernel: NVDVsi_Init ...
2020-05-03T20:45:02.207Z cpu0:1048576)SysInitTable: 69: Finished sysInit step: NVDVsi_Init in 533 us.
2020-05-03T20:45:13.454Z cpu21:1049806)NVD: 501: NVD Root device 0x43006ea2ba40 ACPI0012 found, NFIT ACPI 6.0
2020-05-03T20:45:13.454Z cpu21:1049806)NVD: 183: SPARDT[0]:
2020-05-03T20:45:13.454Z cpu21:1049806)NVD: 201: MDSARMT[0]:
2020-05-03T20:45:13.454Z cpu21:1049806)NVD: 379: NVDIMM 0x100: NVDIMM Representation string 802c-0f-1649-14f2a4b9
2020-05-03T20:45:13.454Z cpu21:1049806)NVD: 227: SPA Range[1]: type 1
2020-05-03T20:45:13.454Z cpu21:1049806)WARNING: NVD: 174: BIOS bug: SPA Range[1]: Range 0x67c000 - 0xa7bfff is not reported with Non-Volatile flag set in ACPI SRAT.
2020-05-03T20:45:13.455Z cpu21:1049806)NVD: 1042: Query ARS capabilities failed. status:Not supported
2020-05-03T20:45:13.455Z cpu21:1049806)Device: 327: Found driver nvdroot for device 0x2c7e430a09a021f6
2020-05-03T20:45:13.455Z cpu21:1049806)Device: 1541: Registered device: 0x430a09a014c0 \_SB_.NVDR.N030 n182c804e323480413131010101 (parent=0x2c7e430a09a021f6)
2020-05-03T20:45:13.455Z cpu21:1049806)VMKAcpi: 676: \_SB_.NVDR.N030: registered ACPI PNP Device (n182c804e323480413131010101).
2020-05-03T20:45:18.044Z cpu57:1049992)Loading module intelnvdimm ...
2020-05-03T20:45:18.045Z cpu57:1049992)Elf: 2048: module intelnvdimm has license VMware
2020-05-03T20:45:18.045Z cpu57:1049992)Device: 192: Registered driver 'IntelNVDimm' from 34
2020-05-03T20:45:18.045Z cpu57:1049992)Mod: 4997: Initialization of intelnvdimm succeeded with module ID 34.
2020-05-03T20:45:18.045Z cpu57:1049992)intelnvdimm loaded successfully.
2020-05-03T20:45:18.046Z cpu57:1049992)WARNING: NVD: 1522: NVDIMM (handle 0x100, idString 802c-0f-1649-14f2a4b9): Previous ARM operation failed.
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 1 status Not implemented.
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 1352: NVDIMM 0x100: GetHealthInfo failed, status Not implemented
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 4 status Not implemented.
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 1632: NVDIMM 0x100: Failed to get LSA data, status Not implemented
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 7 status Not implemented.
2020-05-03T20:45:18.046Z cpu57:1049992)NVD: 722: NVDIMM 0x100: GetCELSize failed, status Not implemented
2020-05-03T20:45:18.046Z cpu57:1049992)IntelNVDimm: IntelNVDimmAttachDev:80: Attached to NVDIMM device 0x100
2020-05-03T20:45:18.046Z cpu57:1049992)Device: 327: Found driver IntelNVDimm for device 0x2a64430a09a211f3
2020-05-03T20:47:40.190Z cpu41:1052149 opID=2ab34240)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 1 status Not implemented.
2020-05-03T20:47:40.190Z cpu41:1052149 opID=2ab34240)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 1 status Not implemented.
2020-05-03T20:47:40.190Z cpu41:1052149 opID=2ab34240)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 1 status Not implemented.
0 Kudos
depping
Leadership
Leadership
‎05-04-2020 12:58 AM

Does the Intel board have a setting around Persistent Memory at all? As that is what you would do with Dell normally. You enable Persistent Memory through the Bios if you don't want to use it as regular memory.

0 Kudos
depping
Leadership
Leadership
‎05-04-2020 02:32 AM

The Dell and HPE hosts do have "persistent memory" listed as a supported feature (VMware Compatibility Guide - System Search ) by the way, the Intel S2600 doesn't appear to have that on the HCL: VMware Compatibility Guide - System Search

0 Kudos
TRottig
Enthusiast
Enthusiast
‎11-28-2020 04:13 AM

Hi @depping 

sorry, did not get/see a notification and have not checked here in a long time.

 

I have since tested the same modules on several Supermicro boards (eg X11SPH-nCTPF) and all show the same behaviour; the NVDIMM (-N, not Optane Memory) is working fine on all of them using Windows/Linux/FreeBSD.

They are not working on ESXi (up to 7.0, have not tested since) at all despite being 'seen' by the boot process.

It seems to me as if Dell/HP might expose additional capabilities (via Bios or NVDimm FW) that a regular Micron NVDimm does not (properly) expose; so ESXi expects something but does not seem to see the expected results

2020-05-03T20:47:40.190Z cpu41:1052149 opID=2ab34240)NVD: 213: ACPI device 0x43006ea39fc0: Query for function 1 status Not implemented.

The question is what it expects and why it absolutely needs that... The documentation on this is basically non existent, so its quite difficult to research further.

Unfortunately I dont have a Dell/HP system to flash the modules to the respective FW to give it a try.

 

Cheers

0 Kudos