PSOD Message:: LINT1/NMI (motherboard nonmaskable interrupt) HP ProLiant DL580 G7 (VMware ESXi, 6.0.0, 7967664)

[JohnsVCP5]

Hi Everyone,

Good day,

Could you please help on this.

Model HP ProLiant DL580 G7

VMware : VMware ESXi, 6.0.0, 7967664

We are facing PSOD issue on this Model and server is out of warranty and unable the find issue which hardware failure.

The below mentioned PSOD error message

PSOD Message:: LINT1/NMI (motherboard nonmaskable interrupt), diagnosed as fatal by module "hpe-nmi". This may be a hardware problem; please contact your hardware vendor.

   Backtrace for Current CPU: 0

     0x438080002c30:[0x41801f6782ea]PanicvPanicInt@vmkernel#nover+0x37e stack: 0x438080002cc8, 0x0, 0x1,

     0x438080002cc0:[0x41801f6785b5]Panic_NoSave@vmkernel#nover+0x4d stack: 0x438080002d20, 0x438080002c

     0x438080002d20:[0x41801f674b24]NMI_Interrupt@vmkernel#nover+0x0 stack: 0x0, 0x6800000000000000, 0x6

     0x438080002de0:[0x41801f674ccf]NMI_Interrupt@vmkernel#nover+0x1ab stack: 0x0, 0x0, 0x0, 0x41801f905

     0x438080002e90:[0x41801f65487a]IDTNMIWork@vmkernel#nover+0x10a stack: 0x0, 0x0, 0x0, 0x0, 0x0

     0x438080002f20:[0x41801f655e2d]Int2_NMI@vmkernel#nover+0x19 stack: 0x0, 0x41801f6c8067, 0x10b, 0x0,

     0x438080002f40:[0x41801f6c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x0, 0x0, 0x0, 0x41804000

     0x439243a1bb18:[0x41801f90588a]Power_HaltPCPU@vmkernel#nover+0x1ee stack: 0x417fdf883f20, 0x4180401

     0x439243a1bb68:[0x41801f812548]CpuSchedIdleLoopInt@vmkernel#nover+0x2f8 stack: 0xbf1264acd3d18, 0x1

     0x439243a1bbe8:[0x41801f815bee]CpuSchedDispatch@vmkernel#nover+0x15fe stack: 0x43935ce27100, 0x1, 0

     0x439243a1bd08:[0x41801f8167d4]CpuSchedWait@vmkernel#nover+0x240 stack: 0x0, 0x4314c6667251, 0x3401

     0x439243a1bd88:[0x41801f6b708a]WorldWaitInt@vmkernel#nover+0x28e stack: 0x418000002001, 0x4314c6660

     0x439243a1be08:[0x41801fbcd76a]UserObj_Poll@<None>#<None>+0x106 stack: 0xcc6684000, 0xbf1264ebd1752

     0x439243a1be78:[0x41801fbf2d5e]LinuxFileDesc_Ppoll@<None>#<None>+0x262 stack: 0x3ffec4cb9f8, 0x4314

     0x439243a1bef8:[0x41801fbc77fa]User_LinuxSyscallHandler@<None>#<None>+0x26e stack: 0x0, 0x0, 0x0, 0

     0x439243a1bf28:[0x41801f68ed11]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0x10b, 0x0, 0x0,

     0x439243a1bf38:[0x41801f6c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x10f, 0x2ee286b8, 0x3ffe

I have checked the ILO : Integrated Management Log found the below error log

Severity	Class	Count	Description
Critical	PCI   Bus	1	Uncorrectable   PCI Express Error (Embedded device, Bus 0, Device 3, Function 0, Error status   0x00040000)

I have login to SSH and execute the lspci -v command and found the bus details but unable to find out cause with motherboard or PCI SLOT or Any NIC card /HBA card issues / firmware/driver issues

0000:00:03.0 PCI bridge Bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 [PCIe RP[0000:00:03.0]]

         Class 0604: 8086:340a

​Thanks Advance

Regards,

Johnson.s

[SupreetK]

Please check the below HPE advisories -

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04912076

HPE Support document - HPE Support Center​

Cheers,

Supreet

[JohnsVCP5]

The HP article given for Gen8 server Modal for My server is HP ProLiant DL580 G7

dvisory: (Revision) VMware - HP ProLiant ML/BL/DL Gen8 Servers May Experience Purple Screen Of Death (PSOD): LINT1 Motherboard Interrupt

[SupreetK]

Please check the below KB as well. Even though all these HPE advisories say it is applicable only for Gen8, we have observed it on Gen7 servers as well.

Request you to involve HPE support in case you want a confirmation -

HPE Support document - HPE Support Center​

Cheers,

Supreet

[JohnsVCP5]

Advisory: (Revision) - HP Integrated Lights-Out 4 - FIRMWARE UPDATE REQUIRED: Intermittent Non-Maskable Interrupt (NMI) Events May Occur on ProLiant Gen8 Servers with HP Integrated Lights-Out 4 Firmware Versions 1.30, 1.32, 1.40 and 1.50

We have referred most of the HP Article but nothing matching All referred Gen8 & above My ILO version 1.88 Light-Out 3

[SupreetK]

Although the articles talk only about Gen8, we have seen similar issues on Gen7 as well. Please involve HPE support in case you want a confirmation.

Cheers,

Supreet

[JohnsVCP5]

Server is out of warranty

[SupreetK]

Yep, then we have to follow what we have in hand

Cheers,

Supreet

[JohnsVCP5]

Yep, I have replaced the mother board

[devakumar]

Is the PSOD occuring even with VT-d interrupt remapper enabled?

[ithereal]

Removing the hpe-nmi vib solved the issue for me. As far as I know this vib is only used to send a NMI event (non-maskable interrupt) from within the ILO to the host to simulate a PSOD. I have chosen this feature is not needed and removed the vib by the following command:

esxcli software vib remove -n hpe-nmi