Lint1 ALERT HP DL580 G5

[wonhee]

Hello. I am WonHee.I recently have installed VMware ESX Server 3.5 on HP DL580 G5. There are 6 ESX hosts and All hosts are HP DL580G5 Machine.At the begining of this project there were 3(Host No.1,2,4) hosts that were showing Lint1 ALERT Message. The 3 hosts' mainboards had been replaced and one of the host(Host No.4) showed Lint1 ALERT message again and that host's motherboard has replaced again.Now Host No.2,3,5 show the Lint1 ALERT Message.HP Diagnostics tool shows nothing wrong but VMware Host Console shows the ALERT.VMware SR replied "It is HW issue not VMware issue, check your HW!".As you all know ALERT messages it shows messages in RED on Console so customer really being bugged by this message.Following Messages are the ALERT messages.vmkernel: 6:13:39:08.915 cpu0:1024)ALERT: APIC: 1266: Lint1 interrupt on pcpu 0 (port x61 contains 0xb0)Anyone have same message and Solve this issue?

[kjb007]

Run your hardware check utility for an extended period of time, 24-48 hours. You may have just received bad motherboards, or there is something that is frying the systems in your datacenter. Are there heat issues with your servers? Are you using the same hardware, after the mb is replaced? Maybe it's a bad CPU instead.

-KjB

[Texiwill]

Hello,

As kjb007 stated you need to check your hardware.....

First check the BIOS for the replaced parts is up to the proper level. Then check the BIOS settings are what they should be. You need to enable Full Table APIC for example. HP has this information for ESX.

THen run hardware diags for 24-48 hours. Then run memtest86 for 24-48 hours. It could be the banks on the MB, the CPUs, BIOS settings, etc.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[wonhee]

Thank you all. I really appreciate sharing your knowledge.To me your answers like a cup of water after wandering the huge vast desert of error.I almost give in being a VMware Certified Professional.Here are some more information about this issue. Memtest86 has been ran for 3 or 4 days on all hosts. I would say all has been tested and qualified.1. Has any of you seen these issues practically?It is almost impossible testing hardware for 24-48 hours long cause usage of all hosts shows 90% of cpu and 86% of Memory.There was one host that had HP agent for VMware installed and iLO setting had done, but none shows any messages at the time Lint1 ALERT showed.I have recommended 8 ESX hosts but customer mentioned about money, as an engineer of vmware, money issue is always bugging as others.2. I have another customer who is using DL580 G5 (3 hosts) same model, I also ran Memtest86 almost a week and there are issues.Memtest86 shows "unexpected interrupt Error" after 1 or 2 passes of Memtest86.I have insisted reorganizing memory (changing modules from DIMM to DIMM randomly) of all three hosts.Then 2 hosts passed and one host still showed "unpected interrrupt error" so I insisted replacing motherboard and today the MB has been replaced.Now memtest86 test is processing I have to check it that machine tomorrow. Fixed or not Fixed that is problem.As A VCP I could not allow customer using unqualified hardware that would be some gigantic problem in near future.3. Since I can not use VMware SR as reference on this issue, would you give me your qualification about yourself?My words on this case is powerless, customer would not want to listen.

Thank you.

[Texiwill · Accepted solution]

Hello,

1. Has any of you seen these issues practically?

Yes I have. You need to verify the hardware by running diags. You also MUST check the BIOS settings/revisions. That will take powering off the hardware to do so. If the BIOS is setup incorrectly or the BIOS not at the proper revision you will see such errors.

It is almost impossible testing hardware for 24-48 hours long cause usage of all hosts shows 90% of cpu and 86% of Memory.

I hear this quite a bit, but if the BIOS is improperly configured, not at the proper revision, or hardware not fully vetted, then you could have more serious issues down the line.

There was one host that had HP agent for VMware installed and iLO setting had done, but none shows any messages at the time Lint1 ALERT showed.

HPASM/HPSMH do not catch all issues.

2. I have another customer who is using DL580 G5 (3 hosts) same model, I also ran Memtest86 almost a week and there are issues.

Memtest86 shows "unexpected interrupt Error" after 1 or 2 passes of Memtest86.

I have insisted reorganizing memory (changing modules from DIMM to DIMM randomly) of all three hosts.

Then 2 hosts passed and one host still showed "unpected interrrupt error" so I insisted replacing motherboard and today the MB has been replaced.

Now memtest86 test is processing I have to check it that machine tomorrow. Fixed or not Fixed that is problem.

As A VCP I could not allow customer using unqualified hardware that would be some gigantic problem in near future.

If memtest86 shows any errors then I would once more verify the BIOS settings and revisions first. Then rerun the tests. I would also consider replacing the offending memory chips immediately. However once more the BIOS can affect how memtest86 reports.....

3. Since I can not use VMware SR as reference on this issue, would you give me your qualification about yourself?

You should open a support case about this.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[bse1969]

I ran into these errors a while back. Have you updated the HP agent to the latest versions? Might also want to update3 BIOS/Firmware as well. My errors went away as soon as I updated the agents.

[wonhee]

BIOS Version of this HP Hosts 12/27/2007. At first time Lint1 ALERT showed up, there was none host that HP agent installed. After the first Lint1 ALERT insident, Customer want to have HP agent and that task has done. HP Agent Version 8.0.0 for ESX 3.5

[wonhee]

Conclusion, AFTER ALL THIS YEARS, it was about Firmware Bug on HP Hardware. I Personally and Strongly Not Recommend HP Hardware At all on Virtualization Project.If you are in the situation you can not control then you have to use HP Hardware, I tell you this. Embrace yourself.