VMware Cloud Community
ksf
Contributor
Contributor

Waiting on IPMI initialization on ESX 3.5 U3

Hi,

I have recently built two hosts (BL685c G1's) via an RDP solution using kick start scripts and the build has come down ok, in fact everything appears to be working fine, but on pressing Alt-F1 the following message is displayed...

hpIoctl: Waiting on IPMI initialization

If I ignore the message and try to logon it will overtype the message and allow me to logon. If I then exit the session it goes back to a logon prompt and the message is gone, but returns after a reboot?

The build does load the hpmgmt agent version 8.1.1, and that appears to have installed ok.

Has anyone seen this? Is it a problem?

I saw a thread where someone mentioned running 'chkconfig --level 0123456 ipmi off' to turn off this check - is that the answer?

Any help will be much appreciated and I will of course dish out the points ;o)

Tags (3)
Reply
0 Kudos
9 Replies
jshelly
Enthusiast
Enthusiast

Same issue here.

Any updates?

Reply
0 Kudos
ksf
Contributor
Contributor

I did log this with VMware and they advised that this is down to the HP Mgmnt agent. I tested this by removing this from the automated build and the messages did not appear, but did come back after the HP agent install, so I am now looking to log this with HP.

VMware did advise however that they thought the messages were cosmetic...?:|

When I get further information I'll post it.

Anyone else have any experience with this?

Kevin

Reply
0 Kudos
jerd
Contributor
Contributor

I opened a case with HP and here is their answer. I have not yet tested it.

*bq. HP ProLiant Servers - Interrupt Sharing in VMware 3.x Causes Performance Degradation
Issue
Interrupt sharing in VMware 3.x causes performance degradation.
While this has been noted to occur on some HP hardware, this issue is not limited to HP servers, and the behavior does not occur outside the VMware 3.x environment or with ESX3i and similar VMware implementations that do not use the console operating system for management.
Also, be aware that:

  • Interrupt sharing is a function of the combination of the hardware, ESX operating system, and agents or software (since some agents/software can consume an interrupt). The presence of interrupt sharing on an ESX host is not indicative of a bug; the ESX operating system is functioning as designed.

  • Not all interrupt sharing results in a noticeable performance penalty.*
    Solution
    *
    The following information documents changes are necessary to eliminate interrupt sharing on the VMware console operating system.
    *
    Overview:
    *
    There are four major changes to resolve the USB/IRQ sharing behavior:

  1. The first required change is to disable some USB support in the ESX operating system. The only driver that needs to be disabled is USB-uhci. This will eliminate interrupt sharing of devices associated with the USB controller but still allow physical USB ports to be available to the console operating system. The iLO's virtual media and keyboard will only be available outside of the console operating system (that is, during POST, or within the BIOS, or when booted to other virtual media, such as SmartStart or the Firmware Maintenance CD).

  2. Next, we will need to disable cpqci . The driver cpqci is the iLO information driver. The only impact of removing this will be that the iLO link will no longer be present on the system's management homepage for the management agents. The iLO web page will still be accessible through its specific IP address.

  3. As part of the process of disabling cpuci , it is necessary to configure the agents to utilize the native Open-IPMI rather than the HP-Open-IPMI. This requires that we change the main system management driver from hpasmxld to hpasmlitd . The HP-Open-IPMI driver will induce the same interrupt sharing issue as USB. By making this change, we eliminate interrupt sharing of devices associated with the HP-Open-IPMI driver. Additional configuration changes will allow the iLO to continue to capture data from the service console as before, but information may be delayed as the native Open-IPMI driver uses polling intervals while the HP-Open-IPMI driver uses interrupts when an event is logged. Also, some depth of error reporting may be impacted as the Open-IPMI driver will not include some HP-specific detail.

  4. Finally, in order to regain direct interaction with the console operating system through iLO, we can enable the virtual serial port and virtual serial console in the HP BIOS and within ESX. This is accomplished by a combined BIOS and operating system configuration change.*
    Procedures:
    **
    Modify HP Insight Manager agents to work with the native IPMI driver of the service console.
    *
    NOTE: The execution of the following steps is to be performed at the service console of an ESX 3.5 server running HP Insight Manager Agents 7.91 or later.
    In this example, we are using HP Insight Manager Agents 8.1.1. Perform the following commands at the command line of the console operating system.

  5. service hpasm stop

  6. vi /opt/compaq/cma.conf and add cpqci to the line beginning with the word exclude .

  7. Save the file.

  8. rpm -e hp-OpenIPMI-8.1.1-26.vmware30

  9. service hpasm start*
    Change from HP-Open-IPMI to Native-Open-IPMI.
    *
    1. Edit the file /opt/compaq/hpasmd/etc/hpasmd , and modify the line:
    PFLAGS="-f /dev/ipmi0"
    to appear as follows:
    PFLAGS="-f /dev/ipmi0 -d 4"
    NOTE: This is found under the line containing PNAME="hpasmlited" .
    2. Save the file.
    *
    Enable the virtual serial port and console.
    *+
    BIOS Configuration:
    +

  10. Open the ROM based setup (RBSU).

  11. Open the menu option System Options .

  12. Open the menu option Virtual Serial Port .

  13. Within the Virtual Serial Port settings, choose COM 1 .

  14. Then go back to the BIOS main menu.

  15. Select BIOS Serial Console and EMS . A new menu will appear.

  16. Select BIOS Serial Console Port .

  17. Select the serial port COM 1 .

  18. Then go back to the previous menu.

  19. Select BIOS Serial Console BAUD Rate .

  20. Select the speed 115200 .+
    Operating System Configuration:
    +
    1. Disable the UHCI driver.
    In /etc/modules.conf , add a # sign to the beginning of the line alias USB controller2 USB-uhci .
    2. Enable Service Console Kernel messages on COM 1 .
    Edit the /boot/grub/grub.conf file, adding the following to the end of the first (default) kernel line:
    console=ttyS0,115200 console=tty0
    3. Add the following line to the /etc/inittab file:
    co:2345:respawn:/sbin/agetty ttyS0 115200
    4. Add the serial console as a secure terminal by modifying the /etc/securetty file to include ttyS0 to the bottom.
    5. Reboot the server. When fully booted, a login prompt will be available on the virtual serial console of the iLO, and logins will be allowed.
    Please note that all documentation provided in this e-mail is copyrighted material of HP. It is provided solely to complement our service efforts and provide further information for your reference. The documentation is provided "as is" without any warranties or representations of any kind. In particular, HP does not warrant that the materials are accurate, complete and up-to-date, and HP specifically disclaims all other warranties and conditions, in fact or in law, express or implied, including the warranties and conditions of merchantable quality, merchantability, fitness for a particular purpose, or non-infringement of intellectual property of others.

*

Reply
0 Kudos
ksf
Contributor
Contributor

Hi,

Thanks for this, whilst I can see this relates to IPMI files/configuration does this specifically address this message? It seems to be referring to performance degradation? Is it implied that if this message appears that there will be performance degradation?

The solution seems very involved and I am not sure at this stage if it is necessary to do this.

Cheers

Kevin

Reply
0 Kudos
jerd
Contributor
Contributor

We haven't tested the process yet so I can't say if it will address the issue specifically. Hopefully, we will test it today (4/23) but everyone is really busy and squeezing out debug cycles is tough. I'll post the results as soon as we have tested.

Reply
0 Kudos
atc
Contributor
Contributor

This is what HP sent to me. They want me to reinstall since I didn't want to disable all of those monitoring tools. HP and VMware need to work together on these issues to get them resolved!

Does anyone understand what he is trying to say here?

FR: francisco_cantillo

Hello Jason,

This is regarding Case Number 3605775001

I have been cosulting with our next level.

" In reference to the ipmi issue....

hpasm checks to see if hp open ipmi is installed. If it is then hpasmxld will be used.

If not and the distributions open ipmi is installed, then it will hpasmlited witll be used.

Once hpasmxld is initialized it starts requesting information.

IPMI may not be fully initialized at this point so a message is generated indicating that it is waiting on ipmi to initialize.

Once ipmi is fully initialized then the message should cease. "

The solution would be reinstalling our HP agents, but it is not possible due to the servers are not loading the OS

Therefore, I advise you to reinstall the OS.

If you need more assistance, please let us know.

Thank you for your time,

Francisco Cantillo

Global Solution Center

Technology Solutions Group

Blades Server Support Eng 1.3

Hewlett-Packard Company

Reply
0 Kudos
ksf
Contributor
Contributor

Just a quick update, HP advised today and I quote...

"The message is harmless and we are working to ensure that the customer gets a complete answer to ensure that their concerns are addressed."

So even though this is not an answer, it does confirm that HP see this as cosmetic. Incidentally I also had this same feedback from VMware previously.

I will post more when I receive further updates.

K

Reply
0 Kudos
mkats
Contributor
Contributor

Hi, If you do not mind can you show me your grub.conf after you edited it.

Many Thanks

MK

Reply
0 Kudos
iops
Contributor
Contributor

I get exactly the same message - however, for me the affected ESX 3.5 host is not responsive to connections from VI Client, nor SSH. I am unable to login via the console.

VMs running on the host are still running ok (can RDP onto them fine).

Cosmetic is not really the right word for this! I can't manage my VMWare environment at all!

Duncan

Reply
0 Kudos