VMware Cloud Community
cykVM
Expert
Expert

HP Proliant DL380e Gen8, HP OEM VMWare ESXi 5.5 Update 2 keeps crashing (PSOD)

Hello everyone,

I maintain a single VMWare host running vSphere 5.5 (ESXi) Update 2 OEM HP version at the moment for a mid-size charity.

The hardware in use:

HP Proliant DL380e Gen8 (bought brand new in August 2014), HP SmartArray B320i storage controller, HP H222 host bus adapter (only a HP Ultrium4 tape drive connected to that), HP Intel 4port NIC 366i, 32GB RAM, 2 Quadcore Intel Xeon E5-2407

The box was initially installed and configured in August using HP OEM vSphere 5.5 Update 1 installation CD. vSphere is installed on the RAID array configured on the B320i controller. A VMWare Essentials license is also in use/installed.

It's running 3 Windows 2008 R2 VMs (DC, Exchange 2010 and a backup server with Backup Exec 2010 R3 [I know this is not a recommended/supported configuration, but it worked with 5.5 U1 without issues]) besides 2 Debian Linux VMs.

2 weeks ago during weekend maintenance I first installed the latest HP SPP (Service Pack for Proliant) Sept. 2014 which provided several firmware updates for e.g. the B320i, the 366i NIC etc.

After that I performed an upgrade instalölation of vSphere HP OEM 5.5 Update 2 version, which was also released by HP beginning of Sept..

All those setup/update procedures went through without any issues, error messages or crashes.

The host was running fine for 3 days and suddenly crashed with a PSOD stating: PCPU 0: no heartbeat (2/2 IPIs received) [unfortunately I did not take a screenshot]

I reset/rebooted the host through iLo4 console and kept an eye on the server the next days.

The first PSOD took place during daily (nightly) backup on the connected tape drive.

On the following Friday/Saturday night (about 2 days later) it crashed again with the following PSOD - again with PCPU 0: no heartbeat (2/2 IPIs received):

PSOD1.PNG

So I started investigating this, found some hints here in the VMWare communities leading to recommended BIOS settings of HP Proliant servers and checked the actual settings and changed the values to the recommended ones. The server was running fine without gliutches for about 16 hours then crashed again with this PSOD:

PSOD2.PNG

I continued investigation, and especially took an eye on power management setting in BIOS, vSphere and in the Windows VMs.

Also checked installed firnware versions of the storage controllers and NIC and driver versions in use. All OK there (as recommended in HP VMWare recipe Sept. 2014).

Server was running fine for about a week after the reboot then another PSOD early this morning at about 3 a.m.:

PSOD3.PNG

The server/VMs were mostly idle at this time, no heavy I/O activity.

The first two PSODs happened during backup but not at a certain time (one at about 10 p.m. the other early in the morning between 2 and 3 a.m.).

I read through tons of hints to faulty NIC drivers/firmware, BIOS confgurations etc. but nothing helps or even everything is configured exactly as in HP recommondations for vSphere 5.x.

For the BIOS settings I followed this list/table:Recommended BIOS Settings on HP ProLiant DL580 G7 for VMware vSphere | Boerlowie's Blog

vSphere is configured to "High Performance Mode" and the Windows VMs, too.

I'm somehow stuck now, so maybe someone here has a good hint for me?

If you need any further hardware/software/configuration/whatever details, just ask.

Cheers and thanks in advance for any help,

cykVM

122 Replies
rubensinfo
Contributor
Contributor

Hello guys

I'm using:

      -HP DL 360e Gen8

      -B320i

      -VMware OEM HP 5.5.0 build-2068190

      - driver update for B320i: 0.90

Is running at more than three weeks without PSOD but my oscilavar performance.

Then change the format of the vmdk to thick eager zeroed and yet is stable

Enclosed driver

Reply
0 Kudos
abelliot
Contributor
Contributor

Hi,

Veeam backup on a dedicated vdisk.

It 's also replicate VM on the datastore.

No PSOD yesterday...

Reply
0 Kudos
ff0000
Contributor
Contributor

I'm also having this issue with a PSOD indicating no heartbeat from a CPU on a DL 360 G8 dual 6-Core 32GB B320i system. Had to do a shift-R to get back to 5.1U1. I never got the machine to boot, it would just PSOD during the loading screen. I tried tweaking a few BIOS settings but didn't want to mess too much with it since the system is running just fine with 5.1U1. Has anyone found an actual fix for this issue or is it just wait for HP to fix their ISO?

Thanks

Reply
0 Kudos
cykVM
Expert
Expert

Hi,

I guess there will be no fixed 5.5 U2 ISO. What you can do is upgrade to 5.5 Update 1 (HP customized) which runs fine for me and other users in this discussion and in HP forums.

Maybe you can even patch the 5.5 U2 ISO with the -90 hpvsa driver.

cykVM

Reply
0 Kudos
ff0000
Contributor
Contributor

Ok good idea, but from what I understand I can't edit VMs using the free ESXi license with the vSphere Client until 5.5U2. If this is true, I can't go to 5.5U1 because I need to be able to edit my virtual machines using the standard vSphere Client (Not the web licensed vCenter-Only version). If this is the case I will have to stay on 5.1U1 until it's fixed.

KB: VMware KB: Editing virtual machine settings fails with the error: You cannot use the vSphere client ...

Thx

Reply
0 Kudos
cykVM
Expert
Expert

Are you running hardware version 10 VMs?

With the free license you should not upgrade to hardware version 10, stay at version 8 and everything is OK. You can edit settings with 5.5 U1 and vSphere (non-web) client without a problem. I do not run vCenter and web-client either, and with hw version 8 machines everything works fine.

hw version 10 should only be used if you have vCenter running.

That's also what the KB you referred to tells you. It's referring to the web-client and not the (stand-alone) vSphere client.

Reply
0 Kudos
ff0000
Contributor
Contributor

Ok great, good to know. They are all Version 8 VMs.  I will update to 5.5U1 for now then.

Thanks for the info!

Reply
0 Kudos
abelliot
Contributor
Contributor

Hi,

No PSOD after upgrade hpvsa driver to -90 but horrible disk performance !!!!

We'll revert back to 5.5 U1...

no news from HP ?

Reply
0 Kudos
cykVM
Expert
Expert

Hi,

I did not contact HP directly yet. I somehow have the feeling that there is some kind of dispute between HP and VMWare about (kernel) support for "not in VMWare HCL" hardware or the passthrough feature for devices in the background.

No real feedback in this discussion or in the one at HP forums from either HP or VMWare.

I will stick with 5.5 U1 which runs fine.

cykVM

Reply
0 Kudos
Aliceblue
Contributor
Contributor

Hi cykVM

By all means, please try.

1. Using  VMware-ESXi-5.5.0-Update2-2068190-HP-5.77.3-Nov2014.ISO  install

2. download

http://support.vmware.com/selfsupport/download/

ESXi550-201410001.zip  Product:ESXi (Embedded and Installable) 5.5.0   Download Size:656.5 MB

3. upload files esxi somewhore datastore (enable esxi ssh,shell)

esxcli software vib update --dry-run -d /vmfs/volumes/[some where datastore]/ESXi550-201410001.zip

VIBSs Intsalled:

>    tools-light   5.5.0-2.39.2143827
>    esx-base5.5.0-2.39.2143827
>    sata-ahci3.0-21vmw.550.2.39.2143827
>    misc-drivers  5.5.0-2.39.2143827

4. esxcli software vib update -d /vmfs/volumes/[some where datastore]/ESXi550-201410001.zip

Reply
0 Kudos
andreyka65
Contributor
Contributor

HI!

HP ProLiant ML 350e Gen8 (CPU E5-2407; RAM 32605,31 MB; B120i RAID Controller).

I have done everything in your instruction, and a month with the server, everything was in perfect order.

But today morning I found the server in PSOD, and now it does not boot, immediately falls in PSOD.

IMG_0907.JPGIMG_0908.JPG

Is it possible to somehow improve the situation?

Reply
0 Kudos
cykVM
Expert
Expert

andreyka65 schrieb:

HI!

HP ProLiant ML 350e Gen8 (CPU E5-2407; RAM 32605,31 MB; B120i RAID Controller).

I have done everything in your instruction, and a month with the server, everything was in perfect order.

But today morning I found the server in PSOD, and now it does not boot, immediately falls in PSOD.

[...]

Is it possible to somehow improve the situation?

The only way to improve the situation for me was to go back to 5.5 U1. I did not try the instructions Aliceblue posted above thoiugh, because I personally could not believe that this fixes the PSOD problem, but this is only my impression/opinion.

5.5 U1 runs fine since I went back to that version and no performance issues since then either.

Maybe a 5.5 U3 will be released at some point.

andreyka65
Contributor
Contributor

Rollback to version 5.5.0 U1, two days the server is running without PSOD.... Thanks for the tip!

Reply
0 Kudos
AlbertWT
Virtuoso
Virtuoso

Whoa.. I didn't know that VMware ESXi latest release causing so much problem lately.

I'm currently still on ESXi 5.1 Update 1 and still find it hard to move on to 5.5 Update 2

Let us know the solution then.

/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
cykVM
Expert
Expert

If you follow this discussion from my initial posting as of end of September 2014 you'll see that no suggestions really worked on 5.5 U2. Nearly everyone went back to 5.5 U1.

There is also some strange things going on with the customized HP VMWare 5.5 U2 ISOs. As it looks to me there were many support cases with the AMS (agentless management system) on HP VMWare 5.5 U2.

Version 5.75 of the customized ISO was released in December 2014 and the latest release 5.77 in November  :smileyconfused:

See VMWare KBhttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=208561...

and HP advisory HP Support document - HP Support Center

So maybe disabling/uninstalling AMS helps, but did not try that yet.

Reply
0 Kudos
sunonfire
Contributor
Contributor

I have installed HP customized 5.5 Update 2 from scratch on a CD. after 3 days my HP server had this problem as well.

Has anyone tried to use the new release ESXi 6.0?

Thanks,

Reply
0 Kudos
cykVM
Expert
Expert

I'm still on 5.5 Update 1 on that HP server. This runs fine without any PSODs since I went back to U1.

Plan to give 6.0 a try within the next months, but looks like there is no update to the hpvsa driver, 6.0 customized still uses that 5.5.0-90 hpvsa version.

Reply
0 Kudos
jawad
Contributor
Contributor

Have you tried this? iLO 4 2.10 solved issues on our installation.

HP Support document - HP Support Center

In the shadows...
cykVM
Expert
Expert

Thanks for you input, jawad. But this HP advisory addresses Gen9 servers and I could not find any update for my Gen8 server's ilo4 card to version 2.10 - in fact I could not find any update to 2.10 at all at the moment. I have 2.03 (10 Nov 2014) installed which is listed to be latest on all firmware download pages at HP.

Reply
0 Kudos
jawad
Contributor
Contributor

We upgraded our Gen8 with this update. Got it from HP support. Solved our problems with 5.5U2 on Gen8 servers.

In the shadows...