VMware Cloud Community
Dirtrunner
Contributor
Contributor

New ESXi 5.5 Install threw PSOD, Raid controller driver?

Can I get someone to glance at the PSOD I got on a new install of 5.5 ?

Installed this on Friday night and this Monday morning it was sitting at a purple screen. Ran fine all weekend as far as I can tell.

Its a DL380p G8 with both p420i and p420 Raid controllers

using HP-ESXi 5.5.0 iso 1331820

I think its yelling about the Raid controller but cant say for sure.

Looking at the vmkernal log im seeing this line over and over.

2014-03-09T13:08:07.636Z cpu2:286677)<4>hpsa 0000:02:00.0: out of memory at vmkdrivers/src_9/drivers/hpsa/hpsa.c:3562

2014-03-09T14:30:45.964Z cpu11:303182)<4>hpsa 0000:0a:00.0: cp 0x410a2b700000 has status 0x2 Sense: 0x5, ASC: 0x20, ASCQ: 0x0, Returning result: 0x2

I attached vmkwaring.log and vmkernal.log and a screenshot of the error.

Thank you guys!

42 Replies
klimenta
Contributor
Contributor

This is a temporary patch. It's not official. No idea when they'll release the official patch.

Reply
0 Kudos
tonis_o
Contributor
Contributor

Having similar issues. Has anyone used this temporary patch? Does it resolve the issue?

If not, I'll roll back to ESXi 5.0.0 U1. Can't tolerate this anymore in production. However, for us first lock-up was about 10 days after upgrade to 5.5.0.

Reply
0 Kudos
rabittom
Contributor
Contributor

i've installed the patch on all of our affected blades. so far no PSOD but i'm still waiting..

usually my uptime on these blades was about 7 days - i'm on day 6 now..

From an "inofficial" source inside HP i've got the information that HP will be ready with a fixed driver by June/10 (!!!!) - but this does not include the testing from VMWare...

Happy waiting!!!!

tonis_o
Contributor
Contributor

Well, replaced the scsi-hpsa-5.5.0.58-2OEM.550.0.0.1338720.x86_64.vib

with the scsi-hpsa-5.5.0.58-2OEM.550.0.0.1198611.x86_64.vib.

Beginning doesn't seem promising.. this is the vmkernel log after boot up.

vmkernel.png

I would say looking at the above vmkernel log that the 1198611 driver version also suffers from memory leak.

http://community.spiceworks.com/topic/485229-troubleshooting-vsphere-driver-memory-leak

Reply
0 Kudos
rabittom
Contributor
Contributor

Hello,

in case you guys have not recieved the information already - at least HP confirmed and described a kind of a workaround..

look at http://h20564.www2.hp.com/portal/site/hpsc/public/kb/docDisplay/?docId=c04302261

tom

Reply
0 Kudos
vincikb
Contributor
Contributor

I too have a ticket open with VMware and HP on this issue.  VMware told me to apply the non-GA scsi-hpsa-5.5.0.58-2OEM.550.0.0.1198611.x86_64.vib and that would solve my issue until it was officially released.  So I then opened an HP ticket and submitted all of my logs to see what they would have to say.  They told me just the opposite to instead downgrade my driver from 5.5.0.58-1OEM.550.0.0.1331820 to 5.5.0.50-1OEM.550.0.0.1198611 from here:

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-HP-HPSA-55050&productId=353

I have 6 new Gen 8 blades and two of them are purple screening so I downgraded the driver to see what happens. Then I just read this post about the HP advisory posted above and just sent HP that link to see if my issue is the same and if they knew about this advisory or not.  I really struggle with upgrading anymore hosts to 5.5 until this issue is resolved.

Reply
0 Kudos
romanaxbe
Contributor
Contributor

Are there any new messages from the driver update by HP?

Reply
0 Kudos
admin
Immortal
Immortal

Reply
0 Kudos
rabittom
Contributor
Contributor

Hi all,

after i've installed scsi-hpsa-5.5.0.58-2OEM.550.0.0.1198611.x86_64.vib on my affected hosts (G7 and Gen8) the system is stable now - no PSOD since that time (keep fingers crossed...)

tom

Reply
0 Kudos
romanaxbe
Contributor
Contributor

Hi tom

But thas not the older Version 5.5.0.50-1? Where can I find the (new??) version  scsi-hpsa-5.5.0.58-2OEM.550.0.0.1198611.x86_64.vib

Reply
0 Kudos
JPM300
Commander
Commander

If you dig through HP's drivers section for the controllers you can find the files

This is the link the 420i

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdHome/?sp4ts.oid=5295170&spf_p...

you have the option for the bundle or just the vim.  If you get the bundle and look inside the zip you will find the new scsi driver number the pervious user posted.  The site also gives install instructions to both.  You also have the option of using Vmware Update Manager if you want to install it that way as well.

P420

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdHome?javax.portlet.begCacheTo...

Hope this helps

Reply
0 Kudos
vincikb
Contributor
Contributor

JMP300 are you using a new scsi driver or are you simply following what the advisor says to do by downgrading the driver?

From the advisory: Until this has been corrected in a future version of the hpsa driver (targeted for mid-June 2014), the out-of-memory condition can be avoided by downgrading to

HP Smart Array Controller Driver (hpsa) Version 5.0.0-44.1 (ESXi 5.0 and ESXi 5.1) or Version 5.5.0.50-1 (ESXi 5.5).

Reply
0 Kudos
JPM300
Commander
Commander

Oh I'm not the one with the problem, the pervious post was stating they couldn't find the driver files, I was simply supplying the links for the person to point them into the right direction.  It will take a little more digging on the HP site to find the older versions of the drivers.  The link provided was the latest ones released by HP.  If these are wrong I applogise.  

Reply
0 Kudos
vincikb
Contributor
Contributor

Sure thing JPM300 that is why I asked.  Like I said I opened tickets with VMware and HP and HP said to downgrade back to scsi-hpsa-5.5.0.50-1OEM.550.0.0.1198611.x86_64.vib which is in the hpsa-5.5.0-1287942.zip driver from the HP website.  The problem like you pointed out is if you click on the hpsa it recommends you to update to the latest version which has the memory leak.  I told HP they need to remove this message so as not to confuse people.  Instead you click on that driver version, then click on the previous version in order to download the version I have listed above.

Reply
0 Kudos
lahteenj
Contributor
Contributor

Hi All,

Thanks' for the posts I hade the same issue.

HP has released a new driver version where they claim that they have fixed this memory leak issue, 5.5.0.60-1OEM.550.0.0.1331820

Here is a link to the driver.

navigationalState%3Didx%253D3%257CswItem%253DMTX_d18033ac346f468c92062ce127%257CswEnvOID%253D4166%25...

Jari

Reply
0 Kudos
bobarc
Contributor
Contributor

Anyone else run into an issue where Update Manager shows the update as Not Applicable?

Reply
0 Kudos
JHT_Seattle
Hot Shot
Hot Shot

I'm having the same issue here.  Found this post while searching for a way to work around it...

Reply
0 Kudos
bobarc
Contributor
Contributor

It crashed again last night so I went ahead and forced the install via the command line. esxcli software vib update -v /tmp/scsi-hpsa-5.5.0.60-1OEM.550.0.0.1331820.x86_64.vib

Reply
0 Kudos
romanaxbe
Contributor
Contributor

I have installed the driver via CLI, will report after few days about the result

Reply
0 Kudos
lahteenj
Contributor
Contributor

Hi,

I have installed the driver via cli on 6 servers and I have no longer errors in my logs. And the environment has been stable for a week.

Jari