Boot issues with ESX 3.51 (and ESX3.5) on IBM LS21 using QLA4022 and EqualLogic San

[dmarshallx]

We installed ESX 3.5.1 on a new IBM LS-21 blade on an IBM BladeCenter H. The LS-21 is running

BIOS 1.05

Qlogic add-on card with BIOS 1.09a and Firmware 2.00.00.62

32GB of RAM

Two 3.0GHz AMD dual-core processors

The LS-21 blade boots from our EqualLogic SAN. During booting of this system when the CD or OS boots when it reaches the “loading qla4022…” message on the console, the system takes 16-22 minutes to continue. We saw the same problem loading ESX 3.5 and decided to try 3.5.1 after reviewing the release notes. However this does not seem to have improved the situation.

All of our other LS-21 systems are running ESX 3.0.2 from the same EqualLogic SAN and they do not exhibit this behavior.

We’ve monitored the activity on the EqualLogic SAN console. During the boot, the ESX 3.5.1 system repeats a pattern of logins and resets.

The pattern:

- Server logons to the SAN

- Server stays connected for 2 minutes, no bytes are reported read or written

- SAN Session is reset

This pattern repeats within seconds of the Session reset 5 or 6 times before the system stays connected and begins loading the remainder of the OS.

We took a sniffer network trace of this activity of this activity. The bios load completes around packet 150, the qla4022 loading message is about packet 3150 and the os continues loading around packet 5300, by packet 61000 the OS is loaded and up.

The Qlogic card is configured with:

Jumbo Frames

Manual IP (no gateway)

Header and Data digest

Target IP address and strings set

Is this a known issue? If so, what is the resolution.

[adehart]

Out of a batch of 10 QLE4062C cards, I had 3 bad ones. In a few instances, it really wreaked havoc with a number of the tests I did because I was assuming I actually had good cards.

[i2ambler]

Ive tried 4 out of the 9 cards I have and they have all done the same thing in EXS and trying to load fedora 9.. Im sorta stumped... I guess I will have to call DELL to see what they have to say about it.

[adehart]

It just dawned on me when you said Dell you were talking about the EqualLogic boxes weren't you? (I guess I need to pay attention as it does say IBM LS21... sorry...)

[i2ambler]

No biggie.. i just hijacked this thread.. I did try just putting 1 card in, and a hard drive into one of the 2950s.. I set the card up with an IP and attached to a lun. Fedora loaded the qla driver, but didnt see the disk from the install screen.. IM going to see if i can at least see the disk when i am in the OS once it installs to the internal HD.. Its like a merrygo round that i cant get off! hah. Yes, i have an equallogic iscsi san with 3750 switches and qla 4060 cards in 2950 servers

[i2ambler]

Well Ive gotten the kiss-off from dell and qlogic.. I guess i have to call vmware when we get the licenses.. boo. I wouldnt think this would be so difficult.

[i2ambler]

Update: Called VMware support.. they are stumped also.. Id love to install the older firmware - but I cant seem to find it anywhere..

[i2ambler]

Well I just wanted to update this thread with my progress.. hah. This is becoming like a blog or something.. I spent the entire day on the phone with various support persons... These cards are the IBM branded cards "pcie hba for ibm system x" I managed to get ahold of the older firmware, so I threw one of the cards into a windows box to install it. Went to install the STOR driver, and it failed to intialize. Nice.. I called Qlogic - and they couldnt get it to work, and when they found out they were IBM OEM cards, they told me to pound sand. I called IBM, they told me to pound sand right away. So I called Dell back.. he wasnt sure what to do about it either.. doh. Im thinking maybe I got an entire batch of 8 cards that are bad? Maybe the IBM cards are just not compatable with Dell systems at all.. Right now Im out of ideas, fustrated, and am going to let our vendor handle this issue. Lets hope i can get a new batch of cards that work!

Again, the system specs are : Dell 2950, Qlogic 4060c (ibm oem), equallogic san, 3750 switches. San works great when connecting via software initiator. Wish me luck!

[Mazzer]

Once i had problem with Jumbo Frames because one of switchs didnt support that.

If u dont resolve, try to use normal pakages.

Att,

[AlexNG_]

I'm not finding it neither!!!

I'll call our storage guy, maybe they still have it.

AlexNG

i2ambler <communities-emailer@vmware.com>

12/06/2008 20:10

Para

<alex.nieva@morse.com>

cc

Asunto

New message: "Boot issues with ESX 3.51 (and ESX3.5) on IBM LS21 using QLA4022 and EqualLogic San"

,

A new message was posted in the thread "Boot issues with ESX 3.51 (and ESX3.5) on IBM LS21 using QLA4022 and EqualLogic San":

http://communities.vmware.com/message/970098

Author : i2ambler

Profile : http://communities.vmware.com/people/i2ambler

Message:

[AlexNG_]

Hi all,

We're lucky, our st guys have just gave me the firmware 3.0.1.27 (and the 1.13 BIOS).

Please be advice, this rar file is provided as is, downloaded directly from IBM web site, from the link I posted yesterday, before they changed. Downgrade or upgrade at your own risk. In our case it was succesfull and our customer has two ESX 3.5 Update 1 servers up and running.

We flasehd the bios and downgraded fw using a win98 bootable cd.

Regards,

AlexNG

Message was edited by: AlexNG_ ... Forgot the file!

[i2ambler]

Not sure how you were able to flash a 4060c.. when I go to flash the firmware it tells me that there is no qlogic card on the system... did you use iflash.exe?

[BenConrad]

Hi, I know you probably don’t want to hear this but I'd suggest you don't boot-from-san with Qlogic iSCSI cards.

See this thread for the reason:

http://communities.vmware.com/message/780294

In addition to this thread I've also had all the settings applied in the thread blown away while installing ESX patches...

With that said, maybe ESX U1 handles this situation better, for us, it's not worth the risk.

Ben

[dmarshallx]

We know the system boots with ESX 3.0.2 without issue. The problem happens with ESX 3.5 and ESX 3.5.1

The LS21 was shipped with the Qlogic firmware 2.00.00.45 and it exhibited this same problem, we upgraded the firmware to version 2.00.00.62 is the latest from the Qlogic site for this specific card.

We boot ESX directly from the SAN and that works extremely well and very fast. The problem comes when ESX 3.5 loads its "qla4022.o" driver to talk with iSCSI. At that point, we get the 13-minute delay.

We've turned off Jumbo Frames and digest with no affect.

We believe that the problem is with the qla4022.o driver within ESX.

[i2ambler]

I was just told that the IBM branded qlogic card is not supported by vmware... So... Not sure what to do about that one, I guess we will be getting new cards

[BenConrad]

Boot-from-san worked well for us too, until we started having path failover events take down the COS.

We also see our QLA4050C/QLA4052C cards 'reset' and drop connections at times of heavy load, that took down the COS as well.

Ben

[adehart]

I can assure you that I am 100% certain these cards are supported by VMWare and are on the HCL.

I got into a pissing match with them because they don't have part#'s listed on their HCL just descriptions for IBM parts but the QLOGIC flavor of the card is identical. However, at the time, they didn't have the C on the end of the QLE4060C and QLE4062C cards shown so they said the card wasn't the correct card and wouldn't be supported. It took me a lot of screwing around and talking to tech managers to get this resolved. QLOGIC doesn't make a plain 406x, it has to have the C on the end so it was a typo and the support people were just being unreasonable.

Reference the SR# I gave earlier, they have my exact setup (or did) replicated in their engineering group with IBM 4062C or 4060C adapters and x3650 servers. I seriously doubt they would have gone through all the trouble if the card wasn't supported.

Just so we are on the same page I think the IBM part#'s are:

QLogic iSCSI Single-Port PCIe HBA for IBM System x QLE4060C (Option PN 39Y6146, FRU PN 39Y6148)

QLogic iSCSI Dual-Port PCIe HBA for IBM System x QLE4062C (Option PN 42C1770, FRU PN 42C1772)

[AlexNG_]

Hi all,

@i2amber: in our case, we are not booting from SAN, and but up just delays a bit while loading the 4022 driver... We used iflash... I do not know why it says that no qlogic found.... in our case it was qla4062c, and as storage guys told me, it's the same firmware (did not checked that personally).

We are using the following card:

QLogic iSCSI Dual-Port PCIe HBA for IBM System x QLE4062C (Option PN 42C1770, FRU PN 42C1772)

I just can say that if the firmware I posted do not work, we only have the SRs. The more SR we open, the quicker we'll get a fix...

AlexNG

[AlexNG_]

@i2amber,

When we flashed the card, we used the command as follows:

$ iflash.exe /FB

......... Flashed the BIOS ..........

iflash_app&gt; FF

.......... Flasehed the Firmware .....

we then used the options VB, VF and C, to check bios and firmware levels and, checked Flash status. Without parameters it's done on both adapters.

AlexNG_

[i2ambler]

Yeah.. I have tried just about everything to flash the bios.. but it wont find the card.. iflash /ff iflash /i /ff it just comes back with 'no adapter found' Im thinking there has to be some sort of incompatability between the dell riser and the 4060c or something.. I cant think of anything else it could be.

[adehart]

I mentioned this already I think but I absolutely could not flash this card in my servers until I ran Windows with the SAN Surfer software. SAN Surfer did it without a hitch but this admittedly is a royal pain.