Boot issues with ESX 3.51 (and ESX3.5) on IBM LS21 using QLA4022 and EqualLogic San

[dmarshallx]

We installed ESX 3.5.1 on a new IBM LS-21 blade on an IBM BladeCenter H. The LS-21 is running

BIOS 1.05

Qlogic add-on card with BIOS 1.09a and Firmware 2.00.00.62

32GB of RAM

Two 3.0GHz AMD dual-core processors

The LS-21 blade boots from our EqualLogic SAN. During booting of this system when the CD or OS boots when it reaches the “loading qla4022…” message on the console, the system takes 16-22 minutes to continue. We saw the same problem loading ESX 3.5 and decided to try 3.5.1 after reviewing the release notes. However this does not seem to have improved the situation.

All of our other LS-21 systems are running ESX 3.0.2 from the same EqualLogic SAN and they do not exhibit this behavior.

We’ve monitored the activity on the EqualLogic SAN console. During the boot, the ESX 3.5.1 system repeats a pattern of logins and resets.

The pattern:

- Server logons to the SAN

- Server stays connected for 2 minutes, no bytes are reported read or written

- SAN Session is reset

This pattern repeats within seconds of the Session reset 5 or 6 times before the system stays connected and begins loading the remainder of the OS.

We took a sniffer network trace of this activity of this activity. The bios load completes around packet 150, the qla4022 loading message is about packet 3150 and the os continues loading around packet 5300, by packet 61000 the OS is loaded and up.

The Qlogic card is configured with:

Jumbo Frames

Manual IP (no gateway)

Header and Data digest

Target IP address and strings set

Is this a known issue? If so, what is the resolution.

[AlexNG_]

Hi TheCleaner,

You're talking abaout hw initiators?

You shoul see both hbas... strange... maybe the second port is disabled?

If it's sw iSCSI, you just get one hba.

AlexNG

[TheCleaner]

Alex,

thanks, I'll have to check it out after this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100581...

Apparently that is the patch we've been waiting for???

Summaries and Symptoms

This patch updates the Software QLogic iSCSI driver to v3.30.00.08-vm4 and fixes the following issue:

If a system with a QLogic iSCSI HBA has any ports not connected to the network, the driver initialization takes a long time. This occurs because of a driver issue that does not properly time-out the wait for accessing the network through the link-down port. This issue affects QLogic iSCSI HBA models QLA4050c, QLA4052c, QLE4060c, and QLE4062c.

[AlexNG_]

Hi TheCleaner,

Right, this one maybe can do the trick!!! The prob is that IBM iSCSI hbas are not supported with DS3300... you're lucky, you've got de N3300...

Anyway, I'll try to test the patch!

AlexNG

[RODDYM]

AlexNG,

Are you saying the IBM iSCSI HBAs they recommend for the DS3300 on this site...

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=89...

Are not supported by IBM or VMware in ESX configurations?

Thanks,

Mark

[AlexNG_]

Roddym,

Yes thats just what I'm saying. VMware does not support this configuration, or at least it was not supported on april/may (aprox). That's why we changed the hbas for networks cards, the customer wasn't very happy....

AlexNG

[adehart]

I can say wholheartedly and without question that the IBM single port and dual port iSCSI PCIe adapters from IBM (their version of the QLogic adapters) is definitely supported by VMWare ESX 3.5. I'm not sure where that impression came from. That may not have been true in April/May perhaps but it most definitely was in June. I can give you a case # if necessary that supports that as there was a bug with delays in booting from SAN with IBM and QLOGIC adapters and VMWare drive that I was working on with them and VMWare replicated the issue in their own labs with IBM cards, IBM servers, etc. I doubt they'd done so if that was an unsupported configuration. If you are on the phone with someone at VMWare support and they tell you otherwise, they're misinformed as I was told the same until I escalated it to management because they had a misprint on the official vmware supported hardware list.

[adehart]

I should have mentioned the part# of the card I'm referring to (the QLE4062C equivalent) is 42C1772 (42C1770 also) I believe and looking back at the ticket I opened with VMWare it looks like these were supported as far back at least as April. I had word a few weeks ago from VMWare that the boot delay issue was fixed in an update from VMWare. I do believe the latest firmware (.33 and up) and bios (1.13 and up) is necessary for these cards to work with 3.5. I'm not sure what the QLA4050 card (the PCI-X version) part # from IBM is but I can get it if you need it.

[AlexNG_]

Hi adehart,

Yes, thats right, The IBM iSsi hba was on VMware HCLs as supported. Also was (and is) the IBM DS3300 iSCSI Storage. Both were and are supported, but, there was a table on the HCLs where VMware specifies wich configurations were supported and, I can assure you that the IBM qle4062c or the 42c1770 (this was my hba pn) combined with the DS3300 was NOT supported.

We also opened a SR, but I closed before VMware requested more info due to this table!

The bug on that card was just something more, an added problem, but it wasn't the main problem. In our case (qle4260c + ds3300) we were losing paths (either marked as dead or disappeared), the IO performance was very poor... The iSCSI network was on a VLAN, so the problem were the cards and the storage. Once replaced, now everithing works lika a charm!

AlexNG

[adehart]

My apologies. Sometimes the devils in the details and looking back I missed the part @ it being WITH the DS3300 and not just VMWare. Sorry for the confusion.

[RODDYM]

Thanks for the information.

I have the 4060c and DS3300 and no issues so far... maybe it was a bad batch of cards? I will monitor this closely just in case...

[AlexNG_]

Adehart,

I'm with you, sometimes we reed too fast!

Roddym,

Not sure. I think it has something to do with some misunderstood between manufacurers....

AlexNG

[TheCleaner]

OK, I'm still stuck dang it.

With the latest VMWARE patches the qla4022.o loads just fine, but if I have the HBA's connected via iscsi to the IBM N3300 LUN then during boot it takes a while for "loading vmfs3" to finish, then it finds the LUN, assigns it to disk SDB, and then gets stuck at some strange messages about "usb-storage not initialized" and "insmod usb-storage failed".

If I disable ISCSI on the N3300 LUN's then it boots ok, so for some reason if it is attached and sees the SAN via iscsi it won't boot still...<mad>

I've got a ticket open with IBM on this but this is crazy...

[AlexNG_]

Hi TheCleaner,

usb? Sounds strange... Maybe you could also open a SR with VMware, because the driver is developed by VMware...

AlexNG

[TheCleaner]

Got it figured out. For some reason the LUN on the SAN was "corrupt" or something. I removed the lun completely and created a new one and now it boots ok.

Odd behavior though.

[YueAlexa]

Hi everyone!...

In my team, we are trying to achive that vCenter boot from local disk,

not SAN, but we can't still manage how to do it. We have been

eliminating the possible options, but all the configuration seems to be

ok.

I have been searching everywere for some reference on how to do it

(manuals, forums, blogs) but until now I only find comments of why it's

convenient and why not, but not how.

Does any one of you know hot or where can I find a reference?

We are installing ESX 4.0, in a x3550 M2 with HS22 blades. The problem

is that when it's connected the fibre channel, it want to find the

disks by SAN, and if it's changed the boot order it doesn't work

either. If the fibre is disconnected all works fine but it doesn't have

access to the storage.

Hope, someone could help us.

Thanks.