HP DL580 G7 Hangs while booting Vsphere 4.1

[korealife]

Hardward Specification:

HP DL580G7

Xeon x7560(2.26Ghz,8core)x4EA/128GB memory

PCI-Expansion Board(6 PCI slot)

8 single-port Fiber NIC, 2 Dual-port HBA, 1 dual-port UTP NIC

Error Condition:

When PCI-Slot 1 in PCI-Expansion board is occupied

Symptoms:

When PCI-slot 1 is occupied, Vsphere 4.1 CD/USB installation hangs at driver loading stage.

(if NIC is in PCI-slot 1, installation windows hangs at network driver loading stage, if HBA in PCI-Slot 1, vice versa)

When PCI-Slot 1 is occupied, Vsphere 4.1 Kernel loading hangs at specific driver(in slot 1) loading stage.

Current status:

Installed Vsphere 4.1 with PCI-slot 1 removed & Opened SR with VMware & HP

Comment:

All Firmware is up-to-date, HP NMI Driver installed

PCI-Expansion board & Processor/Memory Catridge replaced

Test-installed Windows 2008R2 and RHEL 5.3 but no trouble

-

---

Hello

We recently bought DL580G7 and having trouble with setting it up.

We took irq-sharing could be isssue, but even when using only one PCI-card in PCI-slot 1 generate same symptom.

We have ongoing SR both with VMware & HP, but don't think it will be solved soon.

Does anyone saw similar issue?

I appeciate any further assistance.

[AWo]

Welcome to the forums!

What about HP support to change the board or the whole server?

AWo

VCP 3 & 4

\[:o]===\[o:]

=Would you like to have this posting as a ringtone on your cell phone?=

=Send "Posting" to 911 for only $999999,99!=

[korealife]

As I mentioned above, We tested with new PCI-Express expansion board , and Processor & memory catridge(where expansion board conects to) after.

But unfortunately, both did not work.

HP Guys are saying they are elevating issues to L2(APJ) level

[AWo]

I would wait for HP and have a close look here if someone offers a solution

.
AWo

VCP 3 & 4

\[:o]===\[o:]

=Would you like to have this posting as a ringtone on your cell phone?=

=Send "Posting" to 911 for only $999999,99!=

[MarkTaylor]

I am not sure if I can help much being that I also have a ticket open with HP for a similar issue with my brand new DL580 G7.

But a couple things I found in my testing so far.

I see you have "iommu" listed in your tag's

In the bios there is a VT-d setting under the processor options. Disable that,and the installer will not show the 80.iommu error.

Secondly I don't have the additional riser card, but I do have 4 NICs in the slots and I am having an error on the main screen that says the intervector is out of interrupt vectors.

I am going to go out on a limb and ask is there an NC375i (or T) NIC card in the system?

how long have you let it sit at the starting network drivers page?

[korealife]

Hi Mark,

Thank you for your reply,

We have

8 NC373F PCIe MultiFunctio Gigabit Server Adapter(single port Fiber NIC)

1 NC360T PCIe Dual-port Gigabit Server Adapter(dual port UTP-NIC)

2 AJ764A 8Gb Dual-port PCI-e FC HBA(dual-port HBA)

(Onboard) HP NC375i Quad Port Multifunction Gigabit Server Adapter

First, Our VMware Engineer suggested that setting from the start, And HP Engineer tested with various BIOS parameters for troublshooting.

But I willl share it with VMware Guys for another test run with VT-d off installation.

Second. We were fine with every PCIe slot occupied except PCIe slot 1(5 onboard, 5 expansion). Installation and Vsphere Kernel loading was good.

At driver loading stage, We waited for 5-10minutes for average, but at the event of first occurance, we waited almost 20 minutes for ensure it is hang.

Because at Installation stage, we could see mouse pointers are not moving, but booting stage there is nothing also we can do to cross check.

Thank you againg for your kind replies, and I will share with updates.

Have a nice day!

[MarkTaylor]

Yeah, 20 minutes is a long time to wait. I have just the four quad port NIC's and I wait 2-5 minutes at the loading network drivers section.

I wish you the best of luck.

I am so far unhappy with the G7, and the level of support I am getting from HP I hope this is not a sign of things to come with HP's hardware and support.

[korealife]

Response from VMware suggested turning "VT-d" option off from BIOS for test.

We successfully booted ok with "VT-d disabled"

There was similar issue in vsphere 4.1 release note for DL360 G6 server.

And they solved problem by firmware update.

We are stiil looking forward to VMware's further assistance.

[ckonenonly]

So if I were looking into buying two HP DL580 G7, I should re-consider?

[MarkTaylor]

You may want to investigate options.

The 580g7 that I got working has been working 'ok'

however

Do not buy any NC375(4port 1GB NIC) network cards, of IF you buy those cards, understand that according to VMware you can only have 3 in a system at any given time.

we removed the 375 cards and put other 4 port cards in their place and it seems to have resolved the issue.

HP and VMware have 'promised' a resolution to this. but I have not seen one yet, and as of the time I type this they are still both very limited in the ability to support the G7 Server.

[ckonenonly]

Mark,

Thanks for your response. By default the mother board has an HP Embedded NC375i Quad Port Multifunction GB Adapter. Per your commit below you indicated do not buy NC375 if it is Embedded I don't believe there is an option to remove the card right? So I just disable this card on the Motherboard should there be issues with it?

Again thanks for your time.

Craig

[allencrawford]

I think he's probably referring to the NC375T NIC (which is a PCI express card).

[ckonenonly]

Correct this is in regards to the add on NC375T NIC. I pressed forward and purchased the HP DL580 G7. As long as the add-on card is not these model it appears the problem doesn't surface. I stayed clear of any additional NC375T PCI express cards. All is working superb. I was conducting research when I first found the original post to ensure compatibility.  Thanks for everyone's input.

Craig

[VeyronMick]

Are you using the HP or VMware builds of ESX 4.1?

There are all kinds of driver issues with the G7 that HP have fixed with their own builds of ESX with their drivers embedded in the ISO.

I would also try HPs build of ESXi 4.1 if you get no joy with the classic build.

You can get the HP builds of ESX from their download site (not VMwares one).

[ckonenonly]

Just to let everyone know I have never had any issues at all.  My intial reply was to get more details before I purchased the HP products.  After purchasing the products I stayed away from the reported add on card as recommended and installed VMware builds of ESX 4.1 and also ESXi 4.x, without any issues at all.  ESX is hosted from the onbaord drives of each DL580 and all VMhosts are iSCSI booted. If someone want's to know the add on Nic's I purchased I can repost that if needed.     

[J-D]

Hi, we have a DL580 G7 and just got 4.1u1 installed on it without problems.

However after VMotioning some VM4s to the host we got weird network issues. I am sure the network switch is configured correctly (VLAN tagging) and so is the setup on ESX. Some VM's suddenly loose connection and actually the VMotion made us loose more than 1 ping.

We have these NIC's in the host:

- onboard NC375i quad port which is shown as NetXen

- added quad port NC365T, shown as Intel NC465T

- 2 added dualport NC382T's which are shown in the vSphere gui as Broadcom NC382T

I used the VMware's 4.1 installation CD for ESX. Does HP have their own CD for 4.1? or is it only ESXi?

I guess I need some other drivers to get this to work.

What NIC's did you add?

[MarkTaylor]

Hi JD,

The problem we experienced with the NC375T and I cards was that The ESX Host would not Boot. There would be an interrupt vector error and the system would freeze.

However, during the process of dealing with support I found that the "recommened maximum's" for ESX4.1 is lower than I had anticipated.

QUOTE VmWare Support:

I would like to share one KB article on Recommended configuration maximums for NIC ports on ESX/ESXi 4.1, 4.0 (http://kb.vmware.com/kb/1020808)

....

As per KB article if jumbo frame in enable in environment we can only have maximum 12 1 GB network port on ESX server. ( Also this

Configuration depends on the hardware ,number of cores, overall system memory and other factors)

After checking the system log, I found that there is one HP NC375i Integrated Quad Port card and apart from this we are only able to deploy 2

more quad port cards. This is because we cannot have more than 12 1gb port on ESX server ( considering Jumbo frame is enable in the

environment)

I will update you as soon as I will any further inputs on this issue.

Depending on your network configuration you may be reaching the maximum recommened number of NIC's in that system. I would suggest you review the KB article I posted and see if any of that applies to your situation.

If the ESX host is not locking up with an intervector memory error than it's a different problem then we have experienced.

[VeyronMick]

Do you have firmware 4.0.556 on those cards?

We are seeing more cases with earlier firmware causing that kind of issue.

Check our HP advistory c02964542

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542&jumpid=em_alerts_us...

[MarkTaylor]

Veyron,

We have ours updaeted to recent firmware yes. but we also ditched the 375T cards for 364T cards and that solved our problems.

[J-D]

@VeyronMick : am I glad you pasted that link to that HP advisory!! Thanks, we did the upgrade including using the latest async driver for VMware and for now the issue is gone.

I have to say HP could have made it easier to find that latest driver...it's not in the cross platform section nor the VMware section...I mean I normally don't look under Red Hat 5 for firmwares...