Has HP DL 580 G5 enough interrupts or have others 24 interrupts?

[meistermn]

Are there Servers out that have enough interrupts more than 16 maybe 24?

Current I have a 585 G2 with 4 Dual Port Cards (HP NC 360) und 2 Dual Port Fiberchannel Card (HP 1212).

After looking at the output of /proc/vmware/interrupts file like in following articel described, I have although shared interrupts.

So I think that could be the case why I have 3,6 MB /s throughput on the IBM SAN 8100, when I copy inside the vm a large file onto the same virtual harddisk.

Is there allthough IRQ sharing with the new HP DL585 G2 , IBM 3850 M2 , SUN X4600 M2

Does the X4 Chipsets from provide benefit over the other vendors?

[oreeh]

On my DL585G2 systems I see u to 29 IRQs.

Since ESX uses the APICs there really isn't a "hard" limit with interrupts - I think that your issue has to do with the BIOS configuration.

Depending on how interrupts are shared (which devices share them), the sharing itself is not a problem.

[meistermn]

In this thread although is talked about sharde interrupts.

I had the same problem with rescanning , if the cards wre not plaeced in right order.

We were experiencing the same problems with our DL585 G2's.

It all comes down to IRQ Conflicts, PCI-cards sharing IRQ's.

We had some HP Engineers here at site, they eventually were able to joggle around the PCI-cards, until we got rid of the SAN Rescan issue.

We are still experiencing IRQ conflicts but the Rescan SAN problems are solved.

To totally solve this issue, we are awaiting (as everyone) a new HP BIOS Upgrade!

We divided our PCI devices as follows:

Slot 1: (PCI-X) Empty

Slot 2: (PCI-X) Empty

Slot 3: (PCI-E x4) HP NC380T Dual Port NIC

Slot 4: (PCI-E x4) HP FC1142SR 4Gb HBA

Slot 5: (PCI-E x8) HP NC380T Dual Port NIC

Slot 6: (PCI-E x4) HP FC1142SR 4Gb HBA

Slot 7: (PCI-E x8) HP NC380T Dual Port NIC

Slot 8: (PCI-E x8) HP NC380T Dual Port NIC

Slot 9: (PCI-E x4) HP P400 SAS Controller

With this set-up we are not experiencing any SAN-Rescan Freezes anymore.

Hope this helps for you.

In this thread they are although reporting shared irqs

Did not know that there is script out for find irq conflicts !!! cool!!!

So we have to wait for esx 3.5

[oreeh]

According to posts from VMware engineers in the "Severe network performance issues" thread IRQ sharing only is an issue if the IRQ is shared between the COS and the VMkernel.

IRQ sharing between devices owned by the VMkernel is not / should not be a problem.

[meistermn]

So you think I have no trouble?

I think I will take out 1 Fiber card and 1 Ethernet card and look at performance.

For what are the ISA edge used? Can I disable this

IRQ 16,shars 4 devices

IRQ 17 shares 4 devices

IRQ 18 shares 4 devices

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3 PCPU 4 PCPU 5 PCPU 6 PCPU 7

0x21: 2 0 0 0 0 0 0 0 COS irq 1 (ISA edge), <VMK device>

0x29: 0 0 0 0 0 0 0 0 <COS irq 6 (ISA edge)>

0x31: 0 0 0 0 0 0 0 0 <COS irq 7 (ISA edge)>

0x39: 0 0 0 0 0 0 0 0 <COS irq 8 (ISA edge)>

0x41: 0 0 0 0 0 0 0 0 <COS irq 9 (ISA edge)>

0x49: 1 0 0 0 0 0 0 0 COS irq 12 (ISA edge)

0x51: 0 0 0 0 0 0 0 0 <COS irq 13 (ISA edge)>

0x59: 1 0 0 0 0 0 0 0 COS irq 14 (ISA edge)

0x61: 0 0 0 0 0 0 0 0 <COS irq 15 (ISA edge)>

0x69: 27 0 4190 718980 0 221 70486 0 <COS irq 17 (PCI level)>, VMK vmnic9, VMK vmnic4, VMK qla2300, VMK cciss0

0x71: 33815751 0 1396 0 0 0 0 0 COS irq 18 (PCI level), VMK vmnic8, VMK vmnic7, VMK vmnic3, VMK vmnic2, VMK qla2300, VMK qla2300

0x79: 1449 29 264326 31927255 0 7031 68481 0 <COS irq 16 (PCI level)>, VMK vmnic6, VMK vmnic1, VMK qla2300

0x81: 3 23 2354 156932 0 330 11275 0 <COS irq 23 (PCI level)>, VMK vmnic0

0x89: 3 22 573 128295 0 19 3645 0 <COS irq 24 (PCI level)>, VMK vmnic5

0xdf: 104388189 104398193 104398193 104398193 104398193 104398193 104398193 104398193 VMK timer

0xe1: 0 0 0 0 1319 4819 304042 313853 VMK monitor

0xe9: 463498 122833 64855 13263 238444 288006 26801267 26244835 VMK resched

0xf1: 20 11 20 17 18157 6103 1800 1369 VMK tlb

0xf9: 895197 0 0 0 0 0 0 0 VMK noop

0xfc: 0 0 0 0 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 0 0 0 0 VMK error

0xff: 0 0 0 0 0 0 0 0 VMK spurious

[oreeh]

ISA edge interrupts are used by "legacy" devices (COM, LPT, keyboard, mouse, FDD,...)

You have a problem - 0x69,0x79,0x81 and 0x89 are shared between the COS and the VMkernel.

Try to disable any unneeded / unused devices in the BIOS (for example USB).

[meistermn]

How is our out put form cat /proc/vmware/interrupts ? what network card and fiber card are you using and in which pci express slot did you put for your HP Dl 585 G2?

[oreeh]

Vector    PCPU  0    PCPU  1    PCPU  2    PCPU  3
0x21:           2          0          0          0 COS irq 1 (ISA edge), <VMK device>
0x29:           0          0          0          0 <COS irq 3 (ISA edge)>
0x31:           0          0          0          0 <COS irq 4 (ISA edge)>
0x39:           0          0          0          0 <COS irq 6 (ISA edge)>
0x41:           0          0          0          0 <COS irq 7 (ISA edge)>
0x49:           0          0          0          0 <COS irq 8 (ISA edge)>
0x51:           0          0          0          0 <COS irq 9 (ISA edge)>
0x59:          16          0          0          0 COS irq 12 (ISA edge)
0x61:           0          0          0          0 <COS irq 13 (ISA edge)>
0x69:           1          0          0          0 COS irq 14 (ISA edge)
0x71:           0          0          0          0 <COS irq 15 (ISA edge)>
0x79:           0          0          0          0 COS irq 16 (PCI level)
0x81:       16440          0          0          0 COS irq 17 (PCI level), VMK qla2300
0x89:    38090865   71878491   89500187   92041240 <COS irq 18 (PCI level)>, VMK qla2300, VMK cciss0
0x91:     2144889    4541426    2102959    2474490 <COS irq 23 (PCI level)>, VMK vmnic0
0x99:     1182074    1267829    1527891    1753493 <COS irq 24 (PCI level)>, VMK vmnic1
0xa1:    39374724   82466201   13744469   14570629 <COS irq 19 (PCI level)>, VMK vmnic2
0xa9:   126908577  227387344  566379065  586931024 <COS irq 20 (PCI level)>, VMK vmnic3
0xb1:       99636     335508     261958     281922 <COS irq 21 (PCI level)>, VMK vmnic4
0xb9:    33174483   49149023   14332567   14991793 <COS irq 22 (PCI level)>, VMK vmnic5
0xdf:  1783721952 1791339734 1573313146 1572954314 VMK timer
0xe1:    22966296   20875613    5213626    5254647 VMK monitor
0xe9:   535934178  463331956  132182709  132398542 VMK resched
0xf1:       36474      65712      97230      98986 VMK tlb
0xf9:     4223286          0          0          0 VMK noop
0xfc:           0          0          0          0 VMK thermal
0xfd:           0          0          0          0 VMK lint1
0xfe:           0          0          0          0 VMK error
0xff:           0          0          0          0 VMK spurious

Not sure about the slots (I don't have the HP agents installed and I'm not a the location at the moment)

[meistermn]

So what did you disable in Bios to get IRQ 19.20,21,22. Did you disable USB, Serial Port, Floppy other things.

If you disable usb, then you not all rsb functions?

Are you using 2 Single Port Fiberchannel cards and 2 Dual Port Ethernetcards? Which one ?

Do you use VMOtion with our onboard cards in portgroup with service console?

[oreeh]

I disabled USB and the serial Port.

I'm not sure about the RSB functions, sine I only need the ILO virtual power / reset switch (I use a KVM switch).

HP NC340T (quad port NIC, PCI-X !)

Smart Array P400

FC2243DC (dual channel HBA, PCI-X !)

FC2214DC (dual channel HBA, PCIe)

Do you use VMOtion with our onboard cards in portgroup with service console?

Yes, one onboard NIC and one port of the Intel card are used for this.

To me it seems that I don't have IRQ sharing problems due to the fact that I use PCIe and PCI-X devices.

[meistermn]

Okay I disabled usb, serial and floppy.

That does not free up IRQ'S. Although it is not possible to set the IRQ for the devices in the bios. In my cheap gigabyte motherboard I can set irq's.

Which SAN are you using? What MB/s you get , when you copy to a vm and from a vm. What MB'S you get when you copy inside the vm(san to san copy).

[oreeh]

This box is connected to a HP MSA.

The disk performance is ok (compared to the capabilities of the MSA).

I'm able to modify the IRQ assignment for the PCI-X cards but not the PCIe cards.

This really looks like a HP BIOS problem with PCIe cards to me.

[meistermn]

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=10229

From the above articel :

You are not affected by the problem if, for every group of controllers that share the same Vec number:

All controllers in the group have mode C (assigned to the service console).

All controllers in the group have mode V (assigned to the VMkernel).

So this means having several devices on the same IRQ on mode V ( a shared IRQ) has not negative effects, if is not shared with the same Vec number of mode C

[oreeh]

Some posts regarding PCI-X / PCIe and IRQ sharing issues

http://communities.vmware.com/message/620524#620524

http://communities.vmware.com/message/751841#751841

http://communities.vmware.com/message/611658#611658