Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues

[CiscoKid]

I have 4 Quad-Port NICs installed in a whitebox server that has all devices that are listed in VMware's HCL (2xIntel 1000MT, 2xIntel 1000PT, 1xLSI-320E and an Intel 5000PSL Motherboard). The issue that I am experiencing is in regards to the performance of the Intel Pro 1000PT Quad-ports.

Here is the scenario:

Tested using a Cisco 3560G-24PS with 3 VLANs 10, 50 and 100. I am using external-switch tagging so I have my switch configured to tag VLAN1 but all switchports that the pNICs are attached to have VLAN 1 pruned from the switch configuration. I am specifically adding a port for each VLAN with the VMware Network Configuration. This is not the issue but the scope of the configuration might help with identifying my problem.

I have several guest that I am purposely bouncing around to identify which NIC ports on the Intel Pro 1000PTs are experiencing performance issues. When the VM guests are using ports A & B on the Intel Pro 1000PTs they are running at =/<1ms response time, which is normal for LAN computers. Those same guests are then bounced to port C on the Intel Pro 1000PT. What I am observing is that the firts guest that moves its MAC address to the new NIC port is running at =/<1ms while all other guests fluctuate at =/<1ms and >128ms. Finally, I try to bounce the same VM guests to NIC port D on the Intel Pro 1000PTs and they are running slower than most WAN links averaging >600ms or more.

My conclusion is that there is definitely a performance issues with NIC ports C & D on the Intel Pro 1000PT quad-port while all 4 NIC ports on the Intel Pro 1000MT run at =/<1ms response times.

I have disabled the EISR (Enhanced Intel Speedstep Reduction) and the I/OAT functions with in the mainboard's BIOS and have fully patched the VMware ESX 3.01 all the way to 3/29/2007 which included the ESX-1006511 patch which contains the drivers for the Intel Pro 1000PTs. I disabled those the BIOS feature after everything was installed and patched which caused ESX to do a system reconfiguration followed by a reboot. Could this have caused poor performance on those NIC ports on the Intel Pro 1000PT quad-port NICs?

So far, I would have been better off buying dual-port cards instead of the quad-port cards. Has anyone else experienced this issue and if so has it been resolved? Also, I have read press conference from Intel that mentioned that VMware, Microsoft and Intel are striving to fully adopt the I/OAT technology, has there been any word when VMware plans to implement with ESX 3?

[christianZ]

Well the quad port card was discussed here many times - have you searched the forum ?

[CiscoKid]

Yes, nothing pertaining to the actual tests that I have observed.

[RParker]

I am curious are these in PCI-x (4x) ports on the machine or are 2 on (1x) ports and maybe 2 on another, that could affect performance some.

We have similar, although not quad port, but dual port, and I made sure Dell configured them as 4x speed for both. Because there are 2 kinds of PCI-e/x ports in a server machine, and it will affect the behaviour of the hardware.

Another thing is, the VM's were the migrated from another previous version of VM ware or maybe 2.5 ESX? The VM Tools also affect this. I had a hell of a time with a few, until I figured out that you have to remove the network driver from hardware before you upgrade the tools.

Just a thought...

[CiscoKid]

The Intel Pro 1000PT Quad-ports are currently plugged into PCI-E x8 (x4 Electrical) of the Intel 5000PSLSATA Motherboard.

[Paul_Lalonde]

If you view the output of

cat /proc/vmware/pci

do any of the Intel GigE PCI devices share IRQs with the USB ports?

If so, you've got an IRQ sharing issue between the VMkernel and the Service Console. Disable USB completely and try the tests again.

Paul

[CiscoKid]

Paul, thanks once again for the help. I know that you jumped right in to help with an issue with my ASUS DSBF12/SAS and I ended up dumping that board and went for the Intel 5000PSLSATA motherboard which is on VMware's HCL. Anyhow, do you think that this is an issue more so at the PCI-E bus sharing level? I am kind of new to the PCI-E world and don't quite understand what manufactures mean when the label their slots as x8 and then in parathesis (x4 Electrical). Does that mean that if the slot has to supply power to the card that it will be a x4 slot but a x8 form factor?

I followed your recommendations of disabling the USB controller in the BIOS and the only difference now is that NIC port D on both cards are the ones that fluctuate between </=1ms response times and greater than 128ms response times, while NIC port C on the NIC cards is worse than a WAN link averaging more than 800ms response times. Should I have to do a clean rebuild of ESX after making hardware/BIOS changes to ensure a clean OS or is ESX tolerant enough to not have to do that? Below are the outputs that you normally request to assist with this issue:

**********BEGIN /PROC/VMWARE/PCI*************

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name

Spawned bus

000:00.0 8086:25d8 8086:3476 Host/PCI Intel 0/ / A C

000:02.0 8086:25f7 0000:0000 PCI/PCI Intel 001 C

000:03.0 8086:25e3 0000:0000 PCI/PCI Intel 011 C

000:04.0 8086:25f8 0000:0000 PCI/PCI Intel 012 C

000:05.0 8086:25e5 0000:0000 PCI/PCI Intel 015 C

000:06.0 8086:25f9 0000:0000 PCI/PCI Intel 016 C

000:07.0 8086:25e7 0000:0000 PCI/PCI Intel 017 C

000:16.0 8086:25f0 8086:3476 Host/PCI Intel C

000:16.1 8086:25f0 8086:3476 Host/PCI Intel C

000:16.2 8086:25f0 8086:3476 Host/PCI Intel C

000:17.0 8086:25f1 8086:3476 Host/PCI Intel C

000:19.0 8086:25f3 8086:3476 Host/PCI Intel C

000:21.0 8086:25f5 8086:3476 Host/PCI Intel C

000:22.0 8086:25f6 8086:3476 Host/PCI Intel C

000:28.0 8086:2690 0000:0000 PCI/PCI Intel 018 C

000:30.0 8086:244e 0000:0000 PCI/PCI Intel 022 C

000:31.0 8086:2670 8086:3476 PCI/ISA Intel C

000:31.1 8086:269e 8086:3476 IDE Intel 10/ 20/0x71 A C

000:31.3 8086:269b 8086:3476 SMBus Intel 10/ / B C

001:00.0 8086:3500 0000:0000 PCI/PCI Intel 002 C

001:00.3 8086:350c 0000:0000 PCI/PCI Intel 008 C

002:00.0 8086:3510 0000:0000 PCI/PCI Intel 003 C

002:02.0 8086:3518 0000:0000 PCI/PCI Intel 007 C

003:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 004 C

004:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 005 C

004:01.0 111d:8018 0000:0000 PCI/PCI 0x111d 006 C

005:00.0 8086:10a4 8086:10a4 Ethernet Intel 10/ 16/0x79 A V e1000 vmnic10

005:00.1 8086:10a4 8086:10a4 Ethernet Intel 11/ 17/0x81 B V e1000 vmnic11

006:00.0 8086:10a4 8086:10a4 Ethernet Intel 10/ 16/0x79 A V e1000 vmnic12

006:00.1 8086:10a4 8086:10a4 Ethernet Intel 11/ 17/0x81 B V e1000 vmnic13

007:00.0 8086:1096 8086:3476 Ethernet Intel 5/ 18/0x89 A V e1000 vmnic0

007:00.1 8086:1096 8086:3476 Ethernet Intel 11/ 19/0x91 B V e1000 vmnic1

008:01.0 1014:01a7 0000:0000 PCI/PCI IBM 009 C

008:02.0 1014:01a7 0000:0000 PCI/PCI IBM 010 C

009:04.0 8086:101d 8086:1000 Ethernet Intel 11/ 22/0x99 A V e1000 vmnic2

009:04.1 8086:101d 8086:1000 Ethernet Intel 5/ 23/0xa1 B V e1000 vmnic3

009:06.0 8086:101d 8086:1000 Ethernet Intel 11/ 24/0xa9 A V e1000 vmnic4

009:06.1 8086:101d 8086:1000 Ethernet Intel 10/ 21/0xb1 B V e1000 vmnic5

010:04.0 8086:101d 8086:1000 Ethernet Intel 5/ 23/0xa1 A V e1000 vmnic6

010:04.1 8086:101d 8086:1000 Ethernet Intel 11/ 24/0xa9 B V e1000 vmnic7

010:06.0 8086:101d 8086:1000 Ethernet Intel 10/ 21/0xb1 A V e1000 vmnic8

010:06.1 8086:101d 8086:1000 Ethernet Intel 11/ 22/0x99 B V e1000 vmnic9

012:00.0 8086:0330 0000:0000 PCI/PCI Intel 013 C

012:00.2 8086:0332 0000:0000 PCI/PCI Intel 014 C

013:14.0 1000:0408 1000:0002 RAID Symbios 5/ 18/0x89 A V megaraid2 vmhba0

018:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 019 C

019:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 020 C

019:01.0 111d:8018 0000:0000 PCI/PCI 0x111d 021 C

020:00.0 8086:10a4 8086:10a4 Ethernet Intel 10/ 16/0x79 A V e1000 vmnic14

020:00.1 8086:10a4 8086:10a4 Ethernet Intel 11/ 17/0x81 B V e1000 vmnic15

021:00.0 8086:10a4 8086:10a4 Ethernet Intel 10/ 16/0x79 A V e1000 vmnic16

021:00.1 8086:10a4 8086:10a4 Ethernet Intel 11/ 17/0x81 B V e1000 vmnic17

022:12.0 1002:515e 8086:3476 Display ATI 11/ / A C

**********END /PROC/VMWARE/PCI*************

**********BEGIN /PROC/VMWARE/INTERRUPTS*************

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3 PCPU 4 PCPU 5 PCPU 6 PCPU 7

0x21: 24 0 0 0 0 0 0 0 COS irq 1 (ISA edge), , VMK vmnic8, VMK vmnic5

0xdf: 28206827 28215244 28116876 28116876 28175687 28175687 28148056 28148056 VMK timer

0xe1: 71 290 638 156 0 0 0 0 VMK monitor

0xe9: 148517 114284 115516 114577 48088 60393 59 32899 VMK resched

0xf1: 247 247 308 298 624 515 288 234 VMK tlb

0xf9: 49288 0 0 0 0 0 0 0 VMK noop

0xfc: 0 0 0 0 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 0 0 0 0 VMK error

0xff: 0 0 0 0 0 0 0 0 VMK spurious

**********END /PROC/VMWARE/INTERRUPTS*************

**********BEGIN LSPCI*************

00:00.0 Host bridge: Intel Corporation: Unknown device 25d8 (rev b1)

00:02.0 PCI bridge: Intel Corporation: Unknown device 25f7 (rev b1)

00:03.0 PCI bridge: Intel Corporation: Unknown device 25e3 (rev b1)

00:04.0 PCI bridge: Intel Corporation: Unknown device 25f8 (rev b1)

00:05.0 PCI bridge: Intel Corporation: Unknown device 25e5 (rev b1)

00:06.0 PCI bridge: Intel Corporation: Unknown device 25f9 (rev b1)

00:07.0 PCI bridge: Intel Corporation: Unknown device 25e7 (rev b1)

00:10.0 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:10.1 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:10.2 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:11.0 Host bridge: Intel Corporation: Unknown device 25f1 (rev b1)

00:13.0 Host bridge: Intel Corporation: Unknown device 25f3 (rev b1)

00:15.0 Host bridge: Intel Corporation: Unknown device 25f5 (rev b1)

00:16.0 Host bridge: Intel Corporation: Unknown device 25f6 (rev b1)

00:1c.0 PCI bridge: Intel Corporation: Unknown device 2690 (rev 09)

00:1e.0 PCI bridge: Intel Corporation 82801BA/CA/DB/EB PCI Bridge (rev d9)

00:1f.0 ISA bridge: Intel Corporation: Unknown device 2670 (rev 09)

00:1f.1 IDE interface: Intel Corporation: Unknown device 269e (rev 09)

00:1f.3 SMBus: Intel Corporation: Unknown device 269b (rev 09)

01:00.0 PCI bridge: Intel Corporation: Unknown device 3500 (rev 01)

01:00.3 PCI bridge: Intel Corporation: Unknown device 350c (rev 01)

02:00.0 PCI bridge: Intel Corporation: Unknown device 3510 (rev 01)

02:02.0 PCI bridge: Intel Corporation: Unknown device 3518 (rev 01)

03:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

04:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

04:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

05:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

05:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

06:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

06:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

07:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

08:01.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

08:02.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

09:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0c:00.0 PCI bridge: Intel Corporation: Unknown device 0330 (rev 07)

0c:00.2 PCI bridge: Intel Corporation: Unknown device 0332 (rev 07)

0d:0e.0 RAID bus controller: LSI Logic / Symbios Logic LSI Logic MegaRAID (rev 07)

12:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

13:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

13:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

14:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

14:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

15:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

15:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

16:0c.0 VGA compatible controller: ATI Technologies Inc: Unknown device 515e (rev 02)

**********BEGIN /PROC/VMWARE/PCI*************

[CiscoKid]

I had a fun-filled weekend of nothing but server teardown and assembly trying to figure out what is causing this issue. I have disabled the USB in the BIOS as suggested by Paul and still experience the issue. I removed all the cards (2x1000MT Quad-ports and 1x1000PT Quad-ports) leaving just one of the 1000PT Quad-port and the LSI 320-2E only to experience the same issue where the C and D ports of the 1000PT quad-port NIC are slower than WAN links averaging above 600ms response times.

I then thought it just maybe a hardware limitation and then installed Windows 2003 Standard, loaded the NIC drivers and then built teams for the NICs. After all the configuration was complete with Windows 2003, all tests proved that all ports were operating under normal conditions showing reponse times of =/<1ms.

I am pretty much out of ideas at this point I have created 2 separate teams where I have omitted ports C & D on the 1000PT quad-ports until this issue is resolved. This is definitely something within VMware ESX as I proved that there is no issue with the hardware by all the different tests that I have performed and installing Windows 2003 without any issues. Please help!!!

[CiscoKid]

Sorry all, here is a copy of the switch configuration for all the switchports that the VM NICs are connected to:

interface GigabitEthernet0/1

description ***VMNIC***

power inline never

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,50,100

switchport mode trunk

switchport nonegotiate

switchport voice vlan 100

speed 1000

duplex full

no mdix auto

spanning-tree portfast