8 Replies Latest reply on Apr 30, 2007 10:58 AM by CiscoKid

    Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues

    CiscoKid Hot Shot

      I have 4 Quad-Port NICs installed in a whitebox server that has all devices that are listed in VMware's HCL (2xIntel 1000MT, 2xIntel 1000PT, 1xLSI-320E and an Intel 5000PSL Motherboard).  The issue that I am experiencing is in regards to the performance of the Intel Pro 1000PT Quad-ports.

       

      Here is the scenario:

       

      Tested using a Cisco 3560G-24PS with 3 VLANs 10, 50 and 100.  I am using external-switch tagging so I have my switch configured to tag VLAN1 but all switchports that the pNICs are attached to have VLAN 1 pruned from the switch configuration.  I am specifically adding a port for each VLAN with the VMware Network Configuration.  This is not the issue but the scope of the configuration might help with identifying my problem.

       

      I have several guest that I am purposely bouncing around to identify which NIC ports on the Intel Pro 1000PTs are experiencing performance issues.  When the VM guests are using ports A & B  on the Intel Pro 1000PTs they are running at =/<1ms response time, which is normal for LAN computers. Those same guests are then bounced to port C on the Intel Pro 1000PT. What I am observing is that the firts guest that moves its MAC address to the new NIC port is running at =/<1ms while all other guests fluctuate at =/<1ms and >128ms. Finally, I try to bounce the same VM guests to NIC port D on the Intel Pro 1000PTs and they are running slower than most WAN links averaging >600ms or more. 

       

      My conclusion is that there is definitely a performance issues with NIC ports C & D on the Intel Pro 1000PT quad-port while all 4 NIC ports on the Intel Pro 1000MT run at =/<1ms response times.

       

      I have disabled the EISR (Enhanced Intel Speedstep Reduction) and the I/OAT functions with in the mainboard's BIOS and have fully patched the VMware ESX 3.01 all the way to 3/29/2007 which included the ESX-1006511 patch which contains the drivers for the Intel Pro 1000PTs.  I disabled those the BIOS feature after everything was installed and patched which caused ESX to do a system reconfiguration followed by a reboot.  Could this have caused poor performance on those NIC ports on the Intel Pro 1000PT quad-port NICs?

       

      So far, I would have been better off buying dual-port cards instead of the quad-port cards.  Has anyone else experienced this issue and if so has it been resolved?  Also, I have read press conference from Intel that mentioned that VMware, Microsoft and Intel are striving to fully adopt the I/OAT technology, has there been any word when VMware plans to implement with ESX 3?

        • 1. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
          christianZ Virtuoso

          Well the quad port card was discussed here many times - have you searched the forum ?

          • 2. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
            CiscoKid Hot Shot

            Yes, nothing pertaining to the actual tests that I have observed.

            • 3. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
              RParker Guru

              I am curious are these in PCI-x (4x) ports on the machine or are 2 on (1x) ports and maybe 2 on another, that could affect performance some.

               

              We have similar, although not quad port, but dual port, and I made sure Dell configured them as 4x speed for both.  Because there are 2 kinds of PCI-e/x ports in a server machine, and it will affect the behaviour of the hardware.

               

              Another thing is, the VM's were the migrated from another previous version of VM ware or maybe 2.5 ESX?  The VM Tools also affect this.  I had a hell of a time with a few, until I figured out that you have to remove the network driver from hardware before you upgrade the tools.

               

              Just a thought...

              • 4. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
                CiscoKid Hot Shot

                The Intel Pro 1000PT Quad-ports are currently plugged into PCI-E x8 (x4 Electrical) of the Intel 5000PSLSATA Motherboard.

                • 5. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
                  Paul Lalonde Master

                  If you view the output of

                   

                  cat /proc/vmware/pci

                   

                  do any of the Intel GigE PCI devices share IRQs with the USB ports?

                   

                  If so, you've got an IRQ sharing issue between the VMkernel and the Service Console.  Disable USB completely and try the tests again.

                   

                  Paul

                  • 6. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
                    CiscoKid Hot Shot

                    Paul, thanks once again for the help.  I know that you jumped right in to help with an issue with my ASUS DSBF12/SAS and I ended up dumping that board and went for the Intel 5000PSLSATA motherboard which is on VMware's HCL.  Anyhow, do you think that this is an issue more so at the PCI-E bus sharing level?   I am kind of new to the PCI-E world and don't quite understand what manufactures mean when the label their slots as x8 and then in parathesis (x4 Electrical).  Does that mean that if the slot has to supply power to the card that it will be a x4 slot but a x8 form factor? 

                     

                    I followed your recommendations of disabling the USB controller in the BIOS and the only difference now is that NIC port D on both cards are the ones that fluctuate between </=1ms response times and greater than 128ms response times, while NIC port C on the NIC cards is worse than a WAN link averaging more than 800ms response times.  Should I have to do a clean rebuild of ESX after making hardware/BIOS changes to ensure a clean OS or is ESX tolerant enough to not have to do that?  Below are the outputs that you normally request to assist with this issue:

                     

                    **********BEGIN /PROC/VMWARE/PCI*************

                     

                    Bus:Sl.F Vend:Dvid Subv:Subd Type     Vendor   ISA/irq/Vec P M Module   Name   

                                                                   Spawned bus                     

                    000:00.0 8086:25d8 8086:3476 Host/PCI Intel     0/   /     A C                 

                    000:02.0 8086:25f7 0000:0000 PCI/PCI  Intel        001       C                 

                    000:03.0 8086:25e3 0000:0000 PCI/PCI  Intel        011       C                 

                    000:04.0 8086:25f8 0000:0000 PCI/PCI  Intel        012       C                 

                    000:05.0 8086:25e5 0000:0000 PCI/PCI  Intel        015       C                 

                    000:06.0 8086:25f9 0000:0000 PCI/PCI  Intel        016       C                 

                    000:07.0 8086:25e7 0000:0000 PCI/PCI  Intel        017       C                 

                    000:16.0 8086:25f0 8086:3476 Host/PCI Intel                  C                 

                    000:16.1 8086:25f0 8086:3476 Host/PCI Intel                  C                 

                    000:16.2 8086:25f0 8086:3476 Host/PCI Intel                  C                 

                    000:17.0 8086:25f1 8086:3476 Host/PCI Intel                  C                 

                    000:19.0 8086:25f3 8086:3476 Host/PCI Intel                  C                 

                    000:21.0 8086:25f5 8086:3476 Host/PCI Intel                  C                 

                    000:22.0 8086:25f6 8086:3476 Host/PCI Intel                  C                 

                    000:28.0 8086:2690 0000:0000 PCI/PCI  Intel        018       C                 

                    000:30.0 8086:244e 0000:0000 PCI/PCI  Intel        022       C                 

                    000:31.0 8086:2670 8086:3476 PCI/ISA  Intel                  C                 

                    000:31.1 8086:269e 8086:3476 IDE      Intel    10/ 20/0x71 A C                 

                    000:31.3 8086:269b 8086:3476 SMBus    Intel    10/   /     B C                 

                    001:00.0 8086:3500 0000:0000 PCI/PCI  Intel        002       C                 

                    001:00.3 8086:350c 0000:0000 PCI/PCI  Intel        008       C                 

                    002:00.0 8086:3510 0000:0000 PCI/PCI  Intel        003       C                 

                    002:02.0 8086:3518 0000:0000 PCI/PCI  Intel        007       C                 

                    003:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       004       C                 

                    004:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       005       C                 

                    004:01.0 111d:8018 0000:0000 PCI/PCI  0x111d       006       C                 

                    005:00.0 8086:10a4 8086:10a4 Ethernet Intel    10/ 16/0x79 A V e1000    vmnic10

                    005:00.1 8086:10a4 8086:10a4 Ethernet Intel    11/ 17/0x81 B V e1000    vmnic11

                    006:00.0 8086:10a4 8086:10a4 Ethernet Intel    10/ 16/0x79 A V e1000    vmnic12

                    006:00.1 8086:10a4 8086:10a4 Ethernet Intel    11/ 17/0x81 B V e1000    vmnic13

                    007:00.0 8086:1096 8086:3476 Ethernet Intel     5/ 18/0x89 A V e1000    vmnic0 

                    007:00.1 8086:1096 8086:3476 Ethernet Intel    11/ 19/0x91 B V e1000    vmnic1 

                    008:01.0 1014:01a7 0000:0000 PCI/PCI  IBM          009       C                 

                    008:02.0 1014:01a7 0000:0000 PCI/PCI  IBM          010       C                 

                    009:04.0 8086:101d 8086:1000 Ethernet Intel    11/ 22/0x99 A V e1000    vmnic2 

                    009:04.1 8086:101d 8086:1000 Ethernet Intel     5/ 23/0xa1 B V e1000    vmnic3 

                    009:06.0 8086:101d 8086:1000 Ethernet Intel    11/ 24/0xa9 A V e1000    vmnic4 

                    009:06.1 8086:101d 8086:1000 Ethernet Intel    10/ 21/0xb1 B V e1000    vmnic5 

                    010:04.0 8086:101d 8086:1000 Ethernet Intel     5/ 23/0xa1 A V e1000    vmnic6 

                    010:04.1 8086:101d 8086:1000 Ethernet Intel    11/ 24/0xa9 B V e1000    vmnic7 

                    010:06.0 8086:101d 8086:1000 Ethernet Intel    10/ 21/0xb1 A V e1000    vmnic8 

                    010:06.1 8086:101d 8086:1000 Ethernet Intel    11/ 22/0x99 B V e1000    vmnic9 

                    012:00.0 8086:0330 0000:0000 PCI/PCI  Intel        013       C                 

                    012:00.2 8086:0332 0000:0000 PCI/PCI  Intel        014       C                 

                    013:14.0 1000:0408 1000:0002 RAID     Symbios   5/ 18/0x89 A V megaraid2 vmhba0 

                    018:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       019       C                 

                    019:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       020       C                 

                    019:01.0 111d:8018 0000:0000 PCI/PCI  0x111d       021       C                 

                    020:00.0 8086:10a4 8086:10a4 Ethernet Intel    10/ 16/0x79 A V e1000    vmnic14

                    020:00.1 8086:10a4 8086:10a4 Ethernet Intel    11/ 17/0x81 B V e1000    vmnic15

                    021:00.0 8086:10a4 8086:10a4 Ethernet Intel    10/ 16/0x79 A V e1000    vmnic16

                    021:00.1 8086:10a4 8086:10a4 Ethernet Intel    11/ 17/0x81 B V e1000    vmnic17

                    022:12.0 1002:515e 8086:3476 Display  ATI      11/   /     A C

                     

                    **********END /PROC/VMWARE/PCI*************

                     

                    **********BEGIN /PROC/VMWARE/INTERRUPTS*************

                    Vector    PCPU  0    PCPU  1    PCPU  2    PCPU  3    PCPU  4    PCPU  5    PCPU  6    PCPU  7

                    0x21:          24          0          0          0          0          0          0          0 COS irq 1 (ISA edge), , VMK vmnic8, VMK vmnic5

                    0xdf:    28206827   28215244   28116876   28116876   28175687   28175687   28148056   28148056 VMK timer

                    0xe1:          71        290        638        156          0          0          0          0 VMK monitor

                    0xe9:      148517     114284     115516     114577      48088      60393         59      32899 VMK resched

                    0xf1:         247        247        308        298        624        515        288        234 VMK tlb

                    0xf9:       49288          0          0          0          0          0          0          0 VMK noop

                    0xfc:           0          0          0          0          0          0          0          0 VMK thermal

                    0xfd:           0          0          0          0          0          0          0          0 VMK lint1

                    0xfe:           0          0          0          0          0          0          0          0 VMK error

                    0xff:           0          0          0          0          0          0          0          0 VMK spurious

                     

                    **********END /PROC/VMWARE/INTERRUPTS*************

                     

                    **********BEGIN LSPCI*************

                     

                    00:00.0 Host bridge: Intel Corporation: Unknown device 25d8 (rev b1)

                    00:02.0 PCI bridge: Intel Corporation: Unknown device 25f7 (rev b1)

                    00:03.0 PCI bridge: Intel Corporation: Unknown device 25e3 (rev b1)

                    00:04.0 PCI bridge: Intel Corporation: Unknown device 25f8 (rev b1)

                    00:05.0 PCI bridge: Intel Corporation: Unknown device 25e5 (rev b1)

                    00:06.0 PCI bridge: Intel Corporation: Unknown device 25f9 (rev b1)

                    00:07.0 PCI bridge: Intel Corporation: Unknown device 25e7 (rev b1)

                    00:10.0 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

                    00:10.1 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

                    00:10.2 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

                    00:11.0 Host bridge: Intel Corporation: Unknown device 25f1 (rev b1)

                    00:13.0 Host bridge: Intel Corporation: Unknown device 25f3 (rev b1)

                    00:15.0 Host bridge: Intel Corporation: Unknown device 25f5 (rev b1)

                    00:16.0 Host bridge: Intel Corporation: Unknown device 25f6 (rev b1)

                    00:1c.0 PCI bridge: Intel Corporation: Unknown device 2690 (rev 09)

                    00:1e.0 PCI bridge: Intel Corporation 82801BA/CA/DB/EB PCI Bridge (rev d9)

                    00:1f.0 ISA bridge: Intel Corporation: Unknown device 2670 (rev 09)

                    00:1f.1 IDE interface: Intel Corporation: Unknown device 269e (rev 09)

                    00:1f.3 SMBus: Intel Corporation: Unknown device 269b (rev 09)

                    01:00.0 PCI bridge: Intel Corporation: Unknown device 3500 (rev 01)

                    01:00.3 PCI bridge: Intel Corporation: Unknown device 350c (rev 01)

                    02:00.0 PCI bridge: Intel Corporation: Unknown device 3510 (rev 01)

                    02:02.0 PCI bridge: Intel Corporation: Unknown device 3518 (rev 01)

                    03:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    04:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    04:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    05:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    05:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    06:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    06:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

                    07:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

                    08:01.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

                    08:02.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

                    09:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    09:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    09:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    09:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    0a:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    0a:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    0a:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    0a:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

                    0c:00.0 PCI bridge: Intel Corporation: Unknown device 0330 (rev 07)

                    0c:00.2 PCI bridge: Intel Corporation: Unknown device 0332 (rev 07)

                    0d:0e.0 RAID bus controller: LSI Logic / Symbios Logic LSI Logic MegaRAID (rev 07)

                    12:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    13:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    13:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

                    14:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    14:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    15:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    15:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

                    16:0c.0 VGA compatible controller: ATI Technologies Inc: Unknown device 515e (rev 02)

                     

                    **********BEGIN /PROC/VMWARE/PCI*************

                    • 7. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
                      CiscoKid Hot Shot

                      I had a fun-filled weekend of nothing but server teardown and assembly trying to figure out what is causing this issue.  I have disabled the USB in the BIOS as suggested by Paul and still experience the issue.  I removed all the cards (2x1000MT Quad-ports and 1x1000PT Quad-ports) leaving just one of the 1000PT Quad-port and the LSI 320-2E only to experience the same issue where the C and D ports of the 1000PT quad-port NIC are slower than WAN links averaging above 600ms response times.

                       

                      I then thought it just maybe a hardware limitation and then installed Windows 2003 Standard, loaded the NIC drivers and then built teams for the NICs.  After all the configuration was complete with Windows 2003, all tests proved that all ports were operating under normal conditions showing reponse times of =/<1ms.

                       

                      I am pretty much out of ideas at this point I have created 2 separate teams where I have omitted ports C & D on the 1000PT quad-ports until this issue is resolved.  This is definitely something within VMware ESX as I proved that there is no issue with the hardware by all the different tests that I have performed and installing Windows 2003 without any issues.  Please help!!!

                      • 8. Re: Intel Pro 1000PT ( 1000 PT ) Quad-port perfomance issues
                        CiscoKid Hot Shot

                        Sorry all, here is a copy of the switch configuration for all the switchports that the VM NICs are connected to:

                         

                        interface GigabitEthernet0/1

                        description ***VMNIC***

                        power inline never

                        switchport trunk encapsulation dot1q

                        switchport trunk allowed vlan 10,50,100

                        switchport mode trunk

                        switchport nonegotiate

                        switchport voice vlan 100

                        speed 1000

                        duplex full

                        no mdix auto

                        spanning-tree portfast