4 Replies Latest reply on Apr 18, 2007 6:16 PM by georgerrata

    Help with Intel Pro 1000 PT and possible IRQ changes!!!

    CiscoKid Hot Shot

      Okay, I have been reading all the forums trying to understand the scale of my problem and I have pretty much drove straight into a brickwall.  I am currently building a whitebox unit that will be used for demonstration purposes for clients. 

       

      The motherboard is an ASUS DSBF-D12/SAS that has 2xPCI-X, 2xPCIe (x8) and 1xPCIe (x16) with an onboard LSI 1068 SAS controller.  The SAS controller has been disabled as I have installed a LSI Logic 320-2E controller as I have the standard 68-PIN U320 hard drives.  The motherboard is also equipped with dual-Intel 82563 gigabit adapters.  In addition to what is included on the board I also have 2 Intel Pro 1000PT Quad port PCIe adapters and 2 Intel Pro 1000MT Quad port PCI-X adapters.

       

      The issue that I am experiencing is that when I try to do a fresh install of the ESX 3.01 with all cards installed the VMKernel is not accessible and I get about 8 errors on the login screen that says that the VMKernel is not able to find a suitable MAC address and performance would be degradated (not exact verbiage).  I then proceed with applying the patches starting with ESX-2158032, ESX-1410076 and ESX-1006511.   After I apply the first two some of the original errors start going away and after I install ESX-1006511 ALL VMware related services start to take forever to start and stop.

       

      I originally thought it to be a problem with IRQ address so I did another fresh install of ESX 3.01 with only the LSI Login 320-2E card installed and the onboard Intel Pro 1000 NICs and everything is fine including starting and stopping any VMware related services.  The moment that I add any of the Intel Pro 1000PT PCIe cards all VMware related services start acting up again.

       

      One thing that I am noticing as I am adding the PCIe cards versus installing the ESX 3.01 with the cards already installed is that as I add the PCIe cards the BUS# and DEV# changes for the LSI Logic 320-2E.  If anyone has experienced this issue and knows of a way to resolve this would be most appreciated.

        • 1. Re: Help with Intel Pro 1000 PT and possible IRQ changes!!!
          Paul Lalonde Master

          Please post your output from:

           

          lspci

           

          cat /proc/vmware/pci

           

          cat /proc/vmware/interrupts

           

          IRQ sharing is a known issue on servers with both PCI-X and PCI-e chipsets, and you've sort've self-identified this by removing the PCI-e devices.

           

          Start with the commands above and we'll go from there.

           

          paul

          • 2. Re: Help with Intel Pro 1000 PT and possible IRQ changes!!!
            CiscoKid Hot Shot

            Okay I will give that a try.  Did you want me to perform the requested commands with all cards installed?  Also, should I enable the Plug&Play OS option in the OS?  I am not able to manually control the IRQ address in the BIOS and was told by ASUS that I should enable the Plu&Play option if I wanted to control the IRQs in the OS.  Thanks in advance.

            • 3. Re: Help with Intel Pro 1000 PT and possible IRQ changes!!!
              CiscoKid Hot Shot

              Following devices were detected for networking during the installation.  There are 2x4-port PCI-X, 2x4-port PCI-E  and 2xOnboard gigabit NICs:

               

              5:0:0 - PCI-X 1

              5:0:1 - PCI-X 1

              6:0:0 - PCI-X 1

              6:0:1 - PCI-X 1

              12:0:0 - PCI-X 2

              12:0:1 - PCI-X 2

              13:0:0 - PCI-X 2

              13:0:1 - PCI-X 2

              7:0:0 - Onboard 1

              7:0:1 - Onboard 2

               

              1.  Selected device address 7:0:0 as the COS VMNIC0 during installation.

              2.  Rebooted after successful installation.

              3.  Received the following errors on the login screen:

               

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

               

              4.  Installed Patch ESX-2158032.

              5.  Rebooted after ESX-2158032 installed.

              6.  Received the following error messages on the login screen:

               

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

                   0:00:00:53.871 cpu6:1038) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:53.871 cpu6:1038) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

               

              7.  Installed Patch ESX-1410076.

              8.  Rebooted after ESX-1410076 installed.

              9.  Received the following error messages on the login screen:

               

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

                   0:00:00:53.871 cpu4:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

                   0:00:00:53.871 cpu4:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address.  Various VMKernel subsystems will provide lower quality of service.

               

              10.  Installed Patch ESX-1006511.

              11.  Rebooted after ESX-1006511 installed.

              12.  Services started slowing down when detecting VMNic settings and took about 10 times as long to start each service.  Received the following error messages on the login screen:

               

                   0:00:00:30.033 cpu0:1024) PCI: 1650: Failed for 009:04.0

                   0:00:00:30.033 cpu0:1024) PCI: 1650: Failed for 009:04.1

                   0:00:00:30.035 cpu0:1024) PCI: 1650: Failed for 009:06.0

                   0:00:00:30.035 cpu0:1024) PCI: 1650: Failed for 009:06.1

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

                   0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

                   0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

               

              Below are the requested outputs:

               

              ******************BEGIN LSPCI********************

               

              00:00.0 Host bridge: Intel Corporation: Unknown device 25d8 (rev b1)

              00:02.0 PCI bridge: Intel Corporation: Unknown device 25f7 (rev b1)

              00:03.0 PCI bridge: Intel Corporation: Unknown device 25e3 (rev b1)

              00:04.0 PCI bridge: Intel Corporation: Unknown device 25f8 (rev b1)

              00:05.0 PCI bridge: Intel Corporation: Unknown device 25e5 (rev b1)

              00:06.0 PCI bridge: Intel Corporation: Unknown device 25f9 (rev b1)

              00:07.0 PCI bridge: Intel Corporation: Unknown device 25e7 (rev b1)

              00:08.0 System peripheral: Intel Corporation: Unknown device 1a38 (rev b1)

              00:10.0 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

              00:10.1 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

              00:10.2 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

              00:11.0 Host bridge: Intel Corporation: Unknown device 25f1 (rev b1)

              00:13.0 Host bridge: Intel Corporation: Unknown device 25f3 (rev b1)

              00:15.0 Host bridge: Intel Corporation: Unknown device 25f5 (rev b1)

              00:16.0 Host bridge: Intel Corporation: Unknown device 25f6 (rev b1)

              00:1c.0 PCI bridge: Intel Corporation: Unknown device 2690 (rev 09)

              00:1e.0 PCI bridge: Intel Corporation 82801BA/CA/DB/EB PCI Bridge (rev d9)

              00:1f.0 ISA bridge: Intel Corporation: Unknown device 2670 (rev 09)

              00:1f.1 IDE interface: Intel Corporation: Unknown device 269e (rev 09)

              00:1f.3 SMBus: Intel Corporation: Unknown device 269b (rev 09)

              01:00.0 PCI bridge: Intel Corporation: Unknown device 3500 (rev 01)

              01:00.3 PCI bridge: Intel Corporation: Unknown device 350c (rev 01)

              02:00.0 PCI bridge: Intel Corporation: Unknown device 3510 (rev 01)

              02:02.0 PCI bridge: Intel Corporation: Unknown device 3518 (rev 01)

              03:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              04:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              04:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              05:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              05:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              06:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              06:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

              07:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

              08:01.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

              08:02.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

              09:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              09:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              09:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              09:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              0a:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              0a:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              0a:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              0a:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

              0c:00.0 PCI bridge: Intel Corporation: Unknown device 0330 (rev 07)

              0c:00.2 PCI bridge: Intel Corporation: Unknown device 0332 (rev 07)

              0d:0e.0 RAID bus controller: LSI Logic / Symbios Logic LSI Logic MegaRAID (rev 07)

              10:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              11:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              11:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

              12:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              12:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              13:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              13:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

              16:02.0 VGA compatible controller: ATI Technologies Inc: Unknown device 515e (rev 02)

               

              ******************END LSPCI********************

               

              ******************BEGIN CAT /PROC/VMWARE/PCI********************

               

              Bus:Sl.F Vend:Dvid Subv:Subd Type     Vendor   ISA/irq/Vec P M Module   Name   

                                                             Spawned bus                     

              000:00.0 8086:25d8 1043:81db Host/PCI Intel     0/   /     A C                 

              000:02.0 8086:25f7 0000:0000 PCI/PCI  Intel        001       C                 

              000:03.0 8086:25e3 0000:0000 PCI/PCI  Intel        011       C                 

              000:04.0 8086:25f8 0000:0000 PCI/PCI  Intel        012       C                 

              000:05.0 8086:25e5 0000:0000 PCI/PCI  Intel        015       C                 

              000:06.0 8086:25f9 0000:0000 PCI/PCI  Intel        016       C                 

              000:07.0 8086:25e7 0000:0000 PCI/PCI  Intel        020       C                 

              000:08.0 8086:1a38 1043:81db 0x880    Intel                  C                 

              000:16.0 8086:25f0 1043:81db Host/PCI Intel                  C                 

              000:16.1 8086:25f0 1043:81db Host/PCI Intel                  C                 

              000:16.2 8086:25f0 1043:81db Host/PCI Intel                  C                 

              000:17.0 8086:25f1 1043:81db Host/PCI Intel                  C                 

              000:19.0 8086:25f3 1043:81db Host/PCI Intel                  C                 

              000:21.0 8086:25f5 1043:81db Host/PCI Intel                  C                 

              000:22.0 8086:25f6 1043:81db Host/PCI Intel                  C                 

              000:28.0 8086:2690 0000:0000 PCI/PCI  Intel        021       C                 

              000:30.0 8086:244e 0000:0000 PCI/PCI  Intel        022       C                 

              000:31.0 8086:2670 1043:81dc PCI/ISA  Intel                  C                 

              000:31.1 8086:269e 1043:81dc IDE      Intel    255/   /     A C                 

              000:31.3 8086:269b 1043:81dc SMBus    Intel    10/   /     B C                 

              001:00.0 8086:3500 0000:0000 PCI/PCI  Intel        002       C                 

              001:00.3 8086:350c 0000:0000 PCI/PCI  Intel        008       C                 

              002:00.0 8086:3510 0000:0000 PCI/PCI  Intel        003       C                 

              002:02.0 8086:3518 0000:0000 PCI/PCI  Intel        007       C                 

              003:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       004       C                 

              004:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       005       C                 

              004:01.0 111d:8018 0000:0000 PCI/PCI  0x111d       006       C                 

              005:00.0 8086:10a4 8086:10a4 Ethernet Intel     7/ 18/0x71 A V e1000    vmnic10

              005:00.1 8086:10a4 8086:10a4 Ethernet Intel    10/ 19/0x79 B V e1000    vmnic11

              006:00.0 8086:10a4 8086:10a4 Ethernet Intel    11/ 16/0x81 A V e1000    vmnic12

              006:00.1 8086:10a4 8086:10a4 Ethernet Intel     5/ 17/0x89 B V e1000    vmnic13

              007:00.0 8086:1096 1043:8217 Ethernet Intel     5/   /     A V e1000    vmnic0 

              007:00.1 8086:1096 1043:8217 Ethernet Intel    10/   /     B V e1000    vmnic1 

              008:01.0 1014:01a7 0000:0000 PCI/PCI  IBM          009       C                 

              008:02.0 1014:01a7 0000:0000 PCI/PCI  IBM          010       C                 

              009:04.0 8086:101d 8086:1000 Ethernet Intel    255/   /     A V e1000    vmnic2 

              009:04.1 8086:101d 8086:1000 Ethernet Intel    255/   /     B V e1000    vmnic3 

              009:06.0 8086:101d 8086:1000 Ethernet Intel    255/   /     A V e1000    vmnic4 

              009:06.1 8086:101d 8086:1000 Ethernet Intel    255/   /     B V e1000    vmnic5 

              010:04.0 8086:101d 8086:1000 Ethernet Intel    255/   /     A V e1000    vmnic6 

              010:04.1 8086:101d 8086:1000 Ethernet Intel    255/   /     B V e1000    vmnic7 

              010:06.0 8086:101d 8086:1000 Ethernet Intel    255/   /     A V e1000    vmnic8 

              010:06.1 8086:101d 8086:1000 Ethernet Intel    255/   /     B V e1000    vmnic9 

              012:00.0 8086:0330 0000:0000 PCI/PCI  Intel        013       C                 

              012:00.2 8086:0332 0000:0000 PCI/PCI  Intel        014       C                 

              013:14.0 1000:0408 1000:0002 RAID     Symbios   5/ 18/0x71 A V megaraid2 vmhba0 

              016:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       017       C                 

              017:00.0 111d:8018 0000:0000 PCI/PCI  0x111d       018       C                 

              017:01.0 111d:8018 0000:0000 PCI/PCI  0x111d       019       C                 

              018:00.0 8086:10a4 8086:10a4 Ethernet Intel     7/ 16/0x81 A V e1000    vmnic14

              018:00.1 8086:10a4 8086:10a4 Ethernet Intel    11/ 17/0x89 B V e1000    vmnic15

              019:00.0 8086:10a4 8086:10a4 Ethernet Intel    11/ 16/0x81 A V e1000    vmnic16

              019:00.1 8086:10a4 8086:10a4 Ethernet Intel     5/ 17/0x89 B V e1000    vmnic17

              022:02.0 1002:515e 1043:81de Display  ATI      11/   /     A C                 

               

              ******************END CAT /PROC/VMWARE/PCI********************

               

              ******************BEGIN CAT /PROC/VMWARE/INTERRUPTS********************

               

              Vector    PCPU  0    PCPU  1    PCPU  2    PCPU  3    PCPU  4    PCPU  5    PCPU  6    PCPU  7

              0x21:         434          0          0          0          0          0          0          0 COS irq 1 (ISA edge), , VMK vmnic17, VMK vmnic15, VMK vmnic13

              0xdf:     2589394    2597919    2589621    2589621    2591882    2591882    2593810    2593811 VMK timer

              0xe1:           0          0          0          0          0          0          0          0 VMK monitor

              0xe9:       14118          1         76       2155       1273       1442          0        728 VMK resched

              0xf1:           1          1          1          0          1          1          1          1 VMK tlb

              0xf9:        4607          0          0          0          0          0          0          0 VMK noop

              0xfc:           0          0          0          0          0          0          0          0 VMK thermal

              0xfd:           0          0          0          0          0          0          0          0 VMK lint1

              0xfe:           0          0          0          0          0          0          0          0 VMK error

              0xff:           0          0          0          0          0          0          0          0 VMK spurious

               

              ******************END CAT /PROC/VMWARE/INTERRUPTS********************

               

              Once again, I appreciate all of your efforts regarding this resolution.

              • 4. Re: Help with Intel Pro 1000 PT and possible IRQ changes!!!
                georgerrata Novice

                We had a similar problem but not the exact problem.

                 

                We disabled the Onboard NICs before the install and the Sevice console then bound itself to the external NICs and then worked just fine, after install we enabled the onboards and they worked just fine to.

                 

                Oops sorry just read your post again and I see you have done that.... hmmm how to delete this post

                 

                Message was edited by:

                        georgerrata