VMware Cloud Community
CiscoKid
Enthusiast
Enthusiast

Help with Intel Pro 1000 PT and possible IRQ changes!!!

Okay, I have been reading all the forums trying to understand the scale of my problem and I have pretty much drove straight into a brickwall. I am currently building a whitebox unit that will be used for demonstration purposes for clients.

The motherboard is an ASUS DSBF-D12/SAS that has 2xPCI-X, 2xPCIe (x8) and 1xPCIe (x16) with an onboard LSI 1068 SAS controller. The SAS controller has been disabled as I have installed a LSI Logic 320-2E controller as I have the standard 68-PIN U320 hard drives. The motherboard is also equipped with dual-Intel 82563 gigabit adapters. In addition to what is included on the board I also have 2 Intel Pro 1000PT Quad port PCIe adapters and 2 Intel Pro 1000MT Quad port PCI-X adapters.

The issue that I am experiencing is that when I try to do a fresh install of the ESX 3.01 with all cards installed the VMKernel is not accessible and I get about 8 errors on the login screen that says that the VMKernel is not able to find a suitable MAC address and performance would be degradated (not exact verbiage). I then proceed with applying the patches starting with ESX-2158032, ESX-1410076 and ESX-1006511. After I apply the first two some of the original errors start going away and after I install ESX-1006511 ALL VMware related services start to take forever to start and stop.

I originally thought it to be a problem with IRQ address so I did another fresh install of ESX 3.01 with only the LSI Login 320-2E card installed and the onboard Intel Pro 1000 NICs and everything is fine including starting and stopping any VMware related services. The moment that I add any of the Intel Pro 1000PT PCIe cards all VMware related services start acting up again.

One thing that I am noticing as I am adding the PCIe cards versus installing the ESX 3.01 with the cards already installed is that as I add the PCIe cards the BUS# and DEV# changes for the LSI Logic 320-2E. If anyone has experienced this issue and knows of a way to resolve this would be most appreciated.

Reply
0 Kudos
4 Replies
Paul_Lalonde
Commander
Commander

Please post your output from:

lspci

cat /proc/vmware/pci

cat /proc/vmware/interrupts

IRQ sharing is a known issue on servers with both PCI-X and PCI-e chipsets, and you've sort've self-identified this by removing the PCI-e devices.

Start with the commands above and we'll go from there.

paul

Reply
0 Kudos
CiscoKid
Enthusiast
Enthusiast

Okay I will give that a try. Did you want me to perform the requested commands with all cards installed? Also, should I enable the Plug&Play OS option in the OS? I am not able to manually control the IRQ address in the BIOS and was told by ASUS that I should enable the Plu&Play option if I wanted to control the IRQs in the OS. Thanks in advance.

Reply
0 Kudos
CiscoKid
Enthusiast
Enthusiast

Following devices were detected for networking during the installation. There are 2x4-port PCI-X, 2x4-port PCI-E and 2xOnboard gigabit NICs:

5:0:0 - PCI-X 1

5:0:1 - PCI-X 1

6:0:0 - PCI-X 1

6:0:1 - PCI-X 1

12:0:0 - PCI-X 2

12:0:1 - PCI-X 2

13:0:0 - PCI-X 2

13:0:1 - PCI-X 2

7:0:0 - Onboard 1

7:0:1 - Onboard 2

1. Selected device address 7:0:0 as the COS VMNIC0 during installation.

2. Rebooted after successful installation.

3. Received the following errors on the login screen:

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:51.871 cpu1:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

4. Installed Patch ESX-2158032.

5. Rebooted after ESX-2158032 installed.

6. Received the following error messages on the login screen:

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

0:00:00:53.871 cpu6:1038) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:53.871 cpu6:1038) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

7. Installed Patch ESX-1410076.

8. Rebooted after ESX-1410076 installed.

9. Received the following error messages on the login screen:

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

0:00:00:53.871 cpu4:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

0:00:00:53.871 cpu4:1037) UTIL: 815: Status 0xbad0001 Trying to get a valid VMKernel MAC Address. Various VMKernel subsystems will provide lower quality of service.

10. Installed Patch ESX-1006511.

11. Rebooted after ESX-1006511 installed.

12. Services started slowing down when detecting VMNic settings and took about 10 times as long to start each service. Received the following error messages on the login screen:

0:00:00:30.033 cpu0:1024) PCI: 1650: Failed for 009:04.0

0:00:00:30.033 cpu0:1024) PCI: 1650: Failed for 009:04.1

0:00:00:30.035 cpu0:1024) PCI: 1650: Failed for 009:06.0

0:00:00:30.035 cpu0:1024) PCI: 1650: Failed for 009:06.1

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.0

0:00:00:30.040 cpu0:1024) PCI: 1650: Failed for 010:04.1

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.0

0:00:00:30.042 cpu0:1024) PCI: 1650: Failed for 010:06.1

Below are the requested outputs:

******************BEGIN LSPCI********************

00:00.0 Host bridge: Intel Corporation: Unknown device 25d8 (rev b1)

00:02.0 PCI bridge: Intel Corporation: Unknown device 25f7 (rev b1)

00:03.0 PCI bridge: Intel Corporation: Unknown device 25e3 (rev b1)

00:04.0 PCI bridge: Intel Corporation: Unknown device 25f8 (rev b1)

00:05.0 PCI bridge: Intel Corporation: Unknown device 25e5 (rev b1)

00:06.0 PCI bridge: Intel Corporation: Unknown device 25f9 (rev b1)

00:07.0 PCI bridge: Intel Corporation: Unknown device 25e7 (rev b1)

00:08.0 System peripheral: Intel Corporation: Unknown device 1a38 (rev b1)

00:10.0 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:10.1 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:10.2 Host bridge: Intel Corporation: Unknown device 25f0 (rev b1)

00:11.0 Host bridge: Intel Corporation: Unknown device 25f1 (rev b1)

00:13.0 Host bridge: Intel Corporation: Unknown device 25f3 (rev b1)

00:15.0 Host bridge: Intel Corporation: Unknown device 25f5 (rev b1)

00:16.0 Host bridge: Intel Corporation: Unknown device 25f6 (rev b1)

00:1c.0 PCI bridge: Intel Corporation: Unknown device 2690 (rev 09)

00:1e.0 PCI bridge: Intel Corporation 82801BA/CA/DB/EB PCI Bridge (rev d9)

00:1f.0 ISA bridge: Intel Corporation: Unknown device 2670 (rev 09)

00:1f.1 IDE interface: Intel Corporation: Unknown device 269e (rev 09)

00:1f.3 SMBus: Intel Corporation: Unknown device 269b (rev 09)

01:00.0 PCI bridge: Intel Corporation: Unknown device 3500 (rev 01)

01:00.3 PCI bridge: Intel Corporation: Unknown device 350c (rev 01)

02:00.0 PCI bridge: Intel Corporation: Unknown device 3510 (rev 01)

02:02.0 PCI bridge: Intel Corporation: Unknown device 3518 (rev 01)

03:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

04:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

04:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

05:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

05:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

06:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

06:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

07:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (rev 01)

08:01.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

08:02.0 PCI bridge: International Business Machines PCI-X to PCI-X Bridge (rev 02)

09:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

09:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:04.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:06.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0a:06.1 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (LOM) (rev 01)

0c:00.0 PCI bridge: Intel Corporation: Unknown device 0330 (rev 07)

0c:00.2 PCI bridge: Intel Corporation: Unknown device 0332 (rev 07)

0d:0e.0 RAID bus controller: LSI Logic / Symbios Logic LSI Logic MegaRAID (rev 07)

10:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

11:00.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

11:01.0 PCI bridge: Integrated Device Tech: Unknown device 8018 (rev 04)

12:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

12:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

13:00.0 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

13:00.1 Ethernet controller: Intel Corporation Intel PRO/1000 PT Quad Port Gigabit Ethernet Controller (rev 06)

16:02.0 VGA compatible controller: ATI Technologies Inc: Unknown device 515e (rev 02)

******************END LSPCI********************

******************BEGIN CAT /PROC/VMWARE/PCI********************

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name

Spawned bus

000:00.0 8086:25d8 1043:81db Host/PCI Intel 0/ / A C

000:02.0 8086:25f7 0000:0000 PCI/PCI Intel 001 C

000:03.0 8086:25e3 0000:0000 PCI/PCI Intel 011 C

000:04.0 8086:25f8 0000:0000 PCI/PCI Intel 012 C

000:05.0 8086:25e5 0000:0000 PCI/PCI Intel 015 C

000:06.0 8086:25f9 0000:0000 PCI/PCI Intel 016 C

000:07.0 8086:25e7 0000:0000 PCI/PCI Intel 020 C

000:08.0 8086:1a38 1043:81db 0x880 Intel C

000:16.0 8086:25f0 1043:81db Host/PCI Intel C

000:16.1 8086:25f0 1043:81db Host/PCI Intel C

000:16.2 8086:25f0 1043:81db Host/PCI Intel C

000:17.0 8086:25f1 1043:81db Host/PCI Intel C

000:19.0 8086:25f3 1043:81db Host/PCI Intel C

000:21.0 8086:25f5 1043:81db Host/PCI Intel C

000:22.0 8086:25f6 1043:81db Host/PCI Intel C

000:28.0 8086:2690 0000:0000 PCI/PCI Intel 021 C

000:30.0 8086:244e 0000:0000 PCI/PCI Intel 022 C

000:31.0 8086:2670 1043:81dc PCI/ISA Intel C

000:31.1 8086:269e 1043:81dc IDE Intel 255/ / A C

000:31.3 8086:269b 1043:81dc SMBus Intel 10/ / B C

001:00.0 8086:3500 0000:0000 PCI/PCI Intel 002 C

001:00.3 8086:350c 0000:0000 PCI/PCI Intel 008 C

002:00.0 8086:3510 0000:0000 PCI/PCI Intel 003 C

002:02.0 8086:3518 0000:0000 PCI/PCI Intel 007 C

003:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 004 C

004:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 005 C

004:01.0 111d:8018 0000:0000 PCI/PCI 0x111d 006 C

005:00.0 8086:10a4 8086:10a4 Ethernet Intel 7/ 18/0x71 A V e1000 vmnic10

005:00.1 8086:10a4 8086:10a4 Ethernet Intel 10/ 19/0x79 B V e1000 vmnic11

006:00.0 8086:10a4 8086:10a4 Ethernet Intel 11/ 16/0x81 A V e1000 vmnic12

006:00.1 8086:10a4 8086:10a4 Ethernet Intel 5/ 17/0x89 B V e1000 vmnic13

007:00.0 8086:1096 1043:8217 Ethernet Intel 5/ / A V e1000 vmnic0

007:00.1 8086:1096 1043:8217 Ethernet Intel 10/ / B V e1000 vmnic1

008:01.0 1014:01a7 0000:0000 PCI/PCI IBM 009 C

008:02.0 1014:01a7 0000:0000 PCI/PCI IBM 010 C

009:04.0 8086:101d 8086:1000 Ethernet Intel 255/ / A V e1000 vmnic2

009:04.1 8086:101d 8086:1000 Ethernet Intel 255/ / B V e1000 vmnic3

009:06.0 8086:101d 8086:1000 Ethernet Intel 255/ / A V e1000 vmnic4

009:06.1 8086:101d 8086:1000 Ethernet Intel 255/ / B V e1000 vmnic5

010:04.0 8086:101d 8086:1000 Ethernet Intel 255/ / A V e1000 vmnic6

010:04.1 8086:101d 8086:1000 Ethernet Intel 255/ / B V e1000 vmnic7

010:06.0 8086:101d 8086:1000 Ethernet Intel 255/ / A V e1000 vmnic8

010:06.1 8086:101d 8086:1000 Ethernet Intel 255/ / B V e1000 vmnic9

012:00.0 8086:0330 0000:0000 PCI/PCI Intel 013 C

012:00.2 8086:0332 0000:0000 PCI/PCI Intel 014 C

013:14.0 1000:0408 1000:0002 RAID Symbios 5/ 18/0x71 A V megaraid2 vmhba0

016:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 017 C

017:00.0 111d:8018 0000:0000 PCI/PCI 0x111d 018 C

017:01.0 111d:8018 0000:0000 PCI/PCI 0x111d 019 C

018:00.0 8086:10a4 8086:10a4 Ethernet Intel 7/ 16/0x81 A V e1000 vmnic14

018:00.1 8086:10a4 8086:10a4 Ethernet Intel 11/ 17/0x89 B V e1000 vmnic15

019:00.0 8086:10a4 8086:10a4 Ethernet Intel 11/ 16/0x81 A V e1000 vmnic16

019:00.1 8086:10a4 8086:10a4 Ethernet Intel 5/ 17/0x89 B V e1000 vmnic17

022:02.0 1002:515e 1043:81de Display ATI 11/ / A C

******************END CAT /PROC/VMWARE/PCI********************

******************BEGIN CAT /PROC/VMWARE/INTERRUPTS********************

Vector PCPU 0 PCPU 1 PCPU 2 PCPU 3 PCPU 4 PCPU 5 PCPU 6 PCPU 7

0x21: 434 0 0 0 0 0 0 0 COS irq 1 (ISA edge), , VMK vmnic17, VMK vmnic15, VMK vmnic13

0xdf: 2589394 2597919 2589621 2589621 2591882 2591882 2593810 2593811 VMK timer

0xe1: 0 0 0 0 0 0 0 0 VMK monitor

0xe9: 14118 1 76 2155 1273 1442 0 728 VMK resched

0xf1: 1 1 1 0 1 1 1 1 VMK tlb

0xf9: 4607 0 0 0 0 0 0 0 VMK noop

0xfc: 0 0 0 0 0 0 0 0 VMK thermal

0xfd: 0 0 0 0 0 0 0 0 VMK lint1

0xfe: 0 0 0 0 0 0 0 0 VMK error

0xff: 0 0 0 0 0 0 0 0 VMK spurious

******************END CAT /PROC/VMWARE/INTERRUPTS********************

Once again, I appreciate all of your efforts regarding this resolution.

Reply
0 Kudos
georgerrata
Contributor
Contributor

We had a similar problem but not the exact problem.

We disabled the Onboard NICs before the install and the Sevice console then bound itself to the external NICs and then worked just fine, after install we enabled the onboards and they worked just fine to.

Oops sorry just read your post again and I see you have done that.... hmmm how to delete this post Smiley Happy

Message was edited by:

georgerrata

Reply
0 Kudos