Intel PRO/1000 NICs not working

[MarcBouchard]

New installation of ESX4 in our lab on "unsupported" hardware. The system has 3 NICs. 2 Intel PRO/1000 and an onboard Broadcom NIC. At installation I noticed something strange but figured I'd run thru it anyway. To summarize the issue, only one NIC works (vmnic2, one of the Intel adapters). On the physical switch, I see the link is up on all 3, but ESX says two of them are down

I need those extra adapters for iSCSI support etc...

Any help would be much appreciated!

[MarcBouchard]

Are your ports turned on at the Physcial switch end?

Yes and I took the cable from the one NIC that works and tried in the other 2 NICs and they still say "down"...

[MarcBouchard]

# ethtool vmnic0

Settings for vmnic0:

Supported ports:

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: pumbag

Wake-on: g

Current message level: 0x00000001 (1)

Link detected: no <==== THIS IS THE BUG

#

[MarcBouchard]

Here's a screenshot of what I'm seeing even though all 3 are connected to my switch and the port lights are on at both ends (on the NIC and on the switch).

[admin]

You could also try posting this to the Networking community discussion, as I don't believe this is specific to Installation, and there may be experts over there that don't monitor this forum.

[gamania]

have you tried the latest intel nic driver released for vsphere?

[MarcBouchard]

have you tried the latest intel nic driver released for vsphere?

Thanks for the suggestion. But it didn't help... I installed, rebooted, same result.

[ChaosEnvy]

I've had this problem before. For me it was 1 of three things.

1. The port speed wasn't set correctly on the switch or on the vnic inside of the VIC.

2. The ports on the switch were configured on the wrong vlan. For instance, I had a switch port that was set to use vlan107, but needed to be on vlan1000

3. Something is wrong with the cable, Fluke it.

From your screen shot and your configs, I'd say your install is working just fine. I would tend to think that the problem is outside of esx on the Network level.

Big D

[MarcBouchard]

I would agree with you, however if I take the cable from the NIC that does work and try it in the other 2 NICs, nothing comes up... Plus the fact that until I reinstalled with ESX 4, my ESX 3.5 setup worked perfectly so it has to be ESX related...

Here's what the switch says (port g23 is one of the ports plugged in to one of the NICs that VMware says has no link).

[ChaosEnvy]

What else is shown for that port if you click on details. I'm used to using cisco os. The port shows 1000 as the speed, can you set the speed to autonegotiate or set the speed to 1000 on the nic in Virtual Center? It definately looks like a network issue.. If it is ESX, it'd be the drivers or incompatable hardware. Thats my 2 cents anyhow.

Also, am I understanding that the nic that does show the link as up in VC is a different nic than the nic in question?

Big D

[MarcBouchard]

What else is shown for that port if you click on details. I'm used to using cisco os. The port shows 1000 as the speed, can you set the speed to autonegotiate or set the speed to 1000 on the nic in Virtual Center? It definately looks like a network issue.. If it is ESX, it'd be the drivers or incompatable hardware. Thats my 2 cents anyhow.

Also, am I understanding that the nic that does show the link as up in VC is a different nic than the nic in question?

Big D

This hardware configuration has always worked (and is still working on a 2nd ESX box). If it was a network issue, the switch wouldnt show a link up. ESX is the only thing that says it's not, when the hardware says otherwise. More than likely incompatible hardware BUT why is it that my ONE NIC that works is of the same model with no issue? And again, all of this worked yesterday with ESX 3.5.

Yes the one that says UP is a different physical NIC, it's the other PRO1000PT. So one PRO1000PT works, the other doesnt, and the on board NIC doesnt either. But they all register as UP on the physical switch.

[ChaosEnvy]

My bad, wasn't trying to get a rise. I was simply trying to look at it from different angles. Okay. So your integrated nic is working, and one of the two PCI Nic's is working, yet they are identical nics. Shut Down, swap nics and see if the problem still exists. I mean you are kind of at the point of drawing straws to eliminate possibilities. You have only converted the one server, so you don't know if the same behavior will happen on the next server. The problem could be hardware related. I'm just saying. Maybe reload esx on this server, migrate over, and see if you can repeat the problem on another server(given that they are identical nuts and bolts) and see if you get the same outcome. This may lead to some insight or an answer.

Big D

[MarcBouchard]

Oh no rise here! I'm just extremely frustrated at this thing. You know troubleshooting is fun when there is some logic to it, and this doesn't.

I have only ONE NIC working, and the 2nd one (identical make/model) doesnt. But they are all recognized by ESX, they worked fine yesterday so obviously it's a software issue. I guess it seems them but cant initialize them (and doesn't tell me anywhere in any log either). But it CAN get one of them to work. Makes no sense...

Thanks for taking the time to offer suggestions, much appreciated. Throwing ideas is what gets it resolved eventually.

[ChaosEnvy]

Are all three of your nic cards, single port?

[MarcBouchard]

Yes, 2 add-on PCI Express cards single port PRO/1000 PT and an onboard Intel gigabit adapter (exact chipsets are in a previous post with all the dumps, you can also see them in the image of Vcenter.

[Roelofz]

I've had the same using an Pro 1000GT desktop adapter (even reordered one, suspecting the card), build aside a Pro 1000 PT.

It turned out to be the vswitch config, after throwing away the vswitch and rebuilding (just some unneeded pushing and pulling in the networking config, did'nt make any sense to me) the vswitch, it worked.

Maybe you can get it to work like that!

Regards,

Roelof

[MarcBouchard]

I've had the same using an Pro 1000GT desktop adapter (even reordered one, suspecting the card), build aside a Pro 1000 PT.

It turned out to be the vswitch config, after throwing away the vswitch and rebuilding (just some unneeded pushing and pulling in the networking config, did'nt make any sense to me) the vswitch, it worked.

Maybe you can get it to work like that!

Regards,

Roelof

I did a complete reinstall with ESXi instead of ESX and everything works A1. I will rebuild my 2nd server with ESX and try that. However since the problem was showing during installation, when the vswitches arent created yet... Good suggestion though.

[Paul_Lalonde]

Marc, I'm curious about interrupt routing. Can you post the output from the following?

cat /proc/vmware/interrupts

cat /proc/vmware/chipsets

grep -i 'e1000' /var/log/vmkernel (.1, .2, etc)

Thanks

Paul

[MarcBouchard]

Marc, I'm curious about interrupt routing. Can you post the output from the following?

cat /proc/vmware/interrupts

cat /proc/vmware/chipsets

grep -i 'e1000' /var/log/vmkernel (.1, .2, etc)

Thanks

Paul

I reinstalled both systems with ESXi and they both work A1. I did try to install ESX on the 2nd server, and I got similar problems. Except that both PRO1000 PT worked, but not the on board... Something fishy, but why would it work with ESXi?

Thanks to all for your assistance.

[Datto]

Just a thought -- if you get back to re-installing regular ESX 4.0 on your boxes, try disabling all the USB ports in the System BIOS of your box before starting the ESX 4.0 install and see if that clears up the NIC link problem.

Datto

[rago60]

Hello,

I've just the same problem: Intel Motherboard with onboard Intel 82567 GBit nic and two 82572 PCIe GBit nics. I installed two identically ESX-Server. One shows the onboard and one of the two PCIs cards als down the other displays one of the cards as down. I installed the driver for the onboard nics as posted at vmware, reboot both server but nothing changed. Before updating or reinstalling the two servers, both were running with three networkcards without a problem. I changed the cards in the slots, changed the switch and the cable: nothing changed. I disabled the onboard nic so that the working PCIe card doesn't work anymore. I enabled the onboard nic and the one PCIe card works fine.

There must be a failure in the software. Can anyone help me?