broadcom NIC running at 10MB full duplex

[RUG201110141]

I have a server (ESX 3.0.2) with dual port broadcom 1000MB nics in it. I actually have a dozen ESX servers with the same NIC. In one of the ESX servers one of the ports on one of the dual port broadcom nics is negotiating at 10MB full duplex. I've recabled the network interface. I've had the networking people check the port for any issues. I've updated firmware of the machine. I still can't get the machine to auto-negotiate at 1000MB's. When I set it to 1000MB full duplex manually the network interface disconnects. Is there any trick to getting it to run at 1000MB.

I know if ESX 2.5.x I sometimes had to go into modules.conf and manually set the speed of the network interfaces, especially broadcom nics to get them to run at 1000MB. Has anybody had to do that with ESX 3.x.

[titaniumlegs]

Wild guess:

Is this an Extreme Alpine switch?

We're seeing the same kind of behaviour with 2 different Alpine 3804s with more than just ESX. Other switches, no problem. Even cheap ones.

Whatever the switch, have the network guys check the speed on the switch port. We were seeing 100m on ESX (IBM with Broadcom) and 10m on the switch port. Another device with a negotiation problem was a NetApp FAS3050 (Intel GbE chips).

Also, we saw half duplex in some caes.

Really weird. Extreme support has long since expired, so we don't have access to their KB or downloads.

[dkfbp]

I havent experienced what you are describing. We are also using broadcom nics on most of our servers. Try using a different network cable and connect it to another switchport. If that does not work I would use my service contract on the server to get the faulty hardware replaced.

When you run 1000mbit you need to set it to autonegotiate. You should never force 1000mbit.

[LarsLiljeroth · Accepted solution]

I had the same problem with one of our Ibm X3850 and in the end we got IBM to replace the NIC, then it worked again.

/Lars

[RUG201110141]

I've already tried different switch ports and reran all new cable. It's set to auto-negotiate and I've had my network team look at the switch ports and they indicate no problem. The switchs are Cicso Catalyst 6513's. I guess it's replace the onboard nic time.

[Ulaval]

\|| When you run 1000mbit you need to set it to autonegotiate. You should never force 1000mbit.

On page 21 of www.vmware.com/pdf/vi_performance_tuning.pdf

I read :

Most modern network interface cards can operate in multiple modes (such as 10,

100, or 1000Mbps; half duplex or full duplex). Make sure the network interface

cards are in full duplex mode and are configured at their maximum possible

bandwidth.[/b]

So, should we set it to Full or Auto ???

Sylvain

[cspaz]

Have you tried hard coding the link speed/duplex into the 6513? The commands are 'link speed 1000' and 'duplex full' from the interface on the 6513 (assuming you want gigabit/full).

[dkfbp]

Always set 1000mbit to autoneogotiate! It is part of the RFC standard to do that. Never force 1000mbit.

[e12pilot]

Always set 1000mbit to autoneogotiate! It is part of

the RFC standard to do that. Never force 1000mbit.

I agree, especially with Broadcom NICs and Cisco 65xx switches. We have done testing with both settings, and when hard-coding the Broadcom's @ 1000/full and the Cisco port @ 1000/full we could only push about 60-80Mb/sec through the port.

Other vendors may correctly "negotiate" their Gigabit master/slave settings when you hardcode 1000/full (which is required per the RFC), but the Broadcom/Cisco configuration really benefits from an Auto/Auto setup. Never seen it not negotiate correctly, unless the NIC is bad.

[parkerc930]

I am have the same issue with one of my NICs on ESX 4. 8 servers in production and 6 NICs in each with only one showing up 10 full.

These are also Broadcom NICs. Did anyone find any other solution than a bad Card?