Nexus 1000 interface resets every 30 seconds

[towlesd]

I working on deploying Nexus 1000 on our ESX hosts while migrating to Emulex 10Gig Cards. Its been interseting to say the least but we almost have it working after firmware and dirver issues. Currently i'm seeing a odd behavior that I've been able to recreate. Normally I have two 10 gig ports added to the ESX and added to the distributed virtual switch (DVS) and a virtual port-channel created between the Nexus 5000's they are connected. As as part of my trouble shooting i've removed one of the nics from the DVS uplink. this creates a behavior i can recreate. While the remaining nic is the only nic in the port-channel, and the only nic connected to the DVS.

This causes the nexus 1000 to reset its link with the nexus 5000 every 1minute like clockwork. it will be down for 30 secs, work for 30 secs, then repeat.

heres an example of the log from the nexus 1000.

2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)

2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps

2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full

2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on

2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on

2010 Dec 7 12:12:40 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual

2010 Dec 7 12:12:40 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk

2010 Dec 7 12:13:07 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)

2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps

2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full

2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on

2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on

2010 Dec 7 12:13:44 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual

2010 Dec 7 12:13:44 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk

2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)

2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps

2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full

2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on

2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on

2010 Dec 7 12:14:41 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual

2010 Dec 7 12:14:41 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk

and the following log from esx host.

Dec 7 11:43:02 nkuvmhost9 vmkernel:

Dec 7 11:43:30 nkuvmhost9 vmkernel: 3:21:34:09.172 cpu8:4531)Need to send MAC Move for Inband Port

Dec 7 11:43:30 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 60 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 70 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 200 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 268 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 274 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 275 from the ltl 18

Dec 7 11:44:02 nkuvmhost9 vmkernel:

Dec 7 11:44:31 nkuvmhost9 vmkernel: 3:21:35:10.172 cpu8:4104)Need to send MAC Move for Inband Port

Dec 7 11:44:31 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 60 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 70 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 200 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 268 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.377 cpu8:4319)Not removing sys vlan 274 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.377 cpu8:4319)Not removing sys vlan 275 from the ltl 18

Dec 7 11:45:02 nkuvmhost9 vmkernel:

Dec 7 11:45:32 nkuvmhost9 vmkernel: 3:21:36:11.172 cpu8:4531)Need to send MAC Move for Inband Port

Check my Blog for other Nexus issues I've been solving along the way.

www.ChrisTowles.com

[lwatta]

I'd highly recommend opening a TAC case. We have seen number of issues where host drivers cause problems with LACP negotiation.

louis

[towlesd]

Thanks for the reply but I already had a case open with cisco. We found the problem and my networking guy as write a good post telling the problem better than I. The issues comes down to you can not pass VEM management packets down a port-channel that has 1 nic in it. it causes the port-channel to reset.

Moderator note: link to irrelevant website reported, and removed

My Network Guy - http://cjthenetworkguy.blogspot.com/2010/12/emulex-oce10102-cards.html

[lwatta]

Ah! didn't know it was an attempt at single nic port-channel 🙂

Yeah that is a known issue. The problem is that negotiating LACP is the responsibility of the VSM. The VSM needs to talk to the upstream switch and the VEM to get the port-channel up. When you have just one link what happens is that the link between the VEM and VSM drops when it attempts to start the port-channel. When you have two links you'll see that one link joins the port-channel and the other will cycle between w and h states till the port-chanel is established. Once the PC is established, VSM swaps all control traffic to the link in the PC and then adds the second interface to the PC.

This is being fixed in the next version of code. We will have the ability for the VEM to directly negotiate LACP with the upstream switch.

louis