Etherchannel on DL380 G6 using e1000 and bnx2 nics

[tjcooper]

I can create an etherchannel with two intel nics, but when I add the broadcom it fails. Can you mix nic vendors? Is anyone teaming a mixture of intel and broadcom?

!
interface Port-channel2
switchport
switchport access vlan 200
switchport mode access
no ip address
end

!
interface GigabitEthernet12/46
description testServer vSwitch1 VMNIC7
switchport
switchport access vlan 200
switchport mode access
no ip address
spanning-tree portfast
channel-group 2 mode on
end

!
interface GigabitEthernet13/36
description testServer vSwitch1 VMNIC4
switchport
switchport access vlan 200
switchport mode access
no ip address
spanning-tree portfast
channel-group 2 mode on
end

!
interface GigabitEthernet13/35
description testServer vSwitch1 VMNIC1
switchport
switchport access vlan 200
switchport mode access
no ip address
spanning-tree portfast
channel-group 2 mode on
end

[rickardnobel]

How is the vSwitch setup? Do you use IP hash load balancing?

[chriswahl]

Make sure you set the load distribution policy to src-dst-ip using this command:

port-channel load-balance src-dst-ip

--- This is a global command that sets the distribution policy to hash the source / destination IP

Also, set the port group load balancing policy to "Route based on IP Hash"

[rickardnobel]

Chris Wahl wrote:

Make sure you set the load distribution policy to src-dst-ip using this command:

port-channel load-balance src-dst-ip

--- This is a global command that sets the distribution policy to hash the source / destination IP

I have always wondered about this recommendation from VMware. I think the idea is to somewhat "match" the method used inside the ESXi host when distributing the traffic with IP hash, but it does not really make sense for me. The 802.3ad (link aggregation) does not define how the load balancing should be done and it is always up to the sender to determine how the ethernet frames should be distributed among the links in the link aggregation group.

Since many physical switches has correct support for 802.3ad and works with vSwitch IP Hash, but still can not set/change the balancing method I think this might not be needed at all. For example many HP Procurve devices uses only MAC addresses in their link aggregation and it is not possible to change this.

Also, even if actually changing to the src-dst-ip method on a Cisco switch it is not sure that the exact algorithm is the same as vSwitch IP Hash (which uses something like last byte in IP addresses XOR:ed together modulus number of physical vmnics). The Cisco switch might do something similar, but not exact the same and it should still really work as well.

[milton123]

Agree with rickardnobel ,

Also have a look this kb.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100404...

Cheers, Udin

[tjcooper]

I have an etherchannel setup with 2 intel nics. The load-balance is correct (see below). The issue is specifically mixing intel and broadcom in the same channell.

--------

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address
  MPLS: Label or IP

--------

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi10/22(P) Gi10/25(P)
2      Po2(SU)          -        Gi12/46(P) Gi13/36(P)
3      Po3(SD)          -

--------

[tjcooper]

I dont think you can mix and match NIC vendors, at least not the ones in a DL380 G6. I can create a team with all intel and all broadcom, but not a Mix.

[rickardnobel]

tjcooper wrote:

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi10/22(P) Gi10/25(P)
2      Po2(SU)          -        Gi12/46(P) Gi13/36(P)
3      Po3(SD)          -

--------

Which portgroup goes into the ESXi host? (There is no support for LACP in the VMware vSwitches.)

Could you also provide a screenshot of the vSwitch setup in the host?

[tjcooper]

po2 is the ESX host, the other is a mail server.

The load balancing is Route Based on IP Hash.

Attached is the screenshot of the nicteaming settings on the vSwitch.

I dont think the teaming would matter in the sense the etherchannel woudl still be created properly, but receive traffice would not be balanced.

[rickardnobel]

You have three vmnics in the vSwitch NIC teaming but only two configured in the physical switch?

Have you changed the default failback setting to No?

[tjcooper]

The settings are correct, failback is set to no.

Whatever nic I add first becomes teh vendor that will team (primary), the odd man out is suspended. I cant seem to get around it.

Thank you for the reply

[rickardnobel]

In your physical switch setup you have named the ports as connecting to VMNIC1, 4 and 7, however in the vSwitch configuration you have VMNIC0, 1 and 4.

Could there be a configuration mismatch here?

[tjcooper]

No. I was in the processes of moving ports around to distribute the nics accross 3 modules.

I am currently running two vSwitches comprised of all intel and all broadcom nics in etherchannels. The router indicates all channels are up and healthy.

[rickardnobel]

tjcooper wrote:

I am currently running two vSwitches comprised of all intel and all broadcom nics in etherchannels. The router indicates all channels are up and healthy.

Do you say that you no longer have problems with the etherchannels?

[tjcooper]

I am saying I can create them using NICs from the same vendor without any issues.

[rickardnobel]

tjcooper wrote:

I am saying I can create them using NICs from the same vendor without any issues.

It "should" not matter if the NICs are from the same vendor or not when using vSwitch with IP Hash, but it could of course be possible.

You are sure there is no misconfiguration between the vmnic-names to the physical switch ports? Any such errors will cause very difficult problems to troubleshoot.

Do you have CDP enabled on the vSwitch and physical switches?

[tjcooper]

I think it has to do with settings on the NICs. I found a thread where it says the broadcom and intel "Flow Control" settings dont match. I now need to find the shell commands to pull those settings and change them if they are not the same.

[tjcooper]

I think the issue is flow control (see blow). I dont know in what direction, or both, flow control works. Does my cisco only negotiate RX, or do they negotiate both TX and RX, or neither. In either case, it appears send flow control is not the same.

So, do I manually set the Send, turn off auto and set both, set them to what?

ESXi

Pause parameters for vmnic0 (<- Broadcom)

Autonegotiate:  on

RX:             off

TX:             off

~ # ethtool --show-pause vmnic7 (<-Intel)

Pause parameters for vmnic7:

Autonegotiate:  on

RX:             on

TX:             off

Cisco:

Port              Send FlowControl      Receive FlowControl  RxPause TxPause

                   admin    oper           admin    oper

-----              -------- --------            -------- --------    ------- -------

Gi13/15         desired  off               off      off         0       0

SCYKHQ651301#show flowcontrol int gi12/24

Port              Send FlowControl      Receive FlowControl  RxPause TxPause

                   admin    oper           admin    oper

-----              -------- --------            -------- --------    ------- -------

Gi12/24         desired  on              off      off         0       0