ESxi 5.5 NIC Jumbo Frames, 10GB nics + Link Aggregartion , HP Switch

[haris2887]

Hello All.

I am having a few problems getting Jumbo frames to work on my ESX servers.

Basically I have 3 servers with the same V-switch configuration. I have enable jumbo frames on the iSCSI vlan on my switch. but when I try to do a ping anything larger than the default 1500 size from one host iscsi interface to another I get packet is to long to transmit error.

Is there something special I have to do for HP switches ?

Here are some screen shots.

[dhanarajramesh]

you have to config jumpo frame on switch as well.

example:

5406zl-A# configure

  5406zl-A(config)# vlan 3jumbo

  5406zl-A(config)# end

  5406zl-A# sho vlan 3

  Status and Counters - VLAN Information - Ports - VLAN 3

    VLAN ID : 3

    Name : VLAN3

    Status : Port-based

    Voice : No

    Jumbo : Yes <-----

and also I would suggest you to look your ISCSI kernal NIC teaming. better go thru this documents http://www.vmware.com/files/pdf/iSCSI_design_deploy.pdf and also I suggest change the NIC teaming from IP hash to Virtual port ID based or use explicit fail over order and dedicate one adapter for each ISCSI kernel

[Rubeck]

What switch model and FW version are you using? 

What size has been configured as "jumbo" on the switch....?

"show jumbos" 

/Rubeck

[MKguy]

First off, using etherchannel link aggregation with iSCSI is a very... odd way of setting up storage connectivity. If you configure iSCSI binding it's even unsupported as you can only have one active uplink per iSCSI vmkernel port:

Seriously, Stop Using Port Channels for vSphere Storage Traffic - Wahl Network

Why can you not use NIC Teaming with iSCSI Binding? | VMware vSphere Blog - VMware Blogs

The frames are dropped at the local interface before reaching the physical network.

Specify the outgoing vmkernel interface with the -I option of vmkping or it might try to route the packet through the wrong interface which is not jumbo enabled.

Also in some cases a host reboot may be necessary to enable jumbo frames.

[haris2887]

I believe nic teaming for iSCSI multipathing is setup correcrtly as per screen shots below.

[haris2887]

SHow jumbos on the switch shows

And the switch is running the latest FW . for 8212 ZL switch...

[MKguy · Accepted solution]

It would be correct if you weren't using a port channel on top of it. You're running a port channel on the physical switch and mix in non-channeled interfaces on the ESXi side.

When you're use IP-hash load balancing on a vSwitch, all attached port groups/vmk interfaces must use route based on IP-hash and must have all uplinks active.

Remember, the physical switch has no idea that you set the NICs to unsed for the iSCSI interfaces and assumes everything is just one channel. So it will happily send iSCSI traffic destined for vmk1 through uplink vmk5, which you set to unsed there.

Have a look at these 2 articles which state this is unsupported:

VMware KB:    Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with E...

VMware KB: Host requirements for link aggregation for ESXi and ESX

The only load balancing option for vSwitch or vDistributed Switch that can be used with EtherChannel is IP HASH.

Do not configure standby or unused uplinks with IP HASH load balancing.

[haris2887]

Thanks. You were spot ON.

As soon as i removed Etherchannel and rebooted the hosts.

It all started working ...

Thanks