Jumbo frames and TSO support in 3.5

[stexlollo]

Hello all

i saw day ago some not correct forums answers about Jumbo frames and TSO support in esx 3.5.

some thinks that jumbo frames 9000 byte are only support in guest vm with the enached vmxnet driver , like TSO

some thinks that jumbo and TSO are only supported in phisical nic, but not in virtual ethernet stack (example vmkernel for iscsi software iniziator or vm network tcp ip stack).

here the correct answer from vmware support:

Jumbo frames allow ESX Server 3.5 to send larger frames out onto the physical network. The network must support jumbo frames (end-to-end) for jumbo frames to be effective. Jumbo frames up to 9KB (9000 bytes) are supported. Like TSO, jumbo frames are supported in both the guest operating system and in the ESX Server kernel TCP/IP stack.

Before enabling jumbo frames, ensure the NIC or LOM supports jumbo frames.

Check with your hardware vendor before enabling jumbo frames on your platform. VMware supports jumbo frames with the following vendors: Intel (82546, 82571), Broadcom (5708, 5706, 5709), Netxen (NXB-10GXxR, NXB-10GCX4), and Neterion (Xframe, Xframe II, Xframe E).

To enable jumbo frames in a virtual machine, configure Enhanced VMXNET (supported on a limited number of guests) for the guest. Jumbo frames support is disabled by default in the VMkernel. In ESX Server 3.5, it can be enabled through the Service Console CLI. This configuration option is not available in ESX Server 3i.

EXAMPLE:

Enabling Jumbo Frames:

esxcfg-vmknic -a -i 192.168.128.2 -n 255.255.255.0 -m 9000 iscsi

Enabling Jumbo Frames on a vswitch:

esxcfg-vswitch -m 9000 vSwitch2

Check the configuration:

esxcfg-vswitch -l

esxcfg-nics -l

esxcfg-vmknic

i think this help all in confusion with jumbo and TSO support in esx 3.5 and when are supported.

by.

[bhadzik]

This comes from http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_3_server_config.pdf page 63

Jumbo Frames allow ESX Server 3 to send larger frames out onto the physical network.

The network must support Jumbo Frames end‐to‐end for Jumbo Frames to be effective.

Jumbo Frames up to 9kB (9000 Bytes) are supported. iSCSI with Jumbo Frames is not

supported.

[depping]

Indeed, jumbo frames are supported for the vmkernel and the vswitch, but not for your iscsi initiator. i wrote a blog about it:

http://www.yellow-bricks.com/2008/01/03/what-about-those-jumbo-frames/

Duncan

My virtualisation blog:

[stexlollo]

Hello i read the guide and i ask another support explanation: iscsi software iniziator supports all that vmkernel stack supports in the phisical nic so the answer is yes, ok all the end point must supports these feauturs:

the guide is out date.

Support for vmware Software ISCSI and Jumbo frames is only dependent on the NIC/ Physical Network and Iscsi target as I explain below.

Jumbo Frames allow ESX Server 3 to send larger frames out onto the physical network. <Correct>

The network must support Jumbo Frames end‐to‐end for Jumbo Frames to be effective. <Correct>

Jumbo Frames up to 9kB (9000 Bytes) are supported. iSCSI with Jumbo Frames is not <May be Outdated however I need to open a feature request and internally correct this as Software ISCI is not dependant on the NIC

throughput>

ESX server following enabling jumbo frames will not care if the traffic is passed/used by the VMware software initiator as Software ISCSI only emulates SCSI codes for storage across IP. The VMXnet/NIC and kernel have the capability to carry the larger 9000byte Frames and software ISCSI does not implement anything that manages this layer so Vmware Software iSCSI is supported with Jumbo frames enabled. I must state that this as you stated requires the ISCSI target / Physical Switch/hubs and network card to function with jumbo frames.

I believe that as you pointed out the ambiguity in the Configuration Guide I will need to open a feature request to update this Guide however I feel that we can move to close this SR as you should require no further details on Vmware ISCSI initiator compatability with ESX3.5 and Jumbo frames.

by all

[aroudnev]

(Maybe, it is too old topic, but anyway)

Just testing configuration:

ESXi (not ESX) on dell 2950, 2 x 1 Gbit NIC-s,

Jumbo frames enabled on vSwitch

Guest: Linux SuSe Enterprise 10 SP2 64bits.

iSCSI from the guest to NetApp.

results are strange:

• E1000 driver in Linux can't work with Jumbo Frames in vSwitch at all.

• enhanced NIC and Vmware tools driver can work with Jumbo Frames, but has some unidentified bug so that iSCSI (open-iSCSI in this case)

dies under the heavy READ (not WRITE).

• Everything works (iSCSI + multipath + lvm, with patched multipath service) well with MTU 1500 (on the guest).

So, again, something wrong with the enhanced drivers for the Linux (previous bug was related to the Paravirtialization support - it don't work properly,

making Java VM to look on the modern in-memory spinlocks).

Conclusion - be extremely careful with Ethernet emulation on Vmware guests - it is already second bug related to it (SLES9 SP4 required tso to be disabled to work properly with the MTU discovery).

[mkats]

Hello Everyone,

I'll do my best not to sound redundant, so I will leave my configuration information out of the equation as it appears we are all following the same school of thought.

I have configured my dev environement for Jumbo frames solely for the purpose of reducing vmotion overhead. It appears to work but not to its full potential, the migration will not exceed 4k. Is there a cap on Vmotion? Do I need to configure Jumbo and TSO on the VM's even though I am only using it for Vmotion?

Any insight would be greatly appreciated,

MK