Any way to enable jumbo frames on host NIC's without destroying them?

[golddiggie]

I was able to check my ESXi 4 u1's NIC's configuration last night and see that their MTU is set to 1500. I want to set them to MTU 9000 (jumbo frames) so that all traffic between the host and iSCSI device (and all other devices) is at the highest bandwidth possible. I've already set the iSCSI device's (using Openfiler) NIC's to use MTU 9000, and jumbo frames is enabled on my Gb switch. I did see some reference to being able to set the MTU when you originally configure the NIC's on the host, and about needing to destroy them in order to set them again. I'd rather not have to do that right now.

I'm just looking to eek out as much performance from the configuration I have. I'm in the process of pulling the VM's back off the iSCSI device so that I can reconfigure the volumes/LUNs on it. I made one large LUN originally, but want to carve it up into smaller ones now. Probably three ~600GB, with whatever is left over on the third LUN.

I do have ESXi 4 (244038) on the host, using the vCenter Server, with the vMA appliance online. I do have the PowerCLI software installed on my main workstation too.

I also enabled jumbo frames on my main workstation NICs and saw an immediate drop in the CPU load (one core was always at high usage, that's dropped to less than 50% of what it was)

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[weinstein5]

Have you enabled Jumbo Frames on the virtual switch?

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[golddiggie]

I don't believe so... How do you go about doing that, or checking it? Will that override the MTU value that shows up under the vMA when you get info on the NIC's?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[Eagle11]

You need to enable jumbo frame on virtual switch from the vcenter gui and then enable the vmkernel port group jumbo frame on each host using cli

*************

Nile Alkass

IT Systems Officer

Sent using BlackBerry Wireless Handheld Device.

[golddiggie]

Assume I've never done that before, so I have not seen where the setting is... I'm looking in the GUI but am not seeing anything that (I think) looks like either the jumbo frames or a MTU number...

Is this something that is enabled, at install/config time, automatically if the network switch the host connects to supports it? While I would prefer to stick with ESXi, I'm not opposed to loading up ESX if that's what it's going to take to get this enabled.

Just looking to have all aspects of the LAN/network running at optimal speeds here... This is the last item that needs to be tweaked to get that to happen.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[Eagle11]

follow this link to enable the jumbo frames for your virtual switch

http://www.interworks.com/blogs/kculwell/2010/01/29/enabling-jumbo-frames-vmware-esx-4

also i would recommend you to use Distribution Virtual Switches so that you dont repeat the setup of switches in every single host although the vmkernel set up is per host

[HendersonD]

I just went through this about two months ago. In order for Jumbo Frames to work they must be enabled end to end from storage to virtual machine. In my case these are the components that must have Jumbo Frames enabled:

Netapp filer - this is my iSCSI storage

Cisco 6509 core switch - this is where my Netapp storage is cabled to. My IBM Bladecenter switches are also connected to the 6509. I had to enable Jumbo Frames on all the ports in the 6509 that carry iSCSI traffic. These were ports from the Netapp and ports from the switches in my Bladecenter.

Blade center switches - my IBM Bladecenter has multiple Cisco switches in it. I had to enable jumbo frames on the two switches that carry iSCSI traffic.

Virtual switch - each vSwitch that handles iSCSI traffic needs to have Jumbo Frames enabled. This can be done from the command line, more information below

VMKernel port -for the kernel port that carries iSCSI traffic. This can be done from the command line, more information below

A few things to keep in mind:

For the 6509, Jumbo Frames can be enabled on a port by port basis. On the switches in my Bladecenter, Jumbo Frames is enabled on the entire switch. You need to check your switch documentation to see what your situation is.

Do not bother to enable Jumbo Frames for your public network. Remember that Jumbo Frames need to be enabled end to end in order for them to work. End do end on your public network is from your VMs all the way to laptops and desktops on your network. Since you will not be enabling Jumbo Frames on your desktop and laptop nics, there is no sense enabling them for your public network nics in VMWare

Jumbo Frames are not officially supported for ESX3.5 but are supported for vSphere

Actually enabling Jumbo Frames is what your original question was. You will have to look at your particular storage and physical switches to find the commands to enable Jumbo Frames. The best two articles I found on enabling Jumbo Frames for the vSwitch and Kernel ports are:

The bottom line is this:

You can easily enable Jumbo Frames in the proper vSwitch from the command line

In order to enable Jumbo Frames on the kernel port you must destroy the port first and then recreate it using the command line. You will need to

1. Put your ESX server in Maintenance Mode

2. Remove the storage

3. Delete the iSCSI kernel port

4. Via the command line issue two commands. One command makes the kernel port, the other command enables Jumbo Frames on the port. The articles I linked to will give you the exact commands

5. Add storage back

6. Take server out of maintenance mode

Before adding storage back, from the ESX command line you can do a Jumbo Frames ping on your storage, the command is in one of the articles I referenced. The ping should work correctly if you have enabled Jumbo Frames end to end

Good Luck

[golddiggie]

With just a single host (right now) maintenance mode isn't really much of an option.

I have to wait for the final VM to copy off of the iSCSI storage before I can do any more setting changes. I'm working to get at least the iSCSI traffic set to use jumbo frames. I've already made the settings to the switch (HP ProCurve 2510G-24) as well as the iSCSI solution (Openfiler running on a Dell PowerEdge 860 with the onboard NIC's bonded and set to jumbo, as well as an additional dual port Intel Gb NIC bonded and set to MTU 9000, and a third single port NIC set to MTU 9000).

Worst case, now that I have the info, when I'm setting up the next ESX/ESXi host server, I'll make sure to enable jumbo frames on all the vSwitches, and port groups that need it, before using it for anything. I'm hoping that before the end of the summer, I'll have the second host (and maybe something better for the iSCSI solution).

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[golddiggie]

Funny thing about the onboard NIC on the system I'm using for my ESXi 4u1 host... It's a Broadcom Gb NIC, but it doesn't support jumbo frames. Luckily, I have two other NIC's (one dual, the other quad port) that do support jumbo frames (both are Intel server class cards)... I'm giving some serious thought to changing how the vSwitches are configured, giving the NIC without jumb frame support to just the management network port group, and leaving the rest of the ports for the higher traffic loads... I'll have to see how things run as I've set them first... Hopefully, the changes I've just made will take care of the traffic/load to the iSCSI device...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.