Enable Jumbo Frames? In what Order?

[mrizos]

Hi Guys,

I have 3 ESXi hosts with 4 VM's (so far) on an iSCSI SAN. Performance is absolutely amazing right now, but I'll be adding 20 more VM's spread out across the 3 hosts.

Do you guys think I need to enable Jumbo frames for the iSCSI traffic? If so, what order should I change the MTU's? For example...should I change the SAN iSCSI ports to 9000, then change the switch ports, then change the servers....or???

What would you suggest?

I'm have authorization to shutdown the 4 VM's this weekend if I need to.

Thanks!

[mittim12]

I'm not sure there is a need to do it in a specific order as much as the need to make sure you do it accross all connections involved.

If you found this or any other post helpful please consider the use of the Helpful/Correct buttons to award points

Twitter: http://twitter.com/mittim12

[mrizos]

Yeah, I'm going to enable it on:

-the host iSCSI NICS

-the physical iSCSI switch ports

-the iSCSI ports on the SAN

I suppose I can just do a....esxcfg-vswitch -m 9000 vSwitch1 (2 and 3) on each ESXi host.

[mittim12]

Don't forget to set it up on the vmkernel port too. Here is a KB for setting up jumbo frames, http://kb.vmware.com/kb/1007654.

If you found this or any other post helpful please consider the use of the Helpful/Correct buttons to award points

Twitter: http://twitter.com/mittim12

[mrizos]

I already have the vmkernels created. Is there anyway I can just change their MTU's instead of creating new vmkernels?

[mittim12]

I'm almost positive it has to be set when they are created and can not be modified after the fact.

If you found this or any other post helpful please consider the use of the Helpful/Correct buttons to award points

Twitter: http://twitter.com/mittim12

[a_p_]

is correct, you have to re-create the iSCSI vmkernel portgroups.

I could be wrong, but from what I read so far it seems, there's no real benefit using Jumbo Frames for 1 GBit networks, but significant benefit for 10 GBit/s networks.

André

[mrizos]

Well I'm using 1GB connections, not 10GB, so that's nice to know.

[mrizos]

I'm really wondering if jumbo frames are really worth having to destroy all my iSCSI vmkernels...

[mrizos]

I've decided to rebuild the vmkernels for jumbo frames after reading about them all night. I was going to enable them in this order:

ESX Host then Switch then iSCSI ports on the SAN

Does anyone see a problem with this order? I'm just sorta worried that the 9 KB frame leaving one interface won't be accepted on the 1.5 KB interface, or am I wrong there? Does the 1.5 KB interface break up the 9 KB frame? I'll only have the MTU's mixed for 1-2 days (over the weekend).

[AndreTheGiant · Accepted solution]

Jumbo Frame need to be enabled end-to-end... so there isn't a preferred order... until a single piece is not enable, they will not used.

Remember that you have to enable also on the vSwitch (in this case you do not need to destroy it).

Also for some kind of physical switches it may required a cold reboot.

Andre

[mrizos]

Thanks Andre, I didn't know they're ignored until every link is set at 9000.