Jumbo Frames

[jeffoutwest]

Hello all,

I am designing a vSphere environment consisting of 8 ESXi 4.0 hosts. Each host has a total of four GigE physical interfaces. All I/O traffic is sent out through Fibre Channel. HBAs. Two standard virtual switches using VLANs have been defined:

vSS0 will have two vmkernel ports, one for managment and the second for VMotion traffic. FT will not be used, DRS and VMotion will be configured.

vSS1 will have port groups for the virtual machine traffic, two vmcic's will be teamed to the upstream switch.

Given the assumption that all the switches will support jumbo frames, would you advise us to configure any or all of these interfaces for JF?

Any and all design recommendations would be appreciated.

Thanks, -Jeff

[kac2 · Accepted solution]

Jeff,

i just want to understand what you haev before I start making recommendations.

Each ESX host 4 1gb NICS that will be used for VM traffic, vMotion, Managment and A fiber-channel HBA for SAN traffic.

is that correct?

I think you have a good logic about your vSwitches. IMO, I would set vSwitch 1 for Managment and vMotion to be Active/Stand-by. For instance, if vmnic0 and vmnic3 were for vSwitch1. Managment = vmnic0 Active/ vmnic3 Stand-by, vMotion is vmnic3 Active vmnic 0 Stand-by. This way keeping traffic seperated from those 2 links and still being redundant.

moving on to jumbo frames. you ONLY want to use jumbo frames if it's supported end-to-end. I only enable jumbo frames for my SAN traffic on our iSCSI/NFS network. So I had to configure, the NICs on the host, the core switch, and the SAN. You DO NOT want to enable jumbo frames on your VM traffic NICs, because you have to configure the underlying physical NIC on the host, configure the NIC on the guest (if that's even possible), configure every switch for jumbo frames, then configure the desktop NIC trying to talk to the VM. Jumbo frames will only work if EVERY SINGLE hop is configured to be used for jumbo frames.

I wouldn't even attempt vMotion/Management for jumbo frames, but that's just my personal opinion.

Scott Lowe has a great blog and step-by-step of configuring jumbo frames w/ vSphere

[jeffoutwest]

Kendrick, thanks for taking the time to reply to my post.

Your understanding of my design is correct, with one addition. These ESXi hosts are IBM HS22 blades. There will be four switch modules available in the BladeCenter chassis, two new Nortel's and two older DLINK. Only the Nortel modules will have JF capabilities and will be dedicated to the VMotion and Managment virtual switches. All the switch modules are able to aggregate ports and configure VLANs, so the older DLINKS will be used for the virtual machine traffic. I will move forward with the design incorporating the Active/Standby configurations for the managment and VMotion nics.

You have clarified my understanding of enabling Jumbo Frames. I am going to attempt JF on the VMotion network but understand now that JF outside of the virtual environment would be daunting if not impossible.

Again, thank you for your recommendations. -Jeff

[pearlyshells]

Good day. I read your response on Jumbo Frames and your "personal preference" not to use Jumbo Frames on vMotion. could I ask why you decided not to use it on vMotion? We have a Gig network but were considering Jumbo for vMotion....but, if there isn't a significant advantage I don't want to do it. A vSphere instructor indicated that it was "Best practice" to enable Jumbo Frames on our vMotion network

[kac2]

Pearly,

It really all depends on your setup.

take a look at this on my website:

http://kendrickcoleman.no-ip.org/index.php/Tech-Blog/vsphere-host-nic-configuration.html

If you look at picture number 1, you have 2 dedicated NICS for vMotion. with this example you could put vMotion on jumbo frames because you have a redundant NIC dedicated to vMotion.

Picture number 2 is how I configure my hosts. This scenario uses 2 NICs. 1 NIC is set as Active for Service Console, while Stand-by for vMotion. The other is vice-versa. This will free up 2 NICs for other functions. So if NIC 2 dies, then both Service Console and vMotion will be using NIC 1. this is where Jumbo Frames will not work. The only way this would work is if you set your service console to use Jumbo frames, your desktop NIC to use Jumbo frames and your vCenter NIC to use Jumbo Frames.

So you have to weigh out the costs of having more NICs to have a redundany vMotion network and more configuration. I like to keep it simple and use Jumbo Frames for the Storage Network because that's where you're going to notice the biggest amount of gain.

[pearlyshells]

Thanks for your reply. We have a similar setup as your picture2 (sort of). We have our Service Console and vMotion on the same port group on a single vSwitch with 2 vNics connected. Our connection to the Fibre Channel SAN is GigEthernet. So, I wonder if there is much to be gained by going Jumbo. thanks.