I’m looking for “official” instructions for enabling jumbo frames on an ESXi 4.1 host with fibre channel storage. I don’t know if the instructions are different or if anything else needs to be adjusted depending on your storage.
I was looking at the instructions from this page here and it looks pretty straightforward. It also looks as though you can keep your existing port groups rather than have to remove them and then re-create them. So do these instructions pretty much cover it for the host side?
I saw this page for iSCSI storage so that got me thinking if the procedure differed depending on the storage.
Wasisnt wrote:
What is the reason for having to recreate them?
If you are on 4.1 you do not need to recreate anything, just enable through the CLI.
And can you enable Jumbo Frames on a standard port group or is it just for VMkernel port groups?
You could enable on both VM and vmkernel portgroups. On both you must also do it on the "virtual nics", that is: on the Vmkernel virtual Nics and the virtual nics inside the guest operating systems. On Windows 2003 server I think you must use VMXNET3, but on Windows 2008 the e1000 should support Jumbo frames too.
As has been noted before just make sure that all layer two devices in the path is configured to allow large frames. You could use this to confirm from both Vmkernel and guests: http://rickardnobel.se/archives/992
Are you doing FCoE or pure FC? Jumbo frames are for ethernet networks.
Earl Gay wrote:
Are you doing FCoE or pure FC? Jumbo frames are for ethernet networks.
I agree and since the question is about ESXi 4.1 only fully-hardware offloaded FCoE is supported with CNA adapters - and I think the configuration steps in the articles (jumbo frames on vSwitches and vmkernel interface) does not apply even on FCoE.
I was just informed that its not FCoE so does that mean it cant be done? Is there any documentation that supports this?
Jumbo frames do not apply to pure fibre channel. It's for storage that uses ethernet networks like iSCSI.
We were just discussing this and we have pure FC and also are using an IP network so it will be over Ethernet. Im new here and still figuring the place out!
Can I just follow the instuctions from those links then or do I need to do anything else?
Wasisnt wrote:
We were just discussing this and we have pure FC and also are using an IP network so it will be over Ethernet.
Do you access any storage through ethernet (iSCSI or NFS)? If not, you should not need to worry about Jumbo Frames.
I just did some more checking so we are not concerned about the storage traffic so the FC wont be an issue.
I also noticed on this page it says to delete and recreate your vmkernel interfaces. Is it because this is for 4.0?
http://blogs.vmware.com/networking/2010/03/jumbo-frames-in-vsphere-40.html
"Use the esxcfg-vmknic command to delete and then add a vmkernel interface with an MTU of 9000."
This page says you can change the setting on an existing vmkernel port.
"You can run the following command if vmkernel port already exist."
# esxcfg-vmknic -m 9000 "portgroupname"
So basically I would like to know if I can take our existing vSwitches and VMkernel ports and setup Jumbo Frames without having to remove and recreate anything? Is it just a matter of running the commands on each of the vSwitches and VMkernel port groups?
Here is the setup for one of our hosts
Vmconsole4 - 10.100.10.54
vSwitch0
Port groups
VM Network
Management Network (vmk0 - 10.100.10.54)
vSwitch1
Port group
VMotion-VMkernel (vmk2 – 192.168.15.5)
vSwitch2
Port group
DMZ
vSwitch3
Port group
DHCP
Now based on our configuration would these be the correct commands to enter if we are just doing the first 2 vSwitches? And since we are using 4.1 do I not need to delete and recreate any vSwitches or VMkernel ports?
vSwitches
# esxcfg-vswitch -m 9000 vSwitch0
# esxcfg-vswitch -m 9000 vSwitch1
Vmkernel ports
# esxcfg-vmknic -m 9000 VM Network
# esxcfg-vmknic -m 9000 Management Network
# esxcfg-vmknic -m 9000 VMotion-VMkernel
Wasisnt wrote:
Now based on our configuration would these be the correct commands to enter if we are just doing the first 2 vSwitches? And since we are using 4.1 do I not need to delete and recreate any vSwitches or VMkernel ports?
vSwitches
# esxcfg-vswitch -m 9000 vSwitch0
# esxcfg-vswitch -m 9000 vSwitch1
Vmkernel ports
# esxcfg-vmknic -m 9000 VM Network
# esxcfg-vmknic -m 9000 Management Network
# esxcfg-vmknic -m 9000 VMotion-VMkernel
If you follow this sort of process, you will end up enabling Jumbo Frames on your management network, the VM networks, etc..
Jumbo Frames must be end to end, and whilst this is feasible with storage, it's infeasible on general networks. It's very unlikely every desktop connecting to your server is going to be configured to talk Jumbo Frames. It's therefore likely that enabling Jumbo Frames on your servers will only cause problems.
Your subject discusses "fibre channel storage" and we've already confirmed - Jumbo Frames don't exist on an FC network. There are places that carefully planned Jumbo Frames can be beneficial but it involves careful planning that I feel is beyond the scope of what you're discussing here.
I'd recommend taking a step back, and asking "why" exactly you are seeking Jumbo Frames. If it's "for storage performance" I'll come back to the point that they don't apply to FC.
We will be enabling Jumbo Frames on the switch side as well. We have 2 virtual machines\appliances that do the VM backup as well as a TSM server doing VM data backups as well. We want to enable Jumbo Frames for the backup VMs as well as some of the VM servers. The FC storage is out of the equation so we can forget about that part. The purpose of this is to basically improve VM and backup performance.
Here is what was sent to me when I was tasked with the job.
"The storage is presented to the backup VM (called PHDVBA) by another VM called OpenFiler (an open-source NAS appliance). OpenFiler's access to the actual storage system is via FC - OpenFiler acts as a NAS gateway to the backup system. The connection between PHDVBA and OpenFiler would benefit from jumbo frames."
Will it hurt anything to enable Jumbo Frames on the management and VM network?
Like I said Im new here so I dont know the configuration all to well yet so Im figuring it out as I go along!
My understanding of enabling jumbo frame is that it can support transmition of large frames but as long as the frame is greater than 64 bytes there will not be any problem. PC can be plugged into a Jumbo frame enabled switch port and should not have any adverse effect.
Jumbo frame is only applicable to ethernet network and has no effect on FC and must be enabled end to end.
Wasisnt wrote:
And since we are using 4.1 do I not need to delete and recreate any vSwitches or VMkernel ports?
Yes that is correct. If you have upgraded the vCenter server you could even configure it through the GUI even on ESXi 4.1:
http://rickardnobel.se/archives/1242
However, as several has noted above, there are some risks involved with enabling jumbo frames in environments where you do not control the whole switching path, and for example the Management network could be such - that is you might not know from all places you will connect with the vSphere Client.
For vMotion it could be reasonable since it is a small well controlled network which sends a lot of data and might very well work better with Jumbo Frames. For IP based storage like iSCSI and NFS this could improve performance as well.
Our vCenter server is on 4.1 so it will have to be command line.
Do we have to delete and recreate the vSwitches and port groups via the command line? So does that mean Im reading this part wrong?
You can run the following command if vmkernel port already exist.
# esxcfg-vmknic -m 9000 "portgroupname"
What is the reason for having to recreate them?
And can you enable Jumbo Frames on a standard port group or is it just for VMkernel port groups?
Wasisnt wrote:
What is the reason for having to recreate them?
If you are on 4.1 you do not need to recreate anything, just enable through the CLI.
And can you enable Jumbo Frames on a standard port group or is it just for VMkernel port groups?
You could enable on both VM and vmkernel portgroups. On both you must also do it on the "virtual nics", that is: on the Vmkernel virtual Nics and the virtual nics inside the guest operating systems. On Windows 2003 server I think you must use VMXNET3, but on Windows 2008 the e1000 should support Jumbo frames too.
As has been noted before just make sure that all layer two devices in the path is configured to allow large frames. You could use this to confirm from both Vmkernel and guests: http://rickardnobel.se/archives/992
Ok, so just to be clear and maybe to close out this discussion ![]()
Here is one of the hosts we want to make the change on.
Vmconsole4 - 10.100.10.54
vSwitch0
Port groups
VM Network
Management Network (vmk0 - 10.100.10.54)
vSwitch1
Port group
VMotion-VMkernel (vmk2 – 192.168.15.5)
So these would be the commands I would run?
vSwitches
# esxcfg-vswitch -m 9000 vSwitch0
# esxcfg-vswitch -m 9000 vSwitch1
Vmkernel ports
# esxcfg-vmknic -m 9000 VM Network
# esxcfg-vmknic -m 9000 Management Network
# esxcfg-vmknic -m 9000 VMotion-Vmkernel
Wasisnt wrote:
So these would be the commands I would run?
It certainly looks correct. ![]()
You must however try it out to be really sure, watch out for case-sensitive names and spaces in the portgroup names.
I was reading over your last post and noticed you mentioned Vmkernel virtual NICs and virtual NICs inside the guest operating systems.I get the vitrual NICs inside the guest OS but what do you mean by Vmkenel NICs?
I was also looking at the virtual NICs in the VMs and some are VMXNET3, VMXNET2 (Enhanced) and Flexible.
The Flexible ones are on Windows 2003 and the VMXNET2 (Enhanced) is on Windows 2008
Do I need to change the type of NICs on these?
I found this on the VMware site:
Flexible — The Flexible network adapter identifies itself as a Vlance adapter when a virtual machine boots, but initializes itself and functions as either a Vlance or a VMXNET adapter, depending on which driver initializes it. With VMware Tools installed, the VMXNET driver changes the Vlance adapter to the higher performance VMXNET adapter.
VMXNET 2 (Enhanced) — The VMXNET 2 adapter is based on the VMXNET adapter but provides some high-performance features commonly used on modern networks, such as jumbo frames and hardware offloads. This virtual network adapter is available only for some guest operating systems on ESX/ESXi 3.5 and later.
It sounds like VMXNET 2 (Enhanced) will be fine but doesnt make it clear for Flexible
