Enhanced vmxnet, E1000, Flexible, Jumbo Frames

[eagleh]

ESX3.5 U2 + Windows 2003 server R2 (Data Centre edition)

I have checked all VMs and realized I have 3 different types of vNIC: Enhanced vmxnet, E1000, Flexible (some VMs were migrated from physical boxes)

My question is: Since Enhanced vmxnet supports Jumbo Frames, what if my switch doesn't? Would the VM performance suffer? Should I switch it to E1000 or Flexible? From time to time, I see high CPU usage on "System Process". I am just wondering if this "vmxnet" could cause this problem?

Many thanks,

Eh

[jose_maria_gonz]

Hi Eagleh,

For Jumbo Frames to work, like in a fiscal environment, you must have Jumbo Frames enable end-to-end, that is, vNICs, physical NICs, ESX Servers and physical switch. Otherwise Jumbo Frames won´t work properly.

rgds,

J.

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

-

---

El Blog de Virtualizacion en Español

http://josemariagonzalez.es

[eagleh]

Thanks Jose. So my question is what if my environment doesn't meet this end-to-end Jumbo Frames requirement, would the VM performance somehow suffer from its Enhanced vmxnet network adapter? Or should not matter?

[chrisfmss]

To Jumbo frame enable, you to activate them in the VM and on the vswitch. For enable on the vswitch, you need this command option: esxcfg-vswitch -m=9000. With -l option, it will show you MTU value for all vswitch.

[jose_maria_gonz]

Hi Eagleh,

I have not been in that case scenario where you use the Enhanced vmxnet network adapter but your environment doesn’t meet an end-to-end Jumbo Frames setup. So I cannot tell you if that would affect performance.

Common sense tells me that if you start sending packet from a VM with 9000MTU and your physical switch can only deal with 1500MTU your physical switch would have to do a lot of catching up by breaking down 9000MTU frames onto 1500MTU frames which would certainly increase your physical switch CPU uses.

Rgds,

J.

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

-

---

El Blog de Virtualizacion en Español

http://josemariagonzalez.es

-

---

[Scott_Barbee]

<snip>

Common sense tells me that if you start sending packet from a VM with 9000MTU and your physical switch can only deal with 1500MTU your physical switch would have to do a lot of catching up by breaking down 9000MTU frames onto 1500MTU frames which would certainly increase your physical switch CPU uses.

<snip>-----

For clarification: If a physical switch is not capable of jumbo frames (eg 9000 MTU), it will discard packets above its MTU (likely 1500).

[SuryaVMware]

It dosen't matter if you are using E1000, Enhanced of Flexi as long as your haven't configured your Guest OS/VM to transmit Jumbo Frames. if your environment is not ready for Jumbo Frames dont enable Jumbo frames in VMware environment (vSwitches, VMs and GuestOSs) you should be alright.

The conclusion : Enhanced VMXnet will not enable and transmit JF by default. It requires configuration even in the virtual environment.

-Surya

[SuryaVMware]

were you asking for a clarification or giving clarification?

if you are asking for a clarification your understanding is wrong. If a switch dosen't support JF it will not drop those Frames it will break them donw in to 1500 and transmit it which will increase in CPU overhead with the Physical Switch.

-Surya

[polysulfide]

Jumbo frames is just the MTU setting which can be set on any TCP stack regardless of the adapter. What the vmxnet2 offers is TSO or TCP Segment Offload support.

Enabling TSO should reduce the CPU utilization of the VM.(results vary) Like jumbo frames, TSO also needs to be enabled on your ESX server and also configured in your OS.

Having a default high MTU will cause a very brief negotiation when you initially establish TCP connections but it shouldn't make a noticable difference. Determining appropriate MTU is part of the protocol, no worries.

If it was useful, give me credit

Jason White - VCP

[SuryaVMware]

Regarless of the the adapter?? no way ..not all the adapters and switches support JumboFrames, so you need the adapter supporting it. I understand that TCP stack of a particular OS needs to support it to. However, the adapter and the driver should be aware of the functionality.

I agree with TSO part you are taling about enabling will reduce the CPU utilization of the VM. However, it is enabled by default on ESX Server and as well as in most of the Operating systems all you need is a TSO capible adapter, which vmxnet2 is, so there is absolute zero configuration required to enable TSO in the VMware Virtualization environment.

-Surya

[mcowger]

You do realize that jumbo frames and a large TCP MTU are not the same thing, right?

Jumbo frames are at layer 2, and support a large frame (usually ~9000 bytes, but some switches can go even larger). A large TCP MTU can be limited by your frames size, but its not the same thing....

--Matt

[polysulfide]

Matt, you are correct about Jumbo frames being layer two hence "frame" lol. Most of my issues with configuration have been related to MTUs and not HW jumbo frame support so I concede that I fell prey to tunnel vision. Thank you for putting me back on track

As for TSO, it is enabled by default on ESX versions that support it provided that the VMK port was created by the installer or GUI. When manually creating VMK interfaces, VMware recomends disabling TSO unless it's needed. Windows doesn't always have TCP Chimney enabled by default, it depends on version and SP you're running. I don't think it would be safe to say "just install vmxnet2 and you have TSO working." If I were implementing TSO I would verify uisng esxcfg-vmknic -l that TSO is enabled and check the "TCP Chimney" Registry key before testing.

If it was useful, give me credit

Jason White - VCP

[mcowger]

^^^^^^ Good advice right there.

/me has always wondered why its called 'Chimney' support in Windows.

--Matt