Jumbo frame questions

[nov1ce]

Hi,

If I enable jumbo frames on a standard vSphere switch (as well as on a corresponding physical switch):

1. Do I need to change MTU settings on the VMotion interface too (since it's set to 1500 right now), or the vSphere switch settings will *overwrite* interface settings?

2. The same question is for VMs. Do I need to enable jumbo frames on all VMs that are connected to that switch?

The reason I'm asking is that we started observing occasional spikes of %DRPRX% on *some* VMs. The moment I set "Jumbo Packet" to 9000 on a VM's vmxnet3 interface, %DRPRX% drops to 0.

I also don't quite understand why only some VMs are affected? Is this an indication that affected VMs do require jumbo frames?

Say I have 10 VMs on a same host (different operating systems, but all with vmxnet3 and the default MTU 1500). All 10 are using vmnic1, but only two VMs report %DRPRX. If I adjust MTU on those two to 9000 %DRPRX disappears.

Many thanks.

[virtualDD]

I'm not sure about the VMs part. But on the vSwitch you need to specify the MTU for each vmkernel as well. (Use case: some vmkernel traffic like management might go through a routed network that does not support MTU > 1500; so you'll want that on 1500 and the rest on 9000)

I'm interested in the VM behaviour tough. If you create a new vm now that everything is changed to MTU 9000 does the vmxnet3 configure this as well or does it come with 1500 by default?

[ITaaP]

I assume you mean you enabled jumbo frames within the guest OS since it is not done at the vmxnet3 adapter level? And yes, it would need to be done to all VMs in the guest OS that you want jumbo frames enabled. Although I have rarely seen the need for this, but it all depends on the type of traffic your VMs are handling.

[SureshKumarMuth]

As per my understanding , jumbo frames should be enabled end to end to avoid packet drops also it causes bottlenecks

consider you have a server and end point. Server has MTU 1500 and all other devices (switched till end point has MTU 9000) , when a packet sent from endpoint with size greater than 1500 to server , since server supports only MTU 1500, the packet gets dropped vice versa won't be an issue.

so , its generally suggested to have jumbo frames enabled end to end so that there wont be any packet drops.

[dineshgoundar]

It is best practice to have jumbo frames enabled for vMotion network. Also, similar to what everyone else has said, you should enable jumbo frames end to end. Also, what version of VMware tools are you running? Do you have RSS enabled on the NICs of the VMs that experience the packet drops? Long shot but have a look at these articles, may be helpful.

Large packet loss at the guest operating system level on the VMXNET3 vNIC in ESXi (2039495) | VMware...

Poor network performance or high network latency on Windows virtual machines (2008925) | VMware KB

RUSH POST: VMware Tools and RSS Incompatibility Issues - Virtualize Business Critical Applications -...