Advanced Networking Performance Options

Some of the advanced networking options available in vSphere 4.0 are reviewed in this paper. Many of these options control trade-offs between latency, throughput, CPU utilization, and reliability (e.g., dropped packets). It is not possible to optimize all of these at the same time, so option defaults are chosen to be suitable for the vast majority of applications. These options are provided to meet the stricter requirements of other applications. Advanced options often have subtle side effects, or merely move an issue from one area to another. Therefore it is recommended that VMware Support be engaged before changing such options, especially for production machines.

There are over 100 options that can be set under Configuration → Advanced Settings → Net. Of these, the ones listed below are most likely to be useful for tuning networking performance. Many of the others are for internal testing or enable unreliable features.

All of the options listed here take integer values. For the “Boolean” ones only the default value is shown: 0 for “false”, and 1 for “true”. Other parameters are shown with their default, minimum, and maximum values.

Parameter Name	(Default, Minimum, Maximum)	Description
MaxPortRxQueueLen	(80, 1, 500)	Maximum length of the Rx queue for virtual ports whose clients support queueing. Possibly should be increased if Rx packet drops are seen in the port connected to a VM. Relevant only for e1000 vNICs used with Fault Tolerance (FT) and VLANs.
MaxNetifTxQueueLen	(500, 1, 1000)	Maximum length of the Tx queue for the physical NICs. Increase if Tx packet drops are seen in uplink port to the pNIC.
GuestTxCopyBreak	(64, 60, 4294967295)	Packet header transmits smaller than this in bytes will be copied rather than mapped. More security and functionality than performance implications.
VmxnetTxCopySize	(256, 0, 4294967295)	Transmits smaller than this in bytes will be copied rather than mapped. Copying costs CPU but puts lets pressure on the Tx queue and doesn’t require completion.
VmxnetWinUDPTxFullCopy	(1)	Enable full copy of Windows vmxnet UDP Tx packets. Might disable to save CPU, especially for jumbo frames, at the cost of risking more packet drops.
NetTxDontClusterSize	(0, 0, 8192)	Tx packet size (in bytes) smaller than this are transmitted immediately (coalescing options are over-ruled for these packets). Used to ensure good latency for small packets.
CoalesceTxTimeout	(4000, 1, 4294967295)	The coalesce timeout in micro-seconds, or effectively the maximum latency without transmitting. Smaller values can reduce the packet latency at the cost of CPU. Risky to go below 1000.
CoalesceDefaultOn	(1)	Enable dynamic coalescing. Disable to test if issues are related to coalescing.
CoalesceHandlerPcpu	(1, 0, 128)	pCPU that coalesce timeout handler runs on. May be important to set this if VM CPU pinning is used.
CoalesceTxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Tx packets to coalesce. Reduce if Tx coalescing appears to be too aggressive.
CoalesceRxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Rx packets to coalesce. Reduce if Rx coalescing appears to be too aggressive.
vmxnetThroughputWeight	(0, 0, 255)	How far to favor Tx throughput for vmxnet 2 & 3. “0” is dynamic, otherwise this is a weight where a lower value favors latency and a higher value favors throughput.
TcpipHeapSize	(24, 24, 120)	Initial size of the TCP/IP module heap in megabytes. May need to increase if there are many vmkernel connections (NFS, iSCSI, etc.).
TcpipDefLROMaxLength	(16000, 1, 65535)	Maximum length for the LRO aggregated packet for vmkernel connections. Increasing this reduces the number of acknowledgments, which improves efficiency but may increase latency.
E1000TxZeroCopy	(0)	If disabled copy UDP or non-TSO Tx packets for e1000.
E1000TxTsoZeroCopy	(1)	If enabled do not copy TSO Tx packets for e1000.
E1000IntrCoalesce	(1)	Enable interrupt coalescing for e1000. Disabling can improve latency at the expense of CPU.
MaxPktRxListQueue	(3500, 0, 200000)	Maximum number of packets queued in vmkernel. Increasing this can reduce the number of dropped packets but at the cost of increased vmkernel memory and queuing latency.
Vmxnet3RSSHashCache	(1)	Enable RSS hash cache for vmxnet3 in Windows guests.
VmklnxLROEnabled	(0)	Enable large packets for recent Linux guests with vmxnet 2 & 3. Most likely to benefit hosts with small number of VMs with few sessions each, where each session has a heavy Rx load (more than 1 MB/sec). This is an experimental feature and has not been tested extensively.
VmklnxLROMaxAggr	(6, 0, 24)	Maximum aggregation count in number of packets for vmklinux LRO.