VMware Networking Community
Naekm
Contributor
Contributor

NSX-V VTEP Offload to NIC

Hi all,

Regarding NSX-V VTEP Offload to NIC, as far as I see, there are few articles and information over the internet so any advices on this feature would be appreciated!!

I managed to find how to set up and what default setting is as below.

However I have no idea with configuration criteria of RSS/TSO/VXLAN Offload Do you have any information on them?

In the first place, does someone leverage enabling these NIC offload features with NSX-V? If so, any considerations to introduce them?

VMware® NSX for vSphere (NSX) Network Virtualization Design Guide says to recommend to use VXLAN
Offload feature for Edge Cluster but I wonder if ESXs on Prod VM HA cluster does not requires to enable them?

It is recommended to use hosts with higher core densities to support the required 

Edge VM sizing (2 vCPU to 4 vCPU). Higher clock speed should also be considered
for VXLAN offload while NIC choices made based on the need for 
higher bandwidth.

Host Specification Guideline for Edge Cluster

  • Highly recommended to use NIC that supports VXLAN
    TSO offload and
    RSS support for desired line rate performance.

[root@esxi:~] esxcli system module parameters list -m vmnic10
Name                  Type           Value  Description
--------------------  -------------  -----  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DRSS                  array of uint         Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
RSS                   array of uint         Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
debug_mask            uint                  Enabled debug mask (default: 0)
dropless_fc           uint                  Pause on exhausted host ring. 0: Disable (Default) 1: Disable
enable_dcbx           uint                  Enable DCBX. 0: Disable 1: Enable (Default)
enable_fwdump         uint                  Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
enable_lro            uint                  Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
enable_vxlan_filters  uint                  Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
fairness_threshold    uint                  When set to 1 will enable the fairness threshold; 0 by Default
hw_vlan               uint                  Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
max_vfs               array of uint         Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
mtu                   uint                  MTU when the driver is loaded. Range: 0-9000. (default: 1500)
offload_flags         uint                  Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
rss_engine_nr         array of uint         Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
rx_filters            int                   Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
rxqueue_nr            uint                  Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
rxring_bd_nr          uint                  Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
tx_to_delay           uint                  Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
txqueue_nr            uint                  Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
txring_bd_nr          uint                  Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096

[root@esxi:~] esxcfg-module -i qfle3
esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/qfle3
License: ThirdParty:QLogic_Proprietary
Version: 1.0.86.0-1OEM.670.0.0.8169922
Name-space:
Required name-spaces:
  com.vmware.vmkapi@v2_5_0_0
Parameters:
  DRSS: array of uint
   Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
  RSS: array of uint
   Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
  debug_mask: uint
   Enabled debug mask (default: 0)
  dropless_fc: uint
   Pause on exhausted host ring. 0: Disable (Default) 1: Disable
  enable_dcbx: uint
   Enable DCBX. 0: Disable 1: Enable (Default)
  enable_fwdump: uint
   Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
  enable_lro: uint
   Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
  enable_vxlan_filters: uint
   Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
  fairness_threshold: uint
   When set to 1 will enable the fairness threshold; 0 by Default
  hw_vlan: uint
   Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
  max_vfs: array of uint
   Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
  mtu: uint
   MTU when the driver is loaded. Range: 0-9000. (default: 1500)
  offload_flags: uint
   Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
  rss_engine_nr: array of uint
   Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
  rx_filters: int
   Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
  rxqueue_nr: uint
   Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
  rxring_bd_nr: uint
   Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
  tx_to_delay: uint
   Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
  txqueue_nr: uint
   Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
  txring_bd_nr: uint
   Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096

[root@esxi:~] localcli network nic software list
NIC      IPv4 CSO  IPv4 TSO  Scatter Gather  Offset Based Offload  VXLAN Encap  Geneve Offload  IPv6 TSO  IPv6 TSO Ext  IPv6 CSO  IPv6 CSO Ext  High DMA  Scatter Gather MP  VLAN Tagging  VLAN Untagging
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vmnic10  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

Reply
0 Kudos
0 Replies