0 Replies Latest reply on Dec 2, 2019 12:07 AM by Naekm

    NSX-V VTEP Offload to NIC

    Naekm Lurker

      Hi all,

       

      Regarding NSX-V VTEP Offload to NIC, as far as I see, there are few articles and information over the internet so any advices on this feature would be appreciated!!

       

      I managed to find how to set up and what default setting is as below.

      However I have no idea with configuration criteria of RSS/TSO/VXLAN Offload Do you have any information on them?

      In the first place, does someone leverage enabling these NIC offload features with NSX-V? If so, any considerations to introduce them?

      VMware® NSX for vSphere (NSX) Network Virtualization Design Guide says to recommend to use VXLAN
      Offload feature for Edge Cluster but I wonder if ESXs on Prod VM HA cluster does not requires to enable them?

       

      It is recommended to use hosts with higher core densities to support the required 

      Edge VM sizing (2 vCPU to 4 vCPU). Higher clock speed should also be considered
      for VXLAN offload while NIC choices made based on the need for 
      higher bandwidth.

       

      Host Specification Guideline for Edge Cluster

      • Highly recommended to use NIC that supports VXLAN
        TSO offload and
        RSS support for desired line rate performance.

       

       

      [root@esxi:~] esxcli system module parameters list -m vmnic10
      Name                  Type           Value  Description
      --------------------  -------------  -----  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      DRSS                  array of uint         Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
      RSS                   array of uint         Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
      debug_mask            uint                  Enabled debug mask (default: 0)
      dropless_fc           uint                  Pause on exhausted host ring. 0: Disable (Default) 1: Disable
      enable_dcbx           uint                  Enable DCBX. 0: Disable 1: Enable (Default)
      enable_fwdump         uint                  Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
      enable_lro            uint                  Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
      enable_vxlan_filters  uint                  Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
      fairness_threshold    uint                  When set to 1 will enable the fairness threshold; 0 by Default
      hw_vlan               uint                  Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
      max_vfs               array of uint         Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
      mtu                   uint                  MTU when the driver is loaded. Range: 0-9000. (default: 1500)
      offload_flags         uint                  Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
      rss_engine_nr         array of uint         Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
      rx_filters            int                   Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
      rxqueue_nr            uint                  Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
      rxring_bd_nr          uint                  Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
      tx_to_delay           uint                  Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
      txqueue_nr            uint                  Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
      txring_bd_nr          uint                  Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096

      [root@esxi:~] esxcfg-module -i qfle3
      esxcfg-module module information
      input file: /usr/lib/vmware/vmkmod/qfle3
      License: ThirdParty:QLogic_Proprietary
      Version: 1.0.86.0-1OEM.670.0.0.8169922
      Name-space:
      Required name-spaces:
        com.vmware.vmkapi@v2_5_0_0
      Parameters:
        DRSS: array of uint
         Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
        RSS: array of uint
         Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
        debug_mask: uint
         Enabled debug mask (default: 0)
        dropless_fc: uint
         Pause on exhausted host ring. 0: Disable (Default) 1: Disable
        enable_dcbx: uint
         Enable DCBX. 0: Disable 1: Enable (Default)
        enable_fwdump: uint
         Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
        enable_lro: uint
         Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
        enable_vxlan_filters: uint
         Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
        fairness_threshold: uint
         When set to 1 will enable the fairness threshold; 0 by Default
        hw_vlan: uint
         Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
        max_vfs: array of uint
         Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
        mtu: uint
         MTU when the driver is loaded. Range: 0-9000. (default: 1500)
        offload_flags: uint
         Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
        rss_engine_nr: array of uint
         Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
        rx_filters: int
         Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
        rxqueue_nr: uint
         Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
        rxring_bd_nr: uint
         Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
        tx_to_delay: uint
         Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
        txqueue_nr: uint
         Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
        txring_bd_nr: uint
         Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096

      [root@esxi:~] localcli network nic software list
      NIC      IPv4 CSO  IPv4 TSO  Scatter Gather  Offset Based Offload  VXLAN Encap  Geneve Offload  IPv6 TSO  IPv6 TSO Ext  IPv6 CSO  IPv6 CSO Ext  High DMA  Scatter Gather MP  VLAN Tagging  VLAN Untagging
      ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      vmnic10  off       off       off             off                   off          off             off       off           off       off           off       off                off           off