VMware Networking Community
BrandonArms
Contributor
Contributor
Jump to solution

nsx throughput

i am having throughput issues between VM's once NSX has been installed.  i have tested throughput between VM's prior to NSX insalltion ans was able to achieve 8-9Gbps using netperf on CentOS 7.1 vm's located on different ESX6.0 nodes attached to a dvs portgroup.  I installed NSX6.2 and attached my vm's to logical switches, re-ran the same netperf tests, and am only seeing 3-4Gbps of throughput.

Tags (1)
1 Solution

Accepted Solutions
DaleCoghlan
VMware Employee
VMware Employee
Jump to solution

Confirm if the DRIVER supplied by your hardware vendor supports such features as VXLAN TSO and RSS.

View solution in original post

Reply
0 Kudos
9 Replies
dhanarajramesh
Jump to solution

I would suggest you to use Network I/O Control  (NIOC) for your environment and give more resources to NSX VXLAN dedicated port group. because all the logical NSX vlans travels on single dedicated VXLAN portgroup.

Reply
0 Kudos
BrandonArms
Contributor
Contributor
Jump to solution

this is a DEV environment and does not have any resource contention.  this seems to be an issue with my UCS B200-M3 1240vic doing vxlan encap/decap.  i will be upgrading to ESX6.0U1 and downgrading my UCS firmware to 4.0.4b and retest.  I am also looking into the 1340/1380 vic which supports vxlan offload.

Reply
0 Kudos
DaleCoghlan
VMware Employee
VMware Employee
Jump to solution

Confirm if the DRIVER supplied by your hardware vendor supports such features as VXLAN TSO and RSS.

Reply
0 Kudos
rkisteHL
Contributor
Contributor
Jump to solution

Brandon,

Did you ever determine the issue of the throughput.  I am seeing a similar issue in the M4 Blade.  Unfortunately I am locked into the firmware of my UCS.  I would be interested in hearing your findings.

Reply
0 Kudos
BrandonArms
Contributor
Contributor
Jump to solution

‌a few updates. The enic driver supplied by Cisco to vmware for the 1340/1380 Vic does not support RSS. line rate throughout with this Vic, or the 1240/1280, is not possible.  I am only able to get 6-7gbps with vxlan and nvgre configured on the 1340. The 1240 is 4-5gbps.  the other issue I am working through with TAC is I am only getting roughly 20G to my fabric even though I have the 1340 with a port expander and should get 40g. This is outside of NSX and tested via vds and vss.   TAC has my scenario labbed up and will be testing. I will update as I receive info from them

Reply
0 Kudos
BrandonArms
Contributor
Contributor
Jump to solution

something else to think about, we've noticed that UCS maps traffic streams to different HIF's/NIF's without regard for utilization.  So, a single HIF/NIF could be saturated and other's available or with lower utilization but they are not being used.  what we've seen with our testing is that with UCS, the more vm's we've used for testing, the better it would distribute the traffic.  the problem becomes when you overlay NSX and all of your traffic is encap/decap in the VTEP.  the phys UCS components will not see your vm macs/ip's, only the vteps.  when designing your solution with ucs, you will want to take this into account when determining your number of vteps.  it's possible, with only a single vtep per blade, that your vtep traffic will all be hashed to the same HIF/NIF in UCS.  also keep in mind that due to the lack of RSS support in the enic driver for the 1340/1380, you are only getting 6-7Gbps per HIF.  i am now testing c series with the 1225 vics.  they do not support vxlan offload, and i only get 4-5gbps per link with nsx, but i do not have to take into account any other c series being "pinned or hashed" to my links to the FI's. 

Reply
0 Kudos
BrandonArms
Contributor
Contributor
Jump to solution

the resolution to the UCS issue was removing port channeling in UCS.  this removes the ucs hashing of your traffic to the NIFs.  by doing so each blade is now pinned to to it's own NIF to the fabric.  if you have a 2208xp iom and connect all 8 interfaces up to your FI's, each of your blades now has it's own NIF.  if you couple that with having 2 nics per host (2 vteps) tagged to each fabric you should be able to realize 12-14Gbps with 1340vic and a port expander.  once cisco addresses the enic driver issue for this vic and lack of RSS support, hopefully you can achieve closer to line rate performance.

Reply
0 Kudos
rkisteHL
Contributor
Contributor
Jump to solution

That is great information.  We are running UCS-B200-M4.  I will work with the team to work through your suggestion and report back.  Thank you.

Reply
0 Kudos
rkisteHL
Contributor
Contributor
Jump to solution

Not that we found a fix, however we found a workaround for the time being.  A single VTEP in failover mode only allows for 5.5Gb throughput with UCS M3 blades.  As you add more flows the bandwidth splits in half where two flows get somewhere around 2Gb per flow.  If you change your VTEP strategy to two VTEPs(SRCID/SRCMAC), You can get 5Gb per VTEP.  As you add flows to the host the bandwidth will cut in half per VTEP rather then per host.  It is important to perform this design change before going into production.  Otherwise there is a ton of shuffling around of workloads and clusters.  The pinning idea probably works better but doesn't scale for our requirements.  Until this driver is resolved this is the best solution we found to work around the issue.

Reply
0 Kudos