VMware Cloud Community
billybobSSD
Contributor
Contributor

Slow VMXNET3 performance on 10gig connection

I have an ESXi server with an Intel X520-DA2 10 gig adapter in it. It has an iSCSI data store connected over one port and VM traffic over the other port. The iSCSI speed just couldn't be better but the problem seems to be that none of my VM's will do over 300 megabits/sec. Their all using VMXNET3 adapters. I have gone so far as to hook the second 10gig port directly up to another standalone windows server to eliminate the network/switch as a bottleneck via an SFP+ cable, but am still limited to the ~300 megabit ceiling. Any clues to what could be causing this? Thanks in advance!

Tags (3)
0 Kudos
68 Replies
billybobSSD
Contributor
Contributor

I have changed quite a few of the default VMXNET3 adapter settings to see if it would make any difference. It has not. Before anybody gets smart alecky, I realize changing of any of the TCP settings wont affect the Iperf UDP performance.

settingValue
enable adaptive RX ring sizingdisable
Interrupt moderationdisable
IPV4 checksum offloaddisable
IPV4 Giant TSO offloadno change
Large RX buffersno change
Max TX queuesno change
Offload IP optionsdisable
Offload TCP optionsdisable
Recieve throtttleno change
RSSdisable
Rx Ring #1 sizeno change
Rx ring #2 sizeno change
TCP checksum offload (IPV4)disable
Tx ring sizeno change
UDP checksum offload (IPV4)disable
Small Rx buffersno change
0 Kudos
jhirsh
Contributor
Contributor

I recently came across a similar problem. I kept only being able to hit around 450 Mbps with iperf between two Linux guests (using VMXNET3) operating on two different blades in the same chassis. Enabling jumbo frames in vSphere and upping the MTU on the guests to 9000 allowed the same test to reach around 8 Gbps.

It may not be applicable in your situation, but if you're only able to squeeze out around 500 Mbps, it probably can't hurt to give it a shot.

Cheers

-Joshua

0 Kudos
billybobSSD
Contributor
Contributor

The vswitch is at 9000 MTU and the clients are as well. Thanks for trying!  Smiley Happy

0 Kudos
rklarr
Contributor
Contributor

B-Bob, similar issue here. We created a separate Vswitch with a 1 gig physical connection, and it's flying at expected rates. Just the 10gb crawls. We have a case open.

Oh, and some of the trolls around here will probably be nicer to you if you change your user name to "UberCosmoEurophile". In the grip of their prejudices, it creates cognitive dissonance for them to imagine anyone named BillyBob using kitchen utensils, let alone technology. They lash out.

billybobSSD
Contributor
Contributor

I guess I will change my avatar to Billy Bob Thornton holding an SSD so they will "get it".

Do you have a problem with two VM's on the same vswitch being rate limited as well?

0 Kudos
PCookman
Contributor
Contributor

I am currently out of the Office and will be back on Tuesday . If you need me I can be contacted on 07957 168744.

In my absence please contact Alan.Tung@selection.co.uk or David.Cookson@selection.co.uk.

Regards, Paul..

Paul Cookman

Technical Account Manager

+448448741000

+448448741001

Paul.Cookman@selection.co.uk

http://www.selection.co.uk

[cid:imagee44f5a.JPG@0d50c6ae.42be976f]

This e-mail is confidential and is intended for the exclusive use of the addressee only. Selection Services Ltd accepts no liability for personal views expressed. While every effort has been made to ensure the attachments are virus-free, they must be checked before further use, especially those containing encrypted data. If you have any problems with this e-mail, please contact our IT Manager on Email@Selection.co.uk<mailto:Email@Selection.co.uk>

Registered in England and Wales Registered Number: 2758710 Registered Office: Provident House, 122 High Street, Bromley, Kent BR1 1EZ

--

This message has been scanned for viruses and

dangerous content by MailScanner,

and is believed to be clean.

0 Kudos
Heimatplan
Enthusiast
Enthusiast

Sorry for the not answering for so long, I'm very busy.

When you use your Intel X520-DA2 on Windows, a lot of work gets offloaded to the NIC. My guess is, that ESXi does not use or recognize Intel VT which is mandatory to use offloading. Check out this article, this one perfectly describes your situation: http://maso.dk/2011/10/07/use-10-gigabit-with-intel-vt/

PS: NetQuenue is disabled by default on 4.0 - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100427...

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".
0 Kudos
cdc1
Expert
Expert

Indeed, if you are seeing the issue between vmxnet3 VMs in the same port group on the same vswitch, then it certainly appears to be an issue with ESXi.

Curious to see how this one gets resolved. I'll be keeping an eye on this one.

Good luck!

0 Kudos
vm4u
Contributor
Contributor

HI

Do this settings in the VMs if windows

RSS enabled  = Yes

Large Rx Value = 2048

Ring #2 Size = 32

If you have enabled Jumbo frames enable Jumbo frames.

Regards

0 Kudos
billybobSSD
Contributor
Contributor

Ingo, VMkernel.Boot.netNetqueueEnabled is enabled by default in ESX5. Furthermore I am seeing the issue between vmxnet3 VMs in the same port group on the same vswitch, which leads me to believe the problem has nothing to do with the 10gig ethernet hardware.

0 Kudos
billybobSSD
Contributor
Contributor

vm4u, I tried those settings and get the same results. Thanks!

0 Kudos
billybobSSD
Contributor
Contributor

I just tried Iperf with these VM's again ON A NEW ESXi INSTALL THAT I JUST BUILT, on a completely different piece of hardware and I get THE SAME BANDWITH CAPS. Is there a built in limitation in ESXi 5 ????

Will somebody from VMware answer?

0 Kudos
peterdabr
Hot Shot
Hot Shot

Is the new ESXi install still ESXi5? If so, maybe you should test it on ESXi4.1 with the same 10Gb nic....I realize that it won't solve your problem on ESXi5 but it'd be interesting to see the behavior on the previous version.

Peter D.

0 Kudos
billybobSSD
Contributor
Contributor

Peter, the new hardware I tested it on was ESXi 5 and it did not have 10 gigabit adapters in it, I was testing to see whether the 500 megabit bandwidth limit between vmxnet3 VMs in the same port group on the same  vswitch exists on different hardware, and it does.

I'll probably get annoyed enough any moment now to go and try it on ESXi 4, but then somebody here will complain its on the wrong forum.... :smileylaugh:

0 Kudos
konakcc
Contributor
Contributor

I have a similar problem.  If I install my application on a server with two physical nics, I can see it use about 1.6 Gbps.  It runs on windows 7 or windows 2008.  When I install the OS and application under ESX, it runs fine on the E1000 driver, however, it suffers performance issues.  When I try it with the vmxnet3 interface all kinds of strange things start happening, as if packets are disappering some place.  If in fact, the vmxnet3 interface is limited to 300-450 mbps this would account for my issues as well.  Unfortunately, my account doesnt have support attached to it, so I can not raise an issue with Tech Support.  Has anyone else done so?

0 Kudos
konakcc
Contributor
Contributor

I think only the vmxnet2 driver is available under ESX4.

0 Kudos
billybobSSD
Contributor
Contributor

I just stood up an ESXi 4 server and created two win2008 R2 VM's with VMXNET3 adapters. I'm about to test this all over again.

0 Kudos
trink408
Enthusiast
Enthusiast

Also curious to see what comes of this issue... Keep us posted. 

0 Kudos
billybobSSD
Contributor
Contributor

Ok, just loaded up ESXi 4.1 and created two Server 2008R2 VM's and this is what I got between the two vmxnet3 VM's in the same port group on the same vswitch.

------------------------------------------------------------
Client connecting to 192.168.1.133, UDP port 5123
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.113 port 58518 connected with 192.168.1.133 port 5123
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   304 MBytes   510 Mbits/sec
[  3]  5.0-10.0 sec   303 MBytes   508 Mbits/sec
[  3] 10.0-15.0 sec   307 MBytes   516 Mbits/sec
[  3] 15.0-20.0 sec   306 MBytes   513 Mbits/sec
[  3] 20.0-25.0 sec   306 MBytes   513 Mbits/sec
[  3] 25.0-30.0 sec   309 MBytes   518 Mbits/sec
[  3]  0.0-30.0 sec  1.79 GBytes   513 Mbits/sec
[  3] Sent 1308932 datagrams

So, once again it hits the 500 megabit limit. I keep seeing pdf's posted here where they get much higher throughput. Apparently there is some secret to getting around the bottleneck. I sure wish I knew what it was......

0 Kudos
konakcc
Contributor
Contributor

Can you run the exact same experiment with the E1000 driver and post the results so that they are here in the thread?

I too hope that somebody from vmware wakes up and takes notice.

Thanks.

-Ken

0 Kudos