Unable to get 10GB speeds between VM's

[mcrampton]

Hi, I have 2 ESX hosts with 10Gb cards in them. I have a vSwitch configured with the NICs added, they are all showing 10000 full. The physical switch is showing connected at 10Gb as well

I added 2 VM's with vmxnet3 NICs on the same host to the port group. Inside the VM's, I see vmxnet3 Ethernet Adapters in both, and they both show connected at 10Gb speed.

However, if I take a 3 GB file, and copy it between the VM's, it takes anywhere from 30-50 seconds, which puts my speeds at something like 480-800Mbps, obviously nowhere near 10Gbps.

I should also mention that there is nearly no other traffic on this ESX host, it's still in the testing phase so has only a handful of VM's on it.

Can anyone suggest what I might have done wrong? I assumed when I saw the NIC connected at 10Gbps inside the VM that everything was hunky dory, but these speeds suuuuck.

[a_p_]

With coping a file, it's not only the network which is involved, but also the storage. To test network performance use a tool like Iometer.

André

[Josh26]

If you're testing with Windows SMB, you'll run into its limitations before you hit a network limit. As AP said, try a network tool.

[MKguy]

As others suggested, to derive the actual network performance from a file copy operation is utterly flawed.

Use a tool like iperf to properly measure network throughout:

http://rickardnobel.se/iperf-to-test-network-performance/

[mcrampton]

Alright, will do some other testing today. Thanks.

[mcrampton]

Ok, so using iperf I've been able to get 7 Gbps, which is much better. Since this is basically completely unused new vswitch, however, I'm wondering if I shouldn't be able to get a bit closer to max capacity. Screenshot attached with my settings and results. Do I need to run it with different settings to max out the link? Or should I enable/configure jumbo frames? This will not be used for iSCSI storage, just network traffic & vmotion. Thanks.

[MKguy · Accepted solution]

iperf on Windows doesn't perform very well in general, the results seem quite good regarding that already. You can play with the settings a bit more until you've hit a sweet spot, but it's hard to tell. Try -P 10 and -w 65536 for more parallel sessions and an increased TCP window size.

Also make sure the VMs you use for testing have enough vCPUs to handle the load you're pushing through the virtual wires. Jumbo frames could surely also come in handy.

[SteveFuller2011]

To figure out just how much performance you can get from a VM try running both the iperf server and iperf client on the same VM i.e., open up two command prompts, and then run iperf -s in one window and iperf -c 127.0.0.1 in the second window.

Regards

[mcrampton]

I've been able to get it as high as 8.5 Mbps by playing around with the iperf settings. I think I'm satisfied with this result from inside windows. Thanks for the feedback.

[Josh26]

To clarify, is this iperf between VMs on different hosts?

Your results are reasonable enough that I would ask what switch you are using and how it's configured - tuning and hardware quality may come into effect regarding getting much better than 8.5.

[mcrampton]

Hi Josh,

Yes the 8.5 speed was obtained between VM's on different hosts. My hosts are connected to an HP Procurve E8206zl switch with 2 x 10 Gbps SFP+ modules installed. I've enabled jumbo frames on the switch, and set the vSwitch to an MTU of 9000. I'm using standard ports, no trunking (haven't been able to get that to work for some reason, and not sure I see the need for it). The VLAN I've been testing on is the 'untagged' VLAN on these ports (ie: the native vlan in HP speak).

[MKguy]

Just enabling jumbo frames on the physical and virtual Switch doesn't cut it, you also need to set the vNIC in the guest OS to use jumbo frames (Windows device manager NIC options).

Anyways, I don't think you'll get much more out with iperf on Windows. Test it on Linux instead and you'll probably get something around 9.5Gbps.

Also you can create a VM with a lot of memory and vMotion it across the 10G link, observing network throughput in (r)esxtop during that vMotion.