Slow VM Internal Network Performance On One Host

[Marc_P]

Hi<

We have 3 ESXi 5.5 hosts. 2 are PowerEdge R815 and the 3rd is a PowerEdge R805.

I'm using iperf to test network performance as I've noticed that file copying and backups are slow from the VMs when they are on this problematic host.

This is the results using the same VM (Windows 2008 R2 and VMXNET3) and migrating it to the different hosts. I'm doing a iperf to the same VM I'm hosting from:

1st Host (R815) (Working OK)

[  4] local 172.16.2.72 port 5001 connected with 172.16.2.72 port 52713

[  4]  0.0-10.0 sec  2.54 GBytes  2.18 Gbits/sec

2nd Host (R805) (Working OK)

[  4] local 172.16.2.72 port 5001 connected with 172.16.2.72 port 52713

[  4]  0.0-10.0 sec  2.62 GBytes  2.20 Gbits/sec

3rd Host (R815) (Problematic host)

[  4] local 172.16.2.72 port 5001 connected with 172.16.2.72 port 53063

[  4]  0.0-10.0 sec  1.06 GBytes   912 Mbits/sec

As you can see the 3rd host is half the speed of the others. This test should hopefully exclude pNICs as it is all internal to the one host.

The things I've tried are:

1. Compare vSwitch settings to other hosts and confirm they are exactly the same.

2. Wipe and rebuild the 3rd host from scratch.

3. As a test move the physical connections on our switch to a new one (I didn't expect it to make any difference as the internal speed was slow)

4. Update firmware on everything for the server (Inline with the identical host 1)

I'm at a complete loss and this is really annoying me now. I really don't know what else to try. I'm hoping someone can help?

Thanks

[Rubeck]

Weird indeed.... Are all power management related BIOS settings identical on host 1 and 3?

/Rubeck

[HeathReynolds]

That is odd. Was the iperf memory to memory, or where you writing the output to a disk?

[Marc_P]

It was a memory to memory transfer.

I have changed the Power management in the BIOS to Maximum Performance and this has increased the transfer to 2.08GBytes and 1.76GBits/sec Bandwidth. This is a vast improvement from before, not as fast as the other two but a dramatic improvement.

As a test I put only my test VM on the problematic server and migrated all others off and it didn't affect the throughput. All VMs are now back on the server and it is staying at the 2.08Gbytes.

I'm looking to use Jumbo frames for iSCSI traffic but wanted to get this issue out of the way first.

[grasshopper]

Marc P wrote:

I have changed the Power management in the BIOS to Maximum Performance and this has increased the transfer to 2.08GBytes and 1.76GBits/sec Bandwidth. This is a vast improvement from before

It's quite interesting to see the performance increase that the BIOS power management settings had on the 'bad' host3:
1.06GBytes     912Mbits/sec (bios w/ defaults)

2.08GBytes     1.76GBits/sec (bios w/ maximum performance)

Although it didn't solve your problem, it seems to have almost doubled the throughput on the loopback test.  BTW, have you tried doing tests with iperf in client/server mode? (i.e. 2 VMs on the same host, same portgroup).  Also, Is the Memory and CPU speed the same on both hosts (and populated the same)?  Have you tried a full power off and remove the power cables (go old school on them!).

Unrelated Note:  Back to the BIOS topic real quick... Personally, I always turn these "high performance" BIOS features on for my VMHosts.  However, the direction VMware wants us to go 5.5 and forward is OS Controlled (more info here).  I haven't quite bought into that yet (more for stability reasons than performance).

[Marc_P]

I just done a test between 2 VMs on the same host and this is what I got:

  4]  0.0-10.1 sec  45.8 MBytes  37.8 Mbits/sec

This is the bets of 5 tries at doing a iperf! In my my opinion this is very poor.

However if I copy a large 500mb file between the 2 VMs I get a minimum of 90MB/sec

It's a little confusing

The CPUs on both hosts are identical and show as the same. The memory I'd have to check as I'm not 100% they are identical.

[MKguy]

iperf on windows isn't quite optimal for testing network throughput with the default flags. Use multiple parallel threads (-P) and/or an increased TCP window size (-w).