Severe network performance issues

[oreeh]

This thread is a follow-up to the following threads since these seem to be related:

http://www.vmware.com/community/thread.jspa?threadID=74329

http://www.vmware.com/community/thread.jspa?threadID=75807

http://www.vmware.com/community/thread.jspa?threadID=77075

Description of the issues and "results" we have so far.

juchestyle and sbeaver saw a significant degradation of network throughput on 100 full virtual switches.

The transfer rate never stabilizes and there are significant peaks and valleys when a 650 meg iso file

gets transferred from a physical server to a vm.

Inspired from this I did some short testing with some strange results:

The transfer direction had a significant impact to the transfer speed.

Pushing files from VMs to physical servers was always faster (around 30%) than pulling files from servers.

The assumption that this is related to the behaviour of Windows servers was wrong, since this happened

regardless of the OS and protocol used.

Another interesting result from these tests: e1000 NICs always seem to be 10-20% faster than the vmxnet

and that there is a big difference in PKTTX/s with vmxnet and e1000.

After that acr discovered real bad transfer speeds in a Gigabit VM environment.

The max speed was 7-9 MB/s, even when using ESX internal vSwitches.

A copy from ESX to ESX reached 7-9 MB/s too.

The weird discovery in this scenario: when disabling CDROMs in the VMs the transfer speed goes up to 20 MB/s.

Any ideas regarding this?

I'll mark my question as answered and ask Daryll to lock the thread so we have everything in one thread.

[oreeh]

In all my tests I've tried Auto / 1000 Full / 100 Full and even 100 Half with no difference.

[sasserw_agy]

I feel your pain. I worked with a Technician, we are using a Dell Poweredge 2900, Dual Core X 2 Proc. My VM's are running much better now....since the duplex change. Still testing though.

[JonT]

Ok well today has been a good open day for me. I have built several new hosts that I will be testing the network performance on.

HP DL380 G5

HP DL740

HP DL580 G2

HP DL560

HP DL360 G4

DELL PE2950

Obviously some of this hardware is a generation or two old now but I went into my storeroom and found all I could that was supportable for ESX and still not too much of a dinosaur. I have brand new NIC's in all of these that have shown good performance for me in the AMD processor based systems. I want to see if ANY of the Intel processor based systems that I have perform better than the average I have seen so far.

When I tested the auto-negotiate vs. hard set Speed/Duplex my auto-neg setting actually performed a little better. I am sure that this is because my switch ports that I am testing on are set for Auto as well. All of my standard Datacenter ports are hard set for production servers though.

Just have to share this tid-bit. I just processed a quote for one of my projects for 16 HP DL580 G4's, total cost about $320,000 or so. I wish I had that kind of money for myself.....

Message was edited by:

JonT

[oreeh]

Tried the 3COM stuff today - no luck.

The vmkernel module exists in ESX3 but doesn't seem to work

[JonT]

Ok I am still testing that long list of other systems, but so far I see what looks like a trend with some of the single core xeon systems at the lower end of the performance. My new DL380 G5 just got the highest performance yet for my tests:

DL 380 G5 - 2 types of NIC's

\- Onboard (NetXtreme II BCM5708 1000Base-T)

VM to VM

8k: 700 MBytes 586 Mbits/s

64k: 1.09 GBytes 938 Gbits/s

VM to Localhost

8k: 873 MBytes 731 Mbits/s

64k: 1.76 GBytes 1.51 Gbits/s

\- 82571EB Gigabit Ethernet (PCI-e dual port NIC)

VM to VM

8k: 616 MBytes 516 Mbits/s

64k: 1.06 GBytes 908 Gbits/s

VM to Localhost

8k: 878 MBytes 735 Mbits/s

-64k: 1.67 GBytes 1.43 Gbits/s

[oreeh]

next week I'll get my hands on a DL385G2 with different NICs

[JonT]

I have a 385G2 but it is currently my testbed for VCB. I guess I could swap the drives on it temporarily and build it as an ESX host.....good idea

[JonT]

Ok the 385 G2 is building with 3.0.1 now, but my I just finished testing with the Dell PE2950 and do not believe the results ( I tested 3 times to verify all numbers-averages posted):

Dell PE2950 - (Onboard NetXtreme II BCM5708 1000Base-T)

VM to VM

8K: 890 MBytes 746 Mbits/s

64k: 1.80 GBytes 1.55 Gbits/s

VM to Localhost

8k: 855 MBytes 717 MBits/s

64k: 2.2 GBytes 1.89 Gbits/s

Message was edited by:

JonT

[amilewski]

would somebody mind posting the necessary alterations to the registry that are being used to test 8k/64k.

thanks to everyone contributing, this is a great posting. cant wait for the end results.

[oreeh]

You don't need to change the registry.

Iperf has a command line parameter (-w) for this.

[JonT]

Ok all of my testing is done to this point, aside from the IBM Blades and my HP DL740. I will try to have these last few hosts finished up tomorrow and post all of the results (64k window only) and let everyone interpret them as you wish. Honestly it looks to me right now that the network performance of the VM's is directly affected by the CPU/bus, and then as a secondary concern the NIC/PCI bus.

[daat]

hi jont

i'm having the same probs stated in these posts here, but i had no chance to do some testings, so i dont have any numbers

im currently setting up a hp dl580 g4, so if you can provide me detailed info for testing, i might find the time to do so

our dl580 has intel 82571eb and netxtreme II 5706 nics

you might contact me via email if you like to

regards

rob

[oreeh]

To run the test you need to download iperf http://dast.nlanr.net/Projects/Iperf/#download (available for almost every OS)

and install it in the VMs.

On one VM start iperf in server mode (iperf -s)

On the other VM start iperf in test mode (iperf -c serverip)

On the VM were iperf server is running run iperf in test mode against 127.0.0.1

To use the 64K window size instead of the VM default add -w 64K.

[daat]

any other settings? both vms on the same vswitch, different vswitch, e1000 or whatever nic.......?

[oreeh]

both vms on the same vswitch, different vswitch

both are of interest

e1000 or whatever nic

what type of NIC you use in the guest does matter (usually e1000 is 10-20% faster) - but this might be totally different in your environment

In my tests I always use e1000, JonT and the others use vmxnet.

Message was edited by:

oreeh

[juchestyle]

Another HP Box huh?

Respectfully,

Matthew

[JonT]

Ok well I don't think the networking performance issues being seen so far are HP specific. That would be the one model that I don't have readily available for testing but the 585 G2 that I tested worked in the middle for performance with the on-board NIC's. I would add another NetXtreme NIC to this system but I am all out of PCI-e NIC's. I am finishing up my IBM Blade testing now then will post all of my findings. Not a lot of commonality between processor type, system manufacturer, NIC type and speed/duplex from what I saw.

[daat]

no, i also dont think its hp specific, since we have 2 fujitsu-siemens boxes, that have the same prob

[juchestyle]

Do we think it is a specific NIC, or maybe driver then?

Respectfully,

[oreeh]

To me it seems that it is a problem related to the combination of the PCI bus design and used NIC / driver.

Hopefully the fog will lift when JonT posts his last results.

Unfortunately I'm a little busy at the moment and not able to do further tests myself

Anyway here are the results using a HP DL385G2 with a 4 port Intel NIC (HP brand)

710 MBytes 596Mits/sec (64K window size)

Message was edited by:

oreeh