ESXi 4.1 Slow outgoing network performance.

[codecrank]

Hi,

I've been beating my head against the wall trying to figure out why outgoing transfer rates from VMs is a lot slower than incoming. Here is the scenario.

- vmware esxi 4.1.0 (  10.0.0.5 ) , pretty much stock config

- vSwitch0 has 2 teamed uplink vmnics ( using IP hash , connected to a dell powerconnect 6248 via 1Gb ports )

- 1 linux RHEL / CentOS 5  vm ( 10.0.0.1 )

- a stand alone RHEL / CentOS 5 PC ( 10.0.0.2 ) hooked up to the same powerconnect 6248

I use dd and nc ( netcat ) to test network troughput. I just copy zeros from /dev/zero on one machine to /dev/null on another via port 7600 , this way, it eliminate storage as the issue.

TEST 1 )  PC to VM

root@vm # nc -l 7600 > /dev/null

root@pc # dd if=/dev/zero bs=10M count=50 | nc 10.0.0.1 7600
50+0 records in
50+0 records out
524288000 bytes (524 MB) copied, 5.62489 seconds, 93.2 MB/s

TEST 2) VM to PC

root@pc # nc -l 7600 > /dev/null

root@vm # dd if=/dev/zero bs=10M count=50 | nc 10.0.0.2 7600
50+0 records in
50+0 records out
524288000 bytes (524 MB) copied, 20.7168 seconds, 25.3 MB/s

TEST 3) VM to ESXi host  ( using the management console )

~ # nc -l 7600 > /dev/null

root@vm # dd if=/dev/zero bs=10M count=50 | nc 10.0.0.5 7600
50+0 records in
50+0 records out
524288000 bytes (524 MB) copied, 17.9042 seconds, 29.3 MB/s

I tried the following:

- installed vmware tools on the VM, same results

- unplugged different links on the uplink team to test bad links.

- I added another VM  ( windows XP ) and tested and got same outgoing low tranfer rate

test #3 is particularaly interesting, because it shows that traffic stays inside the esxi host. therefore it tends to eliminate unrelated. Traffic shaping is not enabled anywhere...

Anyone ran into this before or have any clue on what's going on ?

Thanks !

-Dan

[DSTAVERT]

What is your storage configuration? RAID level, Write caching enabled?

BTW any head injuries would not be covered under a VMware warranty.

[dquintana]

I dont understand, you check the performance between two vms into the same host? can you try this first?

Which nic card you have configured in the vm ? Flexible? E1000?

[codecrank]

I am not reading or writing to disk. Storage can't be the problem here.

For those not familiar with unix/linux

/dev/zero is not a real device, it is a kernel device that spits out zeros as fast as possible

/dev/null is not a real device, it a just kernel bit bucket.

I am substituting disk devices with memory devices for testing purposes.

dquintana :

test #3 is a test bteween a VM and the ESXi 's management console itself. basically i ssh to the management console, and type "nc -l 7600 > /dev/null"  , that opens up port 7600 and discards anything coming into that port. then the from VM i dump 500 megs into that port using the disk dump command ( dd ) and netcat ( nc )

I guess I can test  connectivity between 2 vm's on the same esxi host. I'll report back the results.

Thanks

[codecrank]

Ok , so transfering from 1 VM to another VM  results in ~ 25MB/s ... same as transfering to hosts outsides the esxi node.

If I suspend a VM while monitoring the NFS server's NIC were the datastore is located, I see that it the NIC gets pegged out at ~100MB/s . so, it looks like the slow outgoing performance is related to the Virutal Mahcine Ports only. not the vmkernel Port used for datastores.

[cprediger]

Hi codecrank,

I'm having the same problem here. Did you find a reason for that strange behavior yet?

Thanks.

Christian

[J1mbo]

Are tools on RH installed using vmware-install.pl?  In case it's relevant, on Ubuntu at the end of that it recommends updating the vmxnet driver, and this makes a tremendous difference on my test kit anyway.

[cprediger]

I'm using Windows 2008 R2 machines with vmxnet3.

[codecrank]

I setup a RHEL6.0 vm. When I created the VM it automatically used the vmxnet3 module. The VM bound network performance doubled ( compared to RHEL 5.4 with vmware tools ) . but still not at the gigabit level.

[root@vm ~]# dd if=/dev/zero bs=1M count=500 | nc pc 7600

500+0 records in

500+0 records out

524288000 bytes (524 MB) copied, 9.20057 s, 57.0 MB/s

I do NOT have the vmware tools installed on it yet. I will try and see if it changes anything.