Severe network performance issues

oreeh · ‎03-21-2007

This thread is a follow-up to the following threads since these seem to be related:

http://www.vmware.com/community/thread.jspa?threadID=74329

http://www.vmware.com/community/thread.jspa?threadID=75807

http://www.vmware.com/community/thread.jspa?threadID=77075

Description of the issues and "results" we have so far.

juchestyle and sbeaver saw a significant degradation of network throughput on 100 full virtual switches.

The transfer rate never stabilizes and there are significant peaks and valleys when a 650 meg iso file

gets transferred from a physical server to a vm.

Inspired from this I did some short testing with some strange results:

The transfer direction had a significant impact to the transfer speed.

Pushing files from VMs to physical servers was always faster (around 30%) than pulling files from servers.

The assumption that this is related to the behaviour of Windows servers was wrong, since this happened

regardless of the OS and protocol used.

Another interesting result from these tests: e1000 NICs always seem to be 10-20% faster than the vmxnet

and that there is a big difference in PKTTX/s with vmxnet and e1000.

After that acr discovered real bad transfer speeds in a Gigabit VM environment.

The max speed was 7-9 MB/s, even when using ESX internal vSwitches.

A copy from ESX to ESX reached 7-9 MB/s too.

The weird discovery in this scenario: when disabling CDROMs in the VMs the transfer speed goes up to 20 MB/s.

Any ideas regarding this?

I'll mark my question as answered and ask Daryll to lock the thread so we have everything in one thread.

Timber_Wolf · ‎08-15-2007

This is just Sick

ESX 3.0.1

Ping from laptop

Reply from X.X.X.14: bytes=32 time=1ms TTL=63

Reply from X.X.X.14: bytes=32 time<1ms TTL=63

ESX3.0.2

Ping from my laptop

Reply from X.X.X.11: bytes=32 time=10ms TTL=63

Reply from X.X.X.11: bytes=32 time=9ms TTL=63

Reply from X.X.X.11: bytes=32 time=10ms TTL=63

3.0.2 Pinging server itself

PING vmsx11.CLOVER.CO.ZA (X.X.X.11) 56(84) bytes of data.

64 bytes from vmsx11.clover.co.za (X.X.X.11): icmp_seq=0 ttl=64 time=0.017 ms

64 bytes from vmsx11.clover.co.za (X.X.X.11): icmp_seq=1 ttl=64 time=0.017 ms

64 bytes from vmsx11.clover.co.za (X.X.X.11): icmp_seq=2 ttl=64 time=0.009 ms

pinging a host on the esx 3.0.2 server (from my laptop)

Reply from X.X.X.36: bytes=32 time<1ms TTL=127

pinging the host from the esx 3.0.2 server

PING clvs36.CLOVER.CO.ZA (X.X.X.36) 56(84) bytes of data.

64 bytes from clvs36.clover.co.za (X.X.X.36): icmp_seq=0 ttl=128 time=0.453 ms

64 bytes from clvs36.clover.co.za (X.X.X.36): icmp_seq=1 ttl=128 time=0.279 ms

64 bytes from clvs36.clover.co.za (X.X.X.36): icmp_seq=2 ttl=128 time=0.469 ms

How the hell can you explain this, has anybody found a answer?

JonT · ‎08-15-2007

You should test similar to most of the testing we have done to this point using "iperf" and esxtop to get more accurate numbers than a simple ping can produce. What are your VM specifics? OS? Memory? 1 or 2 vCPU?

Thanks

I plan to upgrade a host or two to 3.0.2 soon and will be able to help with this effort again.... Oh and I still havent been able to install any of the needed packages on my FreeBSD guest for testing that out. My ability to FTP is only extended to my desktop so I cannot get it to setup properly on the VM. I did however load up a newer VM with Ubuntu and it seems to play very nicely. I will try to get some network perf results from that flavor of Linux soon too.

oreeh · ‎08-21-2007

Interesting and maybe related thread: http://www.vmware.com/community/thread.jspa?threadID=97117

JonT · ‎10-16-2007

After a long and difficult couple of weeks I am back. Yes Oliver this thread was interesting in that it sounds like a similar networking issue but for the console instead of the VM Network interfaces that we had been testing to this point...I haven't tested anything new for the guest networking but will try to see what 3.0.2 results look like now that my lab is fully upgraded.

(edit)

And now that I realized that other thread has 13 pages, i read to the end and see that the issue was patched.....wonder if the upgrade to 3.0.2 with the newest patches will resolve some of these "freaky" network issues for guest VM's, especially your unsupported FreeBSD ones Oliver?

oreeh · ‎10-16-2007

Nice to see you back.

I personally haven't tested 3.0.2 as well - too much work lately.

I really want to test this with ESX 3i as well - someone care to drop me one of those USB keys?

oreeh · ‎10-16-2007

I'm currently running a few FreeBSD VMs in a 3.0.2 environment with really heavy network I/O (SMTP servers with 40.000 - 80.000 messages per hour) - no issues so far.

This probably means that the issue was somehow fixed in 3.0.2 - but I'm not sure yet.

hmolvig · ‎10-17-2007

I just upgraded my environment to 3.02 - and added the patches from oct. 8 - Console network performance is fixed in these patches !

oreeh · ‎05-20-2008

This thread has been moved to the Performance Forum.

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

All