SLow Filecopy over Gigabit between two vm's on the same esx host

[meistermn]

Arghh!

I do not get more than 8 MB/s over gigabit between two vm's on the same esx host.

This means all traffic goes only through the virtual switch , not a physical switch.

The esx host (3.5 update 3 + last patches) is a HP DL 585 G5 4 x Quad Core and 64 GB

The VM's OS are Windows 2003 32 Bit R2 SP2 and Windows 2003 64 Bit R2 SP2.

VMware tools installed are installed in both vm's.

Vm's are both on SAN FC Netapp 6070. In both VM I get 80 MB/s when doing a local xcopy of 10 GB large file. So san not the bottleneck.

TCPWindowSize

This value determines the maximum amount of data (in bytes) that can be outstanding on the network at any given time. It can be set to any value from 1 to 65,535 bytes by using the following registry entry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip \Parameters\TcpWindowSize (REG_DWORD)* The default for a gigabit interface is set to approximately 65,535 (rounded down to the nearest multiple of full TCP packets), 16,384 for a 100 Mbps link, and 8,192 for all interfaces of lower speeds (for example, modems), again rounded down. Ideally, this value should be set to the product of end-to-end network bandwidth (in bytes/s) and the round-trip delay (in seconds), also referred to as the bandwidth-delay product. This value should be set according to the amount of TCP data expected to be received by the computer.

From this Microsoft guide

changed TCPWindowsSize to 1,073,741,823 (with window scaling)

and HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts registry value to 1

rebooted the vm's . No effect.

Anyone able to get 40 MB/s - 80 MB/s

[dilidolo]

I tested 2 windows vm using iperf on the same esx, same subnet. Wire transfer rate goes to about 1.3Gb/s but I haven't tested actual copy files. my tcpwindow size is default 64K.

[Cooldude09]

Try to copy the files between the VM and ESX and see if it works. Is it happening on a particular file or a particular destination. Try to use different options like different destinations, different VM's on other box to narrow down if the problem is withing ESX or VM's.

Regards

Anil

Save the planet, Go Green

if you found my answer to be useful, feel free to mark it as Helpful or Correct.

[meistermn]

To exclude that it is a Windows Os problem, we created two new vm's and installed open 11.1. Although installed vmware tools in suse (vmxnet).

Then started two tests:

1. Test : Copy suse dvd (4.2 GB) from VM1 to VM2 per scp (VM's on the SAME Host)

Result 8-10 MB/s

2. Test ; Copy suse dvd (4.2 GB) from VM1 to ESX per scp

Result 7-8 MB/s

3. Test : Copy suse dvd (4.2 GB) from VM1 on ESX Host1 to VM2 ESX Host2

Result 12 MB/s

[meistermn]

When you copy a file from one Windows VM to the other Windows VM you are using SMB Protocol. SMB Protocoll has much overhead.

So the result from iperf , netio, netperf are no real life scenarios.

[Lightbulb]

I know you say SAN is not an issue, but as a quick sanity check could you create datastore on local storage of DL385 and perform same test?

I know that Netapp defaults to thin provisioning which maybe has something to to with the issue (Please nobody bite my head off I am pretty ignorant of NFS in ESX environment and I will be the first one to admit it). But really it would be a good test in that if you get the same results you will have truly ruled out the SAN and if you do not get the same results you have more data

[meistermn]

The DL 585 G5 has only to DIsk in Raid 1 . On this is ESX installed.

The Raid DP (Raid 6) is configured with 41 FC Disks.

In a Windows 2003 VM i tested following:

1:) Create a 10 Gig file in the WIndows 2003 VM:

f:\ fsutil createnew bigfile.txt 100000000000

2.) Copy the file bigfile with fastcopy (don't uses the OS Buffer) from f:\ to f:\

Get 40-80 MB/s troughput on filecopy in a VM in the same partition.

[Brijn]

Hi,

I have exactly the same question problem, far lower then expected copy rates between VM's

I initially noticed the problem in a P2V'ed W2K3 system. have since installed two Ubuntu x32 servers from scratch to make sure there is no problem with the P2V'ed systems.

Running Iperf between the two linux hosts with 64k block size I get well over 1Gbit of data, but as you said before, that is purely synthetic. I've used FTP between the two host (same vSwitch) and get around 20MB (<200Mbit)

I've enabled the enhadnced vmxnet driver + jumbo frames on the vSwitch + jumbo frames on the client and see the transfer rate increase to 30MB/s (<300Mbit), but that is still quite a bit lower then what I would expect, esxtop shows that the VM get all the CPU cycles they request, so resources are not an issue either

I opened a SR with VMware to see what they have to say. But this seems to be a very problem, so it might be there is no fix and we have to live with these transfer speeds

Bas

[meistermn]

So I connect our Linux Team and ask then to make a filycopy test per scp between to suse 10 sp1

VM's. They are on the same netapp filer.

They get following values. The different is they as server model an IBM X3755

VM to VM: 28.9MB/sec – Transfer time = 24 secs

VM to ESX host: 23.1MB/sec – Transfer time 30 secs

[Cooldude09]

I believe you have found the answers yourself...it seems to be the protocol you using to download the file.....try using winscp to transfer files between windows vm and esx and windows vm and linux vm to see the thruput.

Regards

Anil

Save the planet, Go Green

if you found my answer to be useful, feel free to mark it as Helpful or Correct.

[meistermn]

http://communities.vmware.com/thread/188653

As ArthurKnowles said he gets 25-28 MB/s in windows 2008 . Windows 2008 uses SMB proctol version 2 and windows 2003 uses windows 2003.

On my Windows 2008 Guests I average 25~30 MB/sec continuous transfer rate. It's very steady while on my ESX 3.5 Guests it varies wildly. HIgh peaks and valleys. There does appear to be soemthing wrong with the Guest side of ESX, but I'm not sure what it is. I guess I'll stop by the KNowledgeBase again and browse through it and see what I can find out. If anything.

[meistermn]

with windows 2000 the windows size was default set to 10 Mb/s

"2. The default parameters for the TCP window size and scaling parameters options are suitable for small 10 Mbps LANs or Internet access where the smallest bottleneck is no more than 2 Mbps. However, for Internet access at 10 Mbps or more, or for 100 Mbps LANs, the default parameters are too conservative, and throughput may suffer."

Source:

Windows 2003 SP2 default : ?

[Brijn · Accepted solution]

Hi,

I'm now at the point I'm happy with the performance. I have two W2K8 hosts build from scratch. I enabled jumbo frames on the vSwitch:

1. Find the correct vSwitch name

esxcfg-vswitch -l

1. Set jumbo frames

esxcfg-vswitch -m 9000 vSwitch[x]

1. Check that all associated cards to Jumbo frames now

esxcfg-nics -l

In your Guest OS, also enable jumbo frames. I also did some tweaks suggsted by ms in document "Performance Tuning Guidelines for Windows Server 2008"

• NtfsDisable8dot3NameCreation

HKLM\System\CurrentControlSet\Control\FileSystem\REG_DWORD)

• TreatHostAsStableStorage

HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\(REG_DWORD)

When I copy a 2.6G ISO between these two hosts connected to the same vSwitch the copy starts at 100MB/s, then drops of the 50-60MB/s (I guess when all the caches have filled up)

Bas

[meistermn]

After reding this whitepaper , seems 20 Mb/s is a good value for 1 Gigabit/s for smb filecopy.

Even better the other whitepaper:

The second screen means : Higher Message size ( 8 KB) -> Higher Throughput

[meistermn]

Which networkcard are you running? E1000 or advanced vmxnet?

[dilidolo]

I tested in my test lab, NFS datastore on Openfiler with 4 SATA disks in RAID-5. Copying files between 2 windows 2003 VMs on same ESX host results in 30-40MB/s out of the box, no any tuning done. VM and vmkernal NICs are seperated.

[Brijn]

Hi,

Enhanced vmxnet. There is another white paper comparing the different adapers and on 3.5 the diff between the two. A small victory for e-vmxnet on x64 guests if I remember correctly

Bas