Significant drop in network performance when transfering large files within Windows

[ade333]

--Hardware

Dell Equallogic PS4100e

(2) Dell R620 Servers

(2) Dell PowerConnect 6624 switches, stacked - for iSCSI traffic

--Virtual Env

Long story short, I have a Win 2008 R2 vm on each host

Each VM has 4gb RAM

Each VM has a file share

When I copy small files via UNC path (i.e. \\ServerB) the file copies at full speed (nearly 1gb) and everything is great

When I attempt to copy large files - files that are larger than the amount of free RAM on the Windows VM, the file speed drops off significantly.  Let's say I have 3gb of free RAM and the file I'm copying is 4gb.  The first 3gb of the file transfer goes at the expected speed (nearly 1gb/s) however the transfer speed drops significantly from there on- down somewhere around 2mb/s-10mbs.

Dell has reviewed every inch of my configs and is telling me this is the result of Windows using a buffered copy.  I understand that this is fact a Windows issue, however I have not encountered such a significant performance hit before. Can anyone confirm this as normal/abnormal?  Is there a fix/work around ?

I tried a number of things so far - some probably didn't apply

Switched to the ParaVirtual controller

Switched to VMXNET 3 NICs

Within each Windows VM, disabled TCP Chimney Offload (through netsh and the NIC)

Within each Windows VM, disable receive side scaling

Within each Windows VM, Changed the Large System Cache registry entry to 0

None of these helped.  The only time I see a change is if I increase the RAM.  If I increase it so that the amount of free RAM is greater than the size of the file I am transferring, the transfer speed will not drop down during the copy.

Any help is greatly appreciated.

[lowdownshame]

Hi, curious to know if you're using esxi 5.5. I upgraded to 5.5 a few weeks ago and have noticed now that anytime i do large file transfers (250MB+) either from VM --> VM or separate LAN PC ---> VM, it'll start off fine at 1000 Mbps for a few seconds then suddenly slow down to virtually nothing at less than 1 Mbps with sporadic throughput. Also had experienced a PSOD while attempting to transfer a 1GB file between 2 VM's each with 4GB RAM. Looking for some help to resolve this issue.

[ade333]

No, it was not after an upgrade.  This is a brand new setup - using 5.1

You can test to see if the issue you are having is the same as my by simply increasing the RAM in the VMs.  If you have it available, bump them up to 12gb and see how your transfers work.  As I noted above, when the amount of free RAM is larger than the file size I am copying, I do not see any performance drop offs.

One other thing to note, are you working with a SAN or local attached storage on servers?  If it's local attached storage the issue you are having may be with the RAID controller in your host servers.  You may want to look at this:  Solving Slow Write Speeds When Using Local Storage on a vSphere Host

I have a support call open with MS in an attempt to get the software for Server 2008 R2 as mentioned here:

You experience performance issues in applications and services when the system file cache consumes ...

[lowdownshame]

My storage setup consists of 2 locally attached ssd + 2 raidz1 pools connected through a lsi-9207 hba. Thanks for the tips but i had already rolled back to version 5.1 & now all my transfer speeds seems to be alright again.

[ade333]

Strange that 5.5 caused that for you but it is still new as far as I am concerned so I guess it's not that unexpected.

For anyone else following this thread, the issue appears to be resolved.  I contacted MS and obtained the hotfix for Server 2008 R2.  You need to call and reference this article.

You experience performance issues in applications and services when the system file cache consumes ...

[lowdownshame]

good info. thanks