File copy to ESX/ESXi improving throughput?

[aspex]

On a 10Gbps network I can't get file transfers any better throughput than on a 1Gbps network to either ESX 4 or ESXi 5 hosts using either scp or the SDK provided fileaccess.pl perl script.  I noticed in both environments during a transfer that the limitation seems to be the processing power of a core.  On ESX core 0 went to 100% and found out through the SR that is working by design ESX uses only core 0 for system management.  On ESXi v5 it had a similar problem with using a core to 100% but simply moved the processing thread from core to core.  This is because these two methods employ encryption and the single threaded receive/decryption is limited to a single core.  FYI I had sniffed the network (no issues) and confirmed all NICs and switch settings were set to max speed and full duplex.  I'm not a networking novice.  Also, I can scp the same 12G vmdk file to a file server on the network and achieve throughput more than double to the ESX/ESXi hosts.

The tech support person then confirmed this is a limitation of scp/fileaccess.pl and that both methods are not supported methods to transfer files to ESX and I need to use "network file copy" to get better throughput (or go through VCenter however not all my ESX hosts are fronted by VCenter so I still need a solution for my standalone hosts).  I downloaded the Converter Standalone and it is a Windows binary.  I need to do this on Linux.  The "open source" version is simply a collection of oss libraries used.  Not a binary that can be run.  I was then told to use the Converter SDK to use "network file copy" protocol APIs.  My SR was transfered to the SDK group and they said this is the only forum for support.  Seems bizarre this is the only forum to get support for an SDK (the VCenter SDK has support).

Therefore, how do I utilize "network file copy" protocol through the Converter SDK APIs?  In reading the Converter SDK documentation I came across this page which is the same API fileaccess.pl uses.

http://www.vmware.com/support/developer/converter-sdk/conv50_apireference/vim.FileManager.html

Therefore I assume this is not using the "network file copy" protocol because it doesn't perform any better than an scp file transfer.

In addition, I found an Intel/VMware white paper that documents problems with secure file transfer to ESX and how they resolved it.  The problem is that this solution reqiures maintaining a VM image on the ESX host to do the file transfer which is not desireable or easily maintainable.

http://download.intel.com/support/network/sb/fedexcasestudyfinal.pdf

Any other thoughts on how to get better throughput on a 10Gbps network secure file transfer other than the "network file copy" protocol?

I've searched these forums and not found an answer.

[patanassov]

Hello

I am a little surprised the support advised you that NFC is faster than SCP; Converter has performance issues with NFC on its own (see http://communities.vmware.com/message/1851162).

Any kind of cloning using Converter that has ESX at at least at one end uses NFC, but the API is on a higher level and therefore there is nothing in the API itself about the file transfer protocol.

I was told that Veeam FastScp can achieve high speed cloning to/from ESX but have never tried it.

HTH

Plamen

[aspex]

Thanks for the link to the other discussion.  Certainly not sure why I'm getting the advise I am getting from VMware.

While Veeam might be an option I do not think so in my case.  It is clearly a limitation of the way scp works and the processing power of a single core on the ESX/ESXi host.

So looks like there is not going to be a solution to this problem.