Thanks iaco for testing this out!

I always thought that ncftp was fastest, but after testing this I've realized that your method is better. I've been copying a 4Gb linux system disk between two x440 systems.

However. I managed to tune this command to be even faster:

\# gzip -1 </vmfs/path/file | ssh -c blowfish-cbc machine2 gunzip ">" /vmfs/path/file

/code

On my system I had the following results for my 4Gb system disk.

Your original command:     9 min 32 sec

My ncftp command:          9 min 46 sec

gzip +ssh with blowfish:   8 min 27 sec

/pre

As we can see it was a bit faster using blowfish-cbc over the default ciphering. I suspect that we could have gotten the copying done even faster if netcat had been a standard utility of the COS.

One thing one must have in mind when copying files to a vmfs volume is that each time a vmfs file grows there is put a scsi reservation on that LUN. This is something that might affect the performance of the other active VMs that have their disk files on that LUN. When a file is copied it grows a great number of times during this copy.

Another thing is that this copying works well for vmfs<->vmfs volumes. For copying to other file systems it's recommended to use vmkfstools for exporting the files first.

Copying/exporting/importing with vmkfstools also doesn't affect the performance of the LUN as badly because it only issues 1 scsi reservation.

If your two servers can access the same LUN of the SAN it's fastest to use:

\# vmkfstools -i /vmfs/path/file /vmfs/sharedlun/file

/code

On my system the vmkfstools copy took 2 min 48 sec[/i] for the same 4Gb file.

Lars

you may use a combination of lzop and netcat/socat - at least you should take lzop into account, because it can compress/decrompress large amounts of data astonishingly fast

example:

destination host: socat -u - tcp4-listen:2342 | $\{UNZIP} | tar x -C /dest/dir

source host: tar c /source/dir | $\{ZIP} | socat -u - tcp4:desthost:2342

ZIP can be one of

• cat

• lzop

• gzip

• bzip2

und UNZIP:

• cat

• lzop -d

• gunzip

• bunzip2

regards

roland

you may use a combination of lzop and netcat/socat -

Yes, but as these tools not included in the esx server distro I guess that would be an unsupported solution.

Lars

do you really need "support" for just copying files ?

regards

roland

do you really need "support" for just copying files

?

Not really. But you'll have to compile these tools yourself or find the corresponding rh7.2 rpm files. I guess these tools don't raise any prerequisites that will affect the system negatively so it's probably no big deal.

How much performance gain do you get using these tools?

Lars

lzop is at least twice as fast as gzip - depending on the type of data, the performance advantage could be even bigger

lzop is at least twice as fast as gzip - depending on

the type of data, the performance advantage could be

even bigger

ok.. you made me download these tools and test them. You're of course right. These tools are faster:

lzop -1 through ssh+blowfish        7min 39sec

lzop -1 through netcat+tcp          6min 37sec

/pre

Netcat over udp is probably even faster, but I couldn't find the correct syntax to perform it right now (defektes Rohr).

Lars

Forgot to add that a normal scp[/b] was 10 min 33 sec.

Lars

>Netcat over udp is probably even faster, but I couldn't find the correct

>syntax to perform it right now (defektes Rohr).

whats the problem ?

on remote system:

netcat -l -p 2222 -u

on local system:

echo "Hello" | netcat -u localhost 2222

roland

Yes, I know. It's basicly the same as with tcp. With udp I get the following on the receiving side:

\# netcat  -u  -l -p 12322|lzop -d > deb31system.vmdk

lzop: : Compressed data violation

write(stdout): Broken pipe

/code

Without the -u switch it works great. I also get the similar results for gzip.

Do you have a clue what I'm doing wrong here?

Lars

That you are using udp...  UDP does not guarantee that data are not lost, so sooner or later you'll lost some packet and as netcat over udp just blasts data without any retransmits/handshake, of course once you lost packet whole stream is lost.

I have no idea what's purpose of -u option on netcat...  It is certainly unusable for anything serious - from looking at sources it seems to be designed just to load network...

That you are using udp...  UDP does not guarantee<br>

that data are not lost, so sooner or later you'll<br>

lost some packet and as netcat over udp just blasts<br>

data without any retransmits/handshake, of course<br>

once you lost packet whole stream is lost.<br>

Yes, petr. You seem to be right on to the point here. udp transfers seems work for smaller files, but not for big ones because of this. tcp is the way to go unless we implement some kind of tcp-like controls with less overhead than tcp.

Anyhow.. I'm still testing this out and by using ttcp(another non standard esx app) I can get even better performance.

lzop -1 + ttcp -l24576 -n10000        5min 53sec

/pre

Lars

netcat is network admin`s and hacker's swiss army knife.

it is the favourite console interface to tcp and udp

from an very old mailing list posting :

From: \*Hobbit* <hobbit_at_avian.org>

Date: Sat, 28 Oct 1995 13:40:21 -0400

[Potentially interesting to this group since it helps facilitate bug testing,

port scanning, address spoofing, buffer overflowing, or what have you.]

Netcat 1.0 is the initial release of a simple Unix utility which reads and

writes data across network connections, using TCP or UDP protocol. It is

designed to be a reliable "back-end" tool that can be used directly or easily

driven by other programs and scripts. At the same time it is a feature-rich

network debugging and exploration tool, since it can create almost any kind of

connection you would need and has several interesting built-in capabilities.

Some of netcat's major features are:

        Outbound or inbound connections, TCP or UDP, to or from any ports

        Full DNS forward/reverse checking, with appropriate warnings

        Ability to use any local source port

        Ability to use any locally-configured network source address

        Built-in port-scanning capabilities, with randomizer

        Built-in loose source-routing capability

        Can read command line arguments from standard input

        Slow-send mode, one line every N seconds

        Optional ability to let another program service inbound connections

A very short list of potential uses:

        Script backends

        Scanning ports and inventorying services

        Backup handlers

        File transfers

        Server testing and simulation

        Firewall testing

        Proxy gatewaying

        Network performance testing

        Address spoofing tests

        Protecting X servers

        1001 other uses you'll likely come up with

Netcat is a product of Avian Research, and is freely available in full source

form with no restrictions save an obligation to give credit where due. Get

it via anonymous FTP at avian.org:/src/hacks/nc100.tgz which is a gzipped

tar file. Other distribution formats can be accomodated upon request.

It is also mirrored at the following \[faster] sites:

        zippy.telcom.arizona.edu:/pub/mirrors/avian.org/hacks/nc100.tgz

        ftp.sterling.com:/mirrors/avian.org/src/hacks/nc100.tgz

_H* 951010

Received on Oct 28 1995

Maybe you should file a feature request that they can include it in the next version?

Lars

Message was edited by:

        larstr

sp