1 2 Previous Next 29 Replies Latest reply on Oct 6, 2008 8:03 AM by stromcooper

    fastest network copy of .vmdk files between 2 esx servers

    iaco Novice

      scp is indeed the safest way, but it's slow especially if the image files are half empty like a tipical server

      here is a way to copy files while compressing on the fly:

      let's say you want to copy "file" from "machine1" to "machine2":

       

      ssh machine1

      gzip -1 </path/file | ssh machine2 gunzip ">" /path/file

       

      on a standard 2000 server installation 4G vmdk, with a 100Mb ethernet link, using scp you can have a throughput of 8MB/sec

      with the above command I've obtained 13MB/sec.

        • 1. Re: fastest network copy of .vmdk files between 2 esx servers
          larstr Virtuoso
          vExpert

          Thanks iaco for testing this out!

          I always thought that ncftp was fastest, but after testing this I've realized that your method is better. I've been copying a 4Gb linux system disk between two x440 systems.

           

          However. I managed to tune this command to be even faster:

          \# gzip -1 </vmfs/path/file | ssh -c blowfish-cbc machine2 gunzip ">" /vmfs/path/file

          /code

           

          On my system I had the following results for my 4Gb system disk.

          Your original command:     9 min 32 sec

          My ncftp command:          9 min 46 sec

          gzip +ssh with blowfish:   8 min 27 sec

          /pre

          As we can see it was a bit faster using blowfish-cbc over the default ciphering. I suspect that we could have gotten the copying done even faster if netcat had been a standard utility of the COS.

           

          One thing one must have in mind when copying files to a vmfs volume is that each time a vmfs file grows there is put a scsi reservation on that LUN. This is something that might affect the performance of the other active VMs that have their disk files on that LUN. When a file is copied it grows a great number of times during this copy.

           

          Another thing is that this copying works well for vmfs<->vmfs volumes. For copying to other file systems it's recommended to use vmkfstools for exporting the files first.

           

          Copying/exporting/importing with vmkfstools also doesn't affect the performance of the LUN as badly because it only issues 1 scsi reservation.

           

          If your two servers can access the same LUN of the SAN it's fastest to use:

          \# vmkfstools -i /vmfs/path/file /vmfs/sharedlun/file

          /code

          On my system the vmkfstools copy took 2 min 48 sec[/i] for the same 4Gb file.

           

          Lars

          • 2. Re: fastest network copy of .vmdk files between 2 esx servers
            devzero Master

            you may use a combination of lzop and netcat/socat - at least you should take lzop into account, because it can compress/decrompress large amounts of data astonishingly fast

             

             

            example:

             

            destination host: socat -u - tcp4-listen:2342 | $\{UNZIP} | tar x -C /dest/dir

            source host: tar c /source/dir | $\{ZIP} | socat -u - tcp4:desthost:2342

             

            ZIP can be one of

             

                 

            • cat

                 

            • lzop

                 

            • gzip

                 

            • bzip2

             

            und UNZIP:

             

                 

            • cat

                 

            • lzop -d

                 

            • gunzip

                 

            • bunzip2

             

            regards

            roland

            • 3. Re: fastest network copy of .vmdk files between 2 esx servers
              larstr Virtuoso
              vExpert

              you may use a combination of lzop and netcat/socat -

               

              Yes, but as these tools not included in the esx server distro I guess that would be an unsupported solution.

               

              Lars

              • 4. Re: fastest network copy of .vmdk files between 2 esx servers
                devzero Master

                do you really need "support" for just copying files ?

                 

                regards

                roland

                • 5. Re: fastest network copy of .vmdk files between 2 esx servers
                  larstr Virtuoso
                  vExpert

                  do you really need "support" for just copying files

                  ?

                  Not really. But you'll have to compile these tools yourself or find the corresponding rh7.2 rpm files. I guess these tools don't raise any prerequisites that will affect the system negatively so it's probably no big deal.

                  How much performance gain do you get using these tools?

                   

                  Lars

                  • 6. Re: fastest network copy of .vmdk files between 2 esx servers
                    devzero Master

                    lzop is at least twice as fast as gzip - depending on the type of data, the performance advantage could be even bigger

                    • 7. Re: fastest network copy of .vmdk files between 2 esx servers
                      larstr Virtuoso
                      vExpert

                      lzop is at least twice as fast as gzip - depending on

                      the type of data, the performance advantage could be

                      even bigger

                       

                      ok.. you made me download these tools and test them. You're of course right. These tools are faster:

                      lzop -1 through ssh+blowfish        7min 39sec

                      lzop -1 through netcat+tcp          6min 37sec

                      /pre

                       

                      Netcat over udp is probably even faster, but I couldn't find the correct syntax to perform it right now (defektes Rohr).

                       

                      Lars

                      • 8. Re: fastest network copy of .vmdk files between 2 esx servers
                        larstr Virtuoso
                        vExpert

                        Forgot to add that a normal scp[/b] was 10 min 33 sec.

                         

                        Lars

                        • 9. Re: fastest network copy of .vmdk files between 2 esx servers
                          devzero Master

                          >Netcat over udp is probably even faster, but I couldn't find the correct

                          >syntax to perform it right now (defektes Rohr).

                           

                          whats the problem ?

                           

                          on remote system:

                          netcat -l -p 2222 -u

                           

                          on local system:

                          echo "Hello" | netcat -u localhost 2222

                           

                          roland

                          • 10. Re: fastest network copy of .vmdk files between 2 esx servers
                            larstr Virtuoso
                            vExpert

                            Yes, I know. It's basicly the same as with tcp. With udp I get the following on the receiving side:

                            \# netcat  -u  -l -p 12322|lzop -d > deb31system.vmdk

                            lzop: : Compressed data violation

                            write(stdout): Broken pipe

                            /code

                             

                            Without the -u switch it works great. I also get the similar results for gzip.

                             

                            Do you have a clue what I'm doing wrong here?

                             

                            Lars

                            • 11. Re: fastest network copy of .vmdk files between 2 esx servers
                              petr Champion
                              VMware Employees

                              That you are using udp...  UDP does not guarantee that data are not lost, so sooner or later you'll lost some packet and as netcat over udp just blasts data without any retransmits/handshake, of course once you lost packet whole stream is lost.

                               

                              I have no idea what's purpose of -u option on netcat...  It is certainly unusable for anything serious - from looking at sources it seems to be designed just to load network...

                              • 12. Re: fastest network copy of .vmdk files between 2 esx servers
                                larstr Virtuoso
                                vExpert

                                That you are using udp...  UDP does not guarantee<br>

                                that data are not lost, so sooner or later you'll<br>

                                lost some packet and as netcat over udp just blasts<br>

                                data without any retransmits/handshake, of course<br>

                                once you lost packet whole stream is lost.<br>

                                 

                                Yes, petr. You seem to be right on to the point here. udp transfers seems work for smaller files, but not for big ones because of this. tcp is the way to go unless we implement some kind of tcp-like controls with less overhead than tcp.

                                 

                                Anyhow.. I'm still testing this out and by using ttcp(another non standard esx app) I can get even better performance.

                                lzop -1 + ttcp -l24576 -n10000        5min 53sec

                                /pre

                                 

                                 

                                Lars

                                • 13. Re: fastest network copy of .vmdk files between 2 esx servers
                                  devzero Master

                                  netcat is network admin`s and hacker's swiss army knife.

                                  it is the favourite console interface to tcp and udp

                                   

                                   

                                  from an very old mailing list posting :

                                   

                                  From: \*Hobbit* <hobbit_at_avian.org>

                                  Date: Sat, 28 Oct 1995 13:40:21 -0400

                                   

                                  [Potentially interesting to this group since it helps facilitate bug testing,

                                  port scanning, address spoofing, buffer overflowing, or what have you.]

                                   

                                  Netcat 1.0 is the initial release of a simple Unix utility which reads and

                                  writes data across network connections, using TCP or UDP protocol. It is

                                  designed to be a reliable "back-end" tool that can be used directly or easily

                                  driven by other programs and scripts. At the same time it is a feature-rich

                                  network debugging and exploration tool, since it can create almost any kind of

                                  connection you would need and has several interesting built-in capabilities.

                                   

                                  Some of netcat's major features are:

                                   

                                          Outbound or inbound connections, TCP or UDP, to or from any ports

                                          Full DNS forward/reverse checking, with appropriate warnings

                                          Ability to use any local source port

                                          Ability to use any locally-configured network source address

                                          Built-in port-scanning capabilities, with randomizer

                                          Built-in loose source-routing capability

                                          Can read command line arguments from standard input

                                          Slow-send mode, one line every N seconds

                                          Optional ability to let another program service inbound connections

                                   

                                  A very short list of potential uses:

                                   

                                          Script backends

                                          Scanning ports and inventorying services

                                          Backup handlers

                                          File transfers

                                          Server testing and simulation

                                          Firewall testing

                                          Proxy gatewaying

                                          Network performance testing

                                          Address spoofing tests

                                          Protecting X servers

                                          1001 other uses you'll likely come up with

                                   

                                  Netcat is a product of Avian Research, and is freely available in full source

                                  form with no restrictions save an obligation to give credit where due. Get

                                  it via anonymous FTP at avian.org:/src/hacks/nc100.tgz which is a gzipped

                                  tar file. Other distribution formats can be accomodated upon request.

                                  It is also mirrored at the following \[faster] sites:

                                   

                                          zippy.telcom.arizona.edu:/pub/mirrors/avian.org/hacks/nc100.tgz

                                          ftp.sterling.com:/mirrors/avian.org/src/hacks/nc100.tgz

                                   

                                  _H* 951010

                                  Received on Oct 28 1995

                                  • 14. Re: fastest network copy of .vmdk files between 2 esx servers
                                    larstr Virtuoso
                                    vExpert

                                    Maybe you should file a feature request that they can include it in the next version?

                                     

                                    Lars

                                     

                                    Message was edited by:

                                            larstr

                                    sp

                                    1 2 Previous Next