13 Replies Latest reply on Feb 16, 2016 12:54 PM by brickleberry

    Cloning process very slow.

    ita_mbcl Lurker

      Hello,

       

      I have a strange behavior when using ghettoVCB from one ESXi host.

       

      The source storage is the local RAID and the destination for the backup is an NFS drive.

      The VM that needs to be backed up has to .vmdk files - 150GB and 400GB.

      After the snapshot has been done the cloning of the disk is very slow.

      I can monitor the speed on the NFS and for the incoming traffic is only 2-4MB/s. Both the ESXi and the NFS are on the same 1Gbit/s switch both having 1Gbit/s NICs.

       

      I am able to backup a VM from a different ESXi host and the speed is considerably higher - ~30-40MB/s. This VM is smaller though - one .vmdk 10GB.

       

      Here is the information about the environment.

       

      # ghettoVCB for ESX/ESXi 3.5, 4.x+ and 5.x

      # Author: William Lam

      # http://www.virtuallyghetto.com/

      # Documentation: http://communities.vmware.com/docs/DOC-8760

      # Created: 11/17/2008

      # Last modified: 2013_01_11 Version 0

       

      VMware ESXi 5.0.0 build-623860

       

      Any ideas anybody?

        • 1. Re: Cloning process very slow.
          Phatsta Enthusiast

          I have seen this issue several times, and in almost all of these cases it’s been due to network performance and/or NAS/NFS drive performance. For me personally, a good rule of thumb is to make sure there’s a gigabit connection between NFS drive and esxi box, and only use previously tested NAS’es that I know have good performance. In my own office setup I use a QNAP NAS with raid6. I usually get somewhere around 20-25MB/s, which usually is good enough as long as you’re running thin disks and try to optimize the setup as best you can.

           

          You can always try to disable write cache for EXT4, that might also help some.

           

          /Daniel

           

           

          Från: ita_mbcl communities-emailer@vmware.com

          Skickat: den 1 oktober 2013 12:44

          Till: Daniel Jokinen

          Ämne: New message: "Cloning process very slow."

           

          VMware Communities<https://communities.vmware.com/index.jspa>

           

          Cloning process very slow.

          created by ita_mbcl<https://communities.vmware.com/people/ita_mbcl> in ghettoVCB - View the full discussion<https://communities.vmware.com/message/2294242#2294242>

          • 2. Re: Cloning process very slow.
            Aristodemos Lurker

            I have the same issue in several hosts. The only difference is that I'm backing up to a iSCSI target. The backup is relatively fast in some VMs, slow in others.

            • 3. Re: Cloning process very slow.
              Phatsta Enthusiast

              What disk type are you using? (Look under the “edit settings” for the VM).

               

              If they are thick, a backup will clone the entire virtual drive including empty space (sector by sector). If they are thin, it will clone only the data and the empty space will take almost no time at all.

               

              If you would like to change the disk type from thick to thin, I find it easiest to do with VMware Converter, but there are other ways too.

               

              /Daniel

               

               

              Från: Aristodemos communities-emailer@vmware.com

              Skickat: den 1 oktober 2013 13:29

              Till: Daniel Jokinen

              Ämne: New message: "Cloning process very slow."

               

              VMware Communities<https://communities.vmware.com/index.jspa>

               

              Cloning process very slow.

              created by Aristodemos<https://communities.vmware.com/people/Aristodemos> in ghettoVCB - View the full discussion<https://communities.vmware.com/message/2294383#2294383>

              • 4. Re: Cloning process very slow.
                ita_mbcl Lurker

                The issue is I assume with the ESXi server that is cloning to the NFS server too slow... I have tried copying files over SSH to both ESXi servers and the difference in speed was obvious.

                 

                However in the ESXi that is cloning slow the network card shows that it is 1Gbit/s full duplex... so I am now a bit confused. The NFS can write about 30MB/s, however the ESXi doesn't seem to be able to copy to the NFS fast enough... any ideas what I can test?

                • 5. Re: Cloning process very slow.
                  Phatsta Enthusiast

                  We're actually talking two separate issues here, the way I see it. One is overall slow backups, another is copy speed.


                  Slow backups can be caused simply by the VM being very large; Hence my counter question about what disk type you're using.

                  Slow copy speed can be caused by poor NFS performance, disk performance, network performance etc. But my first advice is to try and turn off write cache for EXT4 if you have that option in your NAS.


                  You say your NAS can perform at 30MB/s, is that factory specs or your own tests with other protocols or what? Because by factory specs my NAS is suppose to manage something like 50MB/s but it never does in reality, at least not with NFS.

                  • 6. Re: Cloning process very slow.
                    ita_mbcl Lurker

                    Right, I forgot to mention before... sorry

                     

                    The VM1 that is cloned fast is thin provisioned.

                    The VM2 that is cloned slow is Lazy Zero provisioned.

                     

                    The VMs reside on different ESXi hosts.

                    VM1 is on a some Dell desktop-turned-esxi  - ESXi1

                    VM2 is on Dell R270 with RAID10 - ESXi2

                     

                    Both ESXi are connected to 1Gbit/s network.

                     

                    The copy speed to the NFS server however I monitor on the NFS server. It is a Synology DiskStation 413. When I back up VM1 i see the average speed to be ~30MB/s, when I backup VM2 I see ~3MB/s.

                     

                    I think that no matter if it is thin or think provision the data needs to be copied on the NFS and this speed is very slow for VM2 from ESXi.

                     

                    I tried copying big .iso file to the ESXi hosts - so no cloning processes, just testing network speed. When I copied to ESXi1 it was much faster than when I copied the file to ESXi2.

                     

                    Now, ESXi servers use only one 1Gbit NIC and in the settings it showed that both NICs are at 1000 Full-duplex.

                    • 7. Re: Cloning process very slow.
                      Phatsta Enthusiast

                      Okay this sounds familiar... I inherited a customer that was sitting on two Dell R270 esxi servers just a year ago. The performance in those shoeboxes was to frustratingly slow that I had to throw them out 6 months later. Sorry to say I couldn't run 2 simultanious VM's without everything getting super-slow, services wouldn't start etc...

                       

                      I remember a thick VM of about 550GB took about 3 days to complete, even though it was 1Gbit all the way to the NAS (which funny enough also happened to be a Synology, but a 212 or somehing like that...). After I replaced the Dells with a Supermicro server for about $3000, the same thick VM took about 20 hours to complete. And later when I converted it to thin disks, it took about 10 hours. I can't remember what the MB/s was before though, but now it's somewhere around 20-30MB/s depending on network load.

                       

                      One thing you could try though; The R270 has two network cards, right? Hook them both up and setup load balancing in esxi. Even though it's 1Gbit, it might get crowded and if so dual cards will give you a bit of a boost.

                      1 person found this helpful
                      • 8. Re: Cloning process very slow.
                        ita_mbcl Lurker

                        Yes this is pretty much my case.

                         

                        OK, I am starting a backup tonight and i will let it run until it is finished - so we will see if the backup is successful.

                         

                        Thank you for the feedback today.

                        • 9. Re: Cloning process very slow.
                          Phatsta Enthusiast

                          No problem at all, I hope it helps in some way.

                           

                          My experience is that much can happen in 3 days, I remember those backups failing more often than they were successful. Maybe if you go dual NIC you might have a bigger chance of getting it through, I don't know. Good luck anyways.

                          • 10. Re: Cloning process very slow.
                            lyncker Novice

                            Dear Forum,

                             

                            I have a similar problem. But in my case I am not cloning between to hosts using a NAS or a second ESXI. I have one ESXi with 2 physical harddrives.

                            My VM is about 500 GB and I allways cloned it in more or less 20 minutes.

                            But now something changed. After a crash I cloned a backup-clone back to the live HD. Now the cloning process needs about 20 hs.

                             

                            What changed, who can I fix this ?

                             

                            Thanks for help,

                             

                            Fil

                            • 11. Re: Cloning process very slow.
                              neilbert Lurker

                              Hi Guys,

                               

                              We have had 18 months of slow cloning and slow copying of files between two brand new DELL T520 servers with v5.5. Copy/Clone from server to itself or server to server is slow (500MBytes a minute); copy/clone from server to iSCSI is OK (3 GBytes/minute). Copying iSCSI to server was slow again. Each of these machines has 48Gb RAM and has only 2 working VMs!

                               

                              We asked others to try this speed test on their servers and they were getting up to 20GBytes/min.

                               

                              We built another system (v5.0) on old servers. We were getting 20GBytes/min.

                               

                              Despite many hours with VMWare Tech support over 18 months - upgrade (ESXi) network card drivers, upgrade server firmware, upgrade (ESXi) RAID drivers, discussions with DELL and VMWare, separately and together, everyone agreeing that the hardware/firmware/software is correct - we are still copying/cloning slowly. Interestingly, the copy speed within the VMs was also slow i.e copying from NAS or iSCSI to a C:\ drive was slow as well. Also, if you take out all the network leads (apart from the Management network - separate subnet) and you try to copy/clone to itself, it is still slow! Only copying to an external iSCSI is reasonable.

                               

                              So, we are still trying to solve this problem. However, yesterday we discovered something very important. We copied a large file from the NAS to the C:\ drive on one of the VMs, it performed at less than 10MByte/s ( = circa. slow copy/cloning rate) as we have always seen. So, we turned of the vCenter server. WOW, the copying of the same file, on the same server was 100MBytes/s.

                               

                              We have just sent this information back to VMWare Tech Support. Starting up the vCenter Server slows down transfer rates within the VMs by a factor of 10. This does not right to me. If you have this problem, I strongly recommend that you do this test. It can save you hours of upgrading firmware and drivers which means system downtime (because it take 15 hours for us to "move" each server, we don't have the luxury of quick switch over).

                               

                              After 18 months of this very slow copy or very slow cloning, we might be getting somewhere...hopefully. But the answer seems to be a problem within ESXi/vCenter server.

                              • 12. Re: Cloning process very slow.
                                dan.linder Lurker

                                neilbert- Any update from VMware support?

                                • 13. Re: Cloning process very slow.
                                  brickleberry Novice

                                  an update? that would require support to follow through with their J.O.B.

                                   

                                  the job they are paid to do.

                                   

                                  their paid support has gone to s__t.

                                   

                                  you have no answer, you never will.

                                   

                                  their main task is to not give any solution, wait for it to be forgotten about, take undeserved $$$.