I have seen this issue several times, and in almost all of these cases it’s been due to network performance and/or NAS/NFS drive performance. For me personally, a good rule of thumb is to make sure there’s a gigabit connection between NFS drive and esxi box, and only use previously tested NAS’es that I know have good performance. In my own office setup I use a QNAP NAS with raid6. I usually get somewhere around 20-25MB/s, which usually is good enough as long as you’re running thin disks and try to optimize the setup as best you can.
You can always try to disable write cache for EXT4, that might also help some.
Från: ita_mbcl firstname.lastname@example.org
Skickat: den 1 oktober 2013 12:44
Till: Daniel Jokinen
Cloning process very slow.
created by ita_mbcl<https://communities.vmware.com/people/ita_mbcl> in ghettoVCB - View the full discussion<https://communities.vmware.com/message/2294242#2294242>
I have the same issue in several hosts. The only difference is that I'm backing up to a iSCSI target. The backup is relatively fast in some VMs, slow in others.
What disk type are you using? (Look under the “edit settings” for the VM).
If they are thick, a backup will clone the entire virtual drive including empty space (sector by sector). If they are thin, it will clone only the data and the empty space will take almost no time at all.
If you would like to change the disk type from thick to thin, I find it easiest to do with VMware Converter, but there are other ways too.
Från: Aristodemos email@example.com
Skickat: den 1 oktober 2013 13:29
Till: Daniel Jokinen
Cloning process very slow.
created by Aristodemos<https://communities.vmware.com/people/Aristodemos> in ghettoVCB - View the full discussion<https://communities.vmware.com/message/2294383#2294383>
The issue is I assume with the ESXi server that is cloning to the NFS server too slow... I have tried copying files over SSH to both ESXi servers and the difference in speed was obvious.
However in the ESXi that is cloning slow the network card shows that it is 1Gbit/s full duplex... so I am now a bit confused. The NFS can write about 30MB/s, however the ESXi doesn't seem to be able to copy to the NFS fast enough... any ideas what I can test?
We're actually talking two separate issues here, the way I see it. One is overall slow backups, another is copy speed.
Slow backups can be caused simply by the VM being very large; Hence my counter question about what disk type you're using.
Slow copy speed can be caused by poor NFS performance, disk performance, network performance etc. But my first advice is to try and turn off write cache for EXT4 if you have that option in your NAS.
You say your NAS can perform at 30MB/s, is that factory specs or your own tests with other protocols or what? Because by factory specs my NAS is suppose to manage something like 50MB/s but it never does in reality, at least not with NFS.
Right, I forgot to mention before... sorry
The VM1 that is cloned fast is thin provisioned.
The VM2 that is cloned slow is Lazy Zero provisioned.
The VMs reside on different ESXi hosts.
VM1 is on a some Dell desktop-turned-esxi - ESXi1
VM2 is on Dell R270 with RAID10 - ESXi2
Both ESXi are connected to 1Gbit/s network.
The copy speed to the NFS server however I monitor on the NFS server. It is a Synology DiskStation 413. When I back up VM1 i see the average speed to be ~30MB/s, when I backup VM2 I see ~3MB/s.
I think that no matter if it is thin or think provision the data needs to be copied on the NFS and this speed is very slow for VM2 from ESXi.
I tried copying big .iso file to the ESXi hosts - so no cloning processes, just testing network speed. When I copied to ESXi1 it was much faster than when I copied the file to ESXi2.
Now, ESXi servers use only one 1Gbit NIC and in the settings it showed that both NICs are at 1000 Full-duplex.
1 person found this helpful
Okay this sounds familiar... I inherited a customer that was sitting on two Dell R270 esxi servers just a year ago. The performance in those shoeboxes was to frustratingly slow that I had to throw them out 6 months later. Sorry to say I couldn't run 2 simultanious VM's without everything getting super-slow, services wouldn't start etc...
I remember a thick VM of about 550GB took about 3 days to complete, even though it was 1Gbit all the way to the NAS (which funny enough also happened to be a Synology, but a 212 or somehing like that...). After I replaced the Dells with a Supermicro server for about $3000, the same thick VM took about 20 hours to complete. And later when I converted it to thin disks, it took about 10 hours. I can't remember what the MB/s was before though, but now it's somewhere around 20-30MB/s depending on network load.
One thing you could try though; The R270 has two network cards, right? Hook them both up and setup load balancing in esxi. Even though it's 1Gbit, it might get crowded and if so dual cards will give you a bit of a boost.
Yes this is pretty much my case.
OK, I am starting a backup tonight and i will let it run until it is finished - so we will see if the backup is successful.
Thank you for the feedback today.
No problem at all, I hope it helps in some way.
My experience is that much can happen in 3 days, I remember those backups failing more often than they were successful. Maybe if you go dual NIC you might have a bigger chance of getting it through, I don't know. Good luck anyways.
I have a similar problem. But in my case I am not cloning between to hosts using a NAS or a second ESXI. I have one ESXi with 2 physical harddrives.
My VM is about 500 GB and I allways cloned it in more or less 20 minutes.
But now something changed. After a crash I cloned a backup-clone back to the live HD. Now the cloning process needs about 20 hs.
What changed, who can I fix this ?
Thanks for help,
We have had 18 months of slow cloning and slow copying of files between two brand new DELL T520 servers with v5.5. Copy/Clone from server to itself or server to server is slow (500MBytes a minute); copy/clone from server to iSCSI is OK (3 GBytes/minute). Copying iSCSI to server was slow again. Each of these machines has 48Gb RAM and has only 2 working VMs!
We asked others to try this speed test on their servers and they were getting up to 20GBytes/min.
We built another system (v5.0) on old servers. We were getting 20GBytes/min.
Despite many hours with VMWare Tech support over 18 months - upgrade (ESXi) network card drivers, upgrade server firmware, upgrade (ESXi) RAID drivers, discussions with DELL and VMWare, separately and together, everyone agreeing that the hardware/firmware/software is correct - we are still copying/cloning slowly. Interestingly, the copy speed within the VMs was also slow i.e copying from NAS or iSCSI to a C:\ drive was slow as well. Also, if you take out all the network leads (apart from the Management network - separate subnet) and you try to copy/clone to itself, it is still slow! Only copying to an external iSCSI is reasonable.
So, we are still trying to solve this problem. However, yesterday we discovered something very important. We copied a large file from the NAS to the C:\ drive on one of the VMs, it performed at less than 10MByte/s ( = circa. slow copy/cloning rate) as we have always seen. So, we turned of the vCenter server. WOW, the copying of the same file, on the same server was 100MBytes/s.
We have just sent this information back to VMWare Tech Support. Starting up the vCenter Server slows down transfer rates within the VMs by a factor of 10. This does not right to me. If you have this problem, I strongly recommend that you do this test. It can save you hours of upgrading firmware and drivers which means system downtime (because it take 15 hours for us to "move" each server, we don't have the luxury of quick switch over).
After 18 months of this very slow copy or very slow cloning, we might be getting somewhere...hopefully. But the answer seems to be a problem within ESXi/vCenter server.
an update? that would require support to follow through with their J.O.B.
the job they are paid to do.
their paid support has gone to s__t.
you have no answer, you never will.
their main task is to not give any solution, wait for it to be forgotten about, take undeserved $$$.