Advice on maximizing the speed that vSphere can clone VMs

[DavidMcKnight]

I have a cluster of VM Hosts that have dual 10Gig NICs. I have a PowerCLI script that will clone out 50 VMs one after another.  I'm using Starwind as my iSCSI storage solution.  When I clone a single VM I'm seeing 6 to 8 gigabit of traffic on my Starwind box.  If I clone two VMs at once I see 14 to 15 gigabit of traffic.

My question is why don't I get 14 to 15 (or better yet 20) gigabit of traffic when cloning a single VM?

I have the bandwidth.  If I run ATTO Disk Benchmark inside a single VM on one of my hosts, it can generate over 20 gigabit of traffic.  I assume a lot of that is hitting the Starwind cache, but at least I know a single host and the Starwind service can saturate my network bandwidth, which does not happen when cloning.

I know I have more storage speed since I can clone two VMs to the same iSCSI target and take only maybe a 10% hit on the storage speed.

Is there some advanced setting that tells vSphere not to use all the available network/storage capacity while cloning?

I'd appreciate any advice on maximizing the speed that vSphere can clone VMs,

Thanks,

David

[AnatolyVilchins]

Hi!

There are a lot of things to start with:

1.You were saying that cloning of 2 VMs consumes 10% of the storage speed. May I ask you to clarify where have you got those numbers?

2.Have you used MPIO or NIC teaming?

3.What device type of StarWind have you used?

4.Have you ensured the speed of the network and the disk subsystem by benchmarking with the proper tool (i.e.iperf, iometer or analogues)?

4 Have you ever seen full utilization of your dual port NIC?

By answering those questions you`ll help community to isolate what area we should look at - network, disks, esx, etc.

Thanks

[tomtom901]

Does your storage array support VAAI (perhaps via a plugin)? Using VAAI, you are able to offload certain tasks to the storage array. Rather than letting the ESXi host copy the blocks over the network, you ask the storage array to do it for you, which can be a big help. If under configuration -> Storage your datastores show Supported under Hardware Acceleration, you're already using VAAI.

If not, something definitely worth looking at.

[DavidMcKnight]

1.You were saying that cloning of 2 VMs consumes 10% of the storage speed. May I ask you to clarify where have you got those numbers?

What I was trying to describe was that when cloning two VMs the network traffic to the datastore nearly doubles.  Nearly meaning the traffic is 190% that of cloning one VM.  All I am doing is looking at the network graph and numbers the Performance Monitor on the datastore and the actual time it takes to clone.

2.Have you used MPIO or NIC teaming?

I'm doing iSCSI port binding on the two 10gig ports on the VMHost.  So I end up with 4 paths to each iSCSI volume.  Then I do round robin across the 4 paths.

3.What device type of StarWind have you used?

I'm doing standard img files for the datastore.

4.Have you ensured the speed of the network and the disk subsystem by benchmarking with the proper tool (i.e.iperf, iometer or analogues)?

On the network yes, like I said, I installed ATTO Disk Benchmark inside a single VM and could see inside of Performance Monitor on the datastore network speeds in excess of 20 gigabit.  On the disk side, no not really, only that when clonging two VMs the datastore is accomplishing twice the work in nearly the same amount of time.

5 Have you ever seen full utilization of your dual port NIC?

Again, like it tried to describe, I installed ATTO Disk Benchmark inside a single VM and could see inside of Performance Monitor on the datastore network speeds in excess of 20 gigabit.

Does your storage array support VAAI (perhaps via a plugin)?

It says unsupported.  I am not using any type of storage i/o control either.

So let me try to ask this another way.  Is there a setting in vSphere that limits cloning speed?  Perhaps so a single cloning process does not saturate a host, network, or datastore?

[AnatolyVilchins]

Well, it looks like the problem is related to the ESX side.

To be honest I haven`t heard about some limitations that persist by default.

I`ve found an article that relates to the vmware converter (yeah, I know that its` different product, but maybe the core of the cloning process is similar), and a lot of people reported the performance increase after disabling the SSL encryption, maybe it will help you too.

Also maybe this article will be useful for you as well:

VMware KB: Diagnosing slow deployment of templates or clones from vCenter Server

I`ll appreciate if you could give us an update if you`ll have some - this questions is pretty interesting for me personally.

[ramkrishna1]

Hi

Welcome to the communities.

Did you monitor storage performance \ utilization ?

Take care!

[AnatolyVilchins]

ramkrishna1 wrote:

Hi

Welcome to the communities.
Did you monitor storage performance \ utilization ?

I don`t really see big sense in it right now - David confirmed that storage do can give 20Gbps speed according to the benchmarking results. I`d rather questioned the utilization of resources on the ESXi side.

[PeterChanning]

Hi David, I know its been a  while since your original post but was wondering if you ever determined what was limiting the speeds of your clones? I'm currently attempting to clone machines, using vSphere 6, from one 3PAR to another over 10 Gbps network and am getting less than 5% utilization of the bandwidth. When I look at the performance charts, I am definitely seeing a coloration between reads on the source size and the bandwidth utilizations but am not sure where the limiting factors are.

Any input you could provide would be appreciated!

Thanks.