Deploying from Template / Storage vMotion Slow (iSCSI)

[mocal]

Hey guys, I've been scratching my head on this one for a while. I've tried everything I could find to try and resolve this. I'm wondering if anyone has any ideas I may have missed...

My issue is that when I deploy a vm from template, or storage vmotion a vm from one datastore to another - it literally takes forever. But, if I convert template to virtual machine, and then pull that virtual machine's image out of the virtual infrastructure using VCB, it takes 5 minutes (about 30Gb).

I have used all the best practices that I'm aware of that VMware recommends for iSCSI setups (regarding seperate networks (1GBps) for vmotion and iSCSI).

I do use the software iscsi initiator built into the ESX host.

I have two ESX hosts sharing two datastores, both are on the same target.

Both datastores are about 300GB and 500Gb (RAID 5) in size with 8 VMs in each. None of the VMs are very I/O intensive.

Even if I deploy a template onto the same datastore that the template resides on, I get the same (lack of) performance.

I've tried a number of templates, also rebuilding templates and trying again.

I'm using an AiO1200r (HPs All in One iSCSI San). When I view the performance on the network card (while deploying a template) on the AiO it barely breaks 5% utilization. When I'm doing backups the utilization is up around 20%.

[purduecjs]

Can you provide details on what your iscsi network looks like? specifically vlan, ip addresses, netmasks, default gateways? Is your VCB proxy on the same network and can you provide the same details for that machine, too? what version of esx are you using?

Cameron J. Smith

System Administrator, Purdue University

[mocal]

For sure.

I'm using ESX3.5 U4 with vCenter.

I dedicated a 1GB switch for the iSCSI network. No routing involved. 10.10.200.x/24 dedicated network.

Since we are a "daily" shop, I run my backups over night on the same network/switch. So yes, my vcb proxy server is on the same network.

My backup server is a dl380 g5 dedicated physical box that handles all our backups.

When I'm deploying a vm from template, it seems that some of my running VMs slow down quite a bit as well, but when I look at what kind of load is on the esx hosts, it doesnt appear to be anything out of the norm.

Any help or ideas would be appreciated. We are currently talking about purchasing a fibre san. This issue has some weight on our decision.

[purduecjs]

I assume that you're not seeing any frame drops on your switches or network interfaces on the devices?

From the 20,000ft view, it seems that your write performance might be the problem. Can you verify write performance on the SAN? It seems that your read speeds are great (~100MB/sec) given the ability to pull backups quickly.

If you're happy with ~100MB/sec I wouldn't invest in FC just yet. I would verify, maybe even independent of ESX, that the performance of your AiO is what you expect. At that point, you can layer ESX and test VM performance. If all that goes well, I can't see why svmotion and template deployment would be a problem.

Cameron J. Smith

System Administrator, Purdue University

[mocal]

I did check out the switches to see if I was losing any packets and I'm not.

I will definately look into the disk performance on my AiO next.

Sorry for the 20,000ft view, but like I said, I appreciate your help.

[purduecjs]

Your info is great - I just wanted to start giving you some ideas about things to look at and needed to change directions away from network and toward the storage.

My money at this point would be on the write speed given that the network looks okay and read speeds are okay.

Cameron J. Smith

System Administrator, Purdue University

[crescendas]

How big is your VM Template?

It is very fast when you convert a Template to VM and back, because it just amend the small VM configuration file to indicate that this is a VM or Template without having to duplicate existing VM datastore.

But if you create a new VM from a Template, it will have to duplicate existing VM datastore. If it's within the same datastore, it should be faster than if it has to copy across the iSCSI ethernet network, which takes about 1 hour for 20GB datastore on a 1GB ethernet network.

Cheers,

Boon Hong.

[mocal]

Cameron, I was just doing some tests within VI.

Deploying VMs from Templates using the following storage;

Shared to Local -> Fast (roughly 6mins)

Local to Shared -> Fast (roughly 6 mins)

Here's the big one...

Shared to Shared -> (drum roll please...) FAST! (roughly 10 mins)

Now, the only thing that has changed since I was having slow Template deployments is I enabled VT on my two esx hosts BIOS settings (HP DL380G5 Servers). Changing this setting was only supposed to allow me to run 64bit VMs... but somehow this has sped up my template deployments.

Currently investigating this. Any ideas? lol

So I guess this is a non-issue... I do very much appreciate your help. I did manage to learn a bit more about RAID levels and why I should be using a RAID10 configuration over the RAID5 config I'm currently using.

[mocal]

Thank you for your information too. I will mark your response as helpful.