Async tasks, hash-tables and arrays - better way to do this!?

[andybgrant]

I am working with a colleague on a script that will deploy a very large number of VM's from a CSV file.  We started with a purely synchronous process that was very slow using a single foreach loop to deploy a VM, configure it, apply a customization spec and start it all based on data from the CSV.

The obvious bottleneck in the process is waiting for the clone operation to complete prior to performing the remaining tasks.  Since the synchronous New-VM line was in a foreach loop, it was painfully slow.

We have since modified it to asynchronously deploy the VM's using the foreach loop based the data from the CSV file dumped into an array.  Once the foreach loop has kicked off all the clone operations, it finishes and then a task monitoring "while" loop kicks off to then perform the remainder of the tasks.  The while loop is based on LucD's code found here: http://www.lucd.info/2010/02/21/about-async-tasks-the-get-task-cmdlet-and-a-hash-table/ (Thanks Luc!).

Now Luc's code used a hash-table so it had to be modified to use the CSV array as best I could.  The CSV array contained the information to perform the configuration and customization of the VM's.  I really couldn't figure out a way to do this very cleanly so I am looking for suggestions on making this a little better.

Right now I am using a simple counting mechanism to walk through the array to determine if the cloning operation has completed or not.  The problem with this is that if there are 100 VM's being cloned, and I am walking through the array one line at a time so it could take a while between successful completion of the clone operation and the monitoring loop actually picking up on it.  It would be nice to have a way to identify which line in the array has the required configuration and customization data without walking through it in a loop, rather referencing an element inside the array (TaskID) using some other method (is there one? - still figuring this powershell stuff out).

Here is the modified version of Luc's code:

# $csv is the array in question with the required data.
# Used to count which line in the array
$csvline = 0

# Count all the running tasks to feed into the while loop
$tasks = $csv | %{$_.TaskID} | ?{$_ -match "Task"}
$runningTasks = $tasks | measure | %{$_.count}

while($runningTasks -gt 0){

# Completion Meter
$percomplete = (1 / $runningTasks)*100
Write-Progress -activity 'Waiting for cloning operation to complete' -status 'Please wait...' -percentComplete ($percomplete)

# Here is where it starts to get messy, there has to be a better way than using $csv[$csvline] and walking though
if ((get-task | select id,state | ?{$_.id -eq $csv[$csvline].TaskId}).state -eq "Success"){
Set-VM $csv[$csvline].name -NumCpu $csv[$csvline].vcpu -MemoryMB $csv[$csvline].MemoryMB -Description $csv[$csvline].Notes -Confirm:$false
Get-vm -name $csv[$csvline].name | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName $csv[$csvline].Network -StartConnected:$true -Confirm:$false
Get-VM $csv[$csvline].name | Start-VM -RunAsync -Confirm:$false
$csv[$csvline].TaskId = ""
$runningTasks--
}
elseif ((get-task | select id,state | ?{$_.id -eq $csv[$csvline].TaskId}).state -eq "Error"){
$csv[$csvline].TaskId = ""
$runningTasks--
}


# Increment $csvline
$csvline++

# Wash rinse repeat over and over (not very pretty)
# Reset $CSV array line counter when greater than count of lines (minus 1 because the array/count starts at zero).
if ($csvline -gt ($csv.count - 1)){
$csvline = 0
}

# Slow down the runningTasks loop since we are waiting for cloning operations to complete.
# IMPACT: If you deploy 100 VM's it could take up to 200 seconds AFTER a VM is finished cloning before being noticed by the while loop
Start-Sleep -Seconds 2
}

It would be nice to create a Get-Task monitoring loop that doesn't have to walk through the array.  But pull any "Sucess"ful tasknames and figure out which line in the $csv array it belongs to (using the TaskId element).

Is there a way to do this other than nesting a foreach ($line in $csv) inside the Get-Task monitoring loop and compare TaskId values?  (That might be quicker regardless - haven't tested yet).

Hopefully I explained things clearly enough.

Thanks for your time.

Andy

[LucD]

Hi Andy, I don't really understand why you can't use a hash table to track the Async tasks.

The $csv array is filled with an Import-Csv cmdlet if I understand it correctly.

The data in that array is used to provide the parameter values for the New-VM cmdlet and the Set-VM cmdlet later on, I assume. Correct ?

There is an easier way to select records from a hash table, use the GetEnumerator method combined with a Where-clause.

If you do that on the hash table with the Tasks, you easily filter out the ones with a State of "Success".

Since you have to use the $csv data later on with the Set-VM cmdlets, I would make that also a has table, with the VM name as the key.

That makes it easier to retrieve the parameter values for the Set-VM cmdlet in my opinion.

[andybgrant]

Hi Luc

Thanks for the suggestions I will definitely play around with them.

To be honest, I didn't know if a hash table would be able to fullfill the role of the data I needed in the $csv array.  You are correct in the $csv contains the vm name,description,template,datastore,oscustomization,vcpus,mb,network,subnet,gw,taskid.... and so forth.

What I couldn't figure out is if I used your original code using a hash table to track the tasks, how I would find which line in the $csv array I needed to perform the set-vm tasks.

Your suggestion to use a hash table instead of an array for $csv, how would I do this with the data mentioned above (name, desc, temp...) and multiple VM's?  I'm not really clear on the limits or usage of a hash table right now.

Thanks

Andy

[LucD · Accepted solution]

Andy, have a look at the thread called error with Get-OsCutomizationNicMapping, it does something similar to what you want to do I assume.

Inside the Import-Csv loop the new VM is created in Async and the Id of the task together with some specific values from the CSV are stored in the Value part of the hash table.

Later on, these values are used to configure the new VM further.