Can’t Get Multiple SVMotions Instances using PowerCLI Hash Tables Working

[mabnic]

I’m using the hash table example that LucD provides in http://www.lucd.info/2010/02/21/about-async-tasks-the-get-task-cmdlet-and-a-hash-table/ with the Move-VM commandlet, but I receive the following errors:

Index operation failed; the array index evaluated to null.

At C:\vmware\test_powershell.ps1:8 char:11

+     $taskTab[ <<<< (Move-VM -VM (Get-VM $Name) -Destination (Get-VMHost $esxN

ame) -RunAsync).Id] = $Name

    + CategoryInfo          : InvalidOperation: (System.Collections.Hashtable:

   Hashtable) [], RuntimeException

    + FullyQualifiedErrorId : NullArrayIndex

Index operation failed; the array index evaluated to null.

At C:\vmware\test_powershell.ps1:8 char:11

+     $taskTab[ <<<< (Move-VM -VM (Get-VM $Name) -Destination (Get-VMHost $esxN

ame) -RunAsync).Id] = $Name

    + CategoryInfo          : InvalidOperation: (System.Collections.Hashtable:

   Hashtable) [], RuntimeException

    + FullyQualifiedErrorId : NullArrayIndex

The SVMotion executes, but the script fails after that.  My script follows:

$esxName = " esx1.test.local"

$Datastore = "volume1"

$moveVmList = "testVM1","testVM2"

$taskTab = @{}

# Storage vMotion all the VMs specified in $moveVmList

foreach($Name in $moveVmList){

       $taskTab[(Move-VM -VM (Get-VM $Name) -Datastore (Get-Datastore $Datastore) -RunAsync).Id] = $Name

}

# vMotion each VM after Storage vMotion is completed

$runningTasks = $taskTab.Count

while($runningTasks -gt 0){

       Get-Task | % {

             if($taskTab.ContainsKey($_.Id) -and $_.State -eq "Success"){

                    Get-VM $taskTab[$_.Id] | Move-VM -Destination (Get-VMHost $esxName)

                    $taskTab.Remove($_.Id)

                    $runningTasks--

             }

             elseif($taskTab.ContainsKey($_.Id) -and $_.State -eq "Error"){

                    $taskTab.Remove($_.Id)

                    $runningTasks--

             }

       }

       Start-Sleep -Seconds 15

}

I’ve tried different variations to the script using other commandlets, but only appears to fail with the Move-VM  commandlet.

Any help would be appreciated.

Thanks!

[LucD]

Which PowerCLI version are you using ?

And against which vSphere version are you running the script ?

[mabnic]

I'm using PowerCLI 5.0.1 on w2k8r2, ESXi 4.1.0 build 502767, and vSphere Server 4.1.0 build 491557.

[LucD]

In PowerCLI 5.x, the Task object that is returned by cmdlets that are run with the RunAsync parameter has changed.

Where the returned object used to be a TaskImpl object, it has become a ClientSideTaskImpl object.

That in itself wouldn't have been a problem, except for the fact that the Id property is $null.

So we need to use a trick to get the Id property, since that is the Key we use in the hash table.

Can you try this version of the script ?

$esxName = "esx1.test.local" 
$Datastore = "volume1" 
$moveVmList = "testVM1","testVM2"
 $taskTab = @{}

# Storage vMotion all the VMs specified in $moveVmList
foreach($Name in $moveVmList){
        $cstask = Move-VM -VM (Get-VM $Name) -Datastore (Get-Datastore $Datastore) -RunAsync
        $task = Get-Task | where {$_.Name -eq $cstask.Name -and $_.StartTime -eq $cstask.StartTime}
        $taskTab[$task.Id] = $Name
}

# vMotion each VM after Storage vMotion is completed
$runningTasks = $taskTab.Count 
while($runningTasks -gt 0){
       Get-Task | % {
             if($taskTab.ContainsKey($_.Id) -and $_.State -eq "Success"){
                    Get-VM $taskTab[$_.Id] | Move-VM -Destination (Get-VMHost $esxName)
                    $taskTab.Remove($_.Id)
                    $runningTasks--
             }
             elseif($taskTab.ContainsKey($_.Id) -and $_.State -eq "Error"){
                    $taskTab.Remove($_.Id)
                    $runningTasks--
             }
       }
       Start-Sleep -Seconds 15
}

[mabnic]

LucD:

Thanks for the assistance.

I’m still getting the same error, shown below:

Index operation failed; the array index evaluated to null.

At C:\vmware\Migrate_VMs.ps1:10 char:18

+         $taskTab[ <<<< $task.Id] = $Name

    + CategoryInfo          : InvalidOperation: (System.Collections.Hashtable:

   Hashtable) [], RuntimeException

    + FullyQualifiedErrorId : NullArrayIndex

Index operation failed; the array index evaluated to null.

At C:\vmware\Migrate_VMs.ps1:10 char:18

+         $taskTab[ <<<< $task.Id] = $Name

    + CategoryInfo          : InvalidOperation: (System.Collections.Hashtable:

   Hashtable) [], RuntimeException

    + FullyQualifiedErrorId : NullArrayIndex

Here is the data that was in $task variable for each VM storage vmotioned:

Client                  :

CmdletTaskInfo          :

ExtensionData           :

Id                      :

IsCancelable            : False

Result                  :

Name                    : RelocateVM_Task

Description             : Relocate virtual machine

State                   : Running

PercentComplete         : 0

StartTime               : 5/3/2012 11:53:33 AM

FinishTime              :

ObjectId                :

Uid                     : /Local=/ClientSideTask=1955e4a3-4090-4e38-b940-434872

                          7f4c50/

NonTerminatingErrorList : {}

TerminatingError        :

Client                  :

CmdletTaskInfo          :

ExtensionData           :

Id                      :

IsCancelable            : False

Result                  :

Name                    : RelocateVM_Task

Description             : Relocate virtual machine

State                   : Running

PercentComplete         : 0

StartTime               : 5/3/2012 11:53:35 AM

FinishTime              :

ObjectId                :

Uid                     : /Local=/ClientSideTask=0638a956-162e-4013-847d-3db557

                          f9a644/

NonTerminatingErrorList : {}

TerminatingError        :

$task shows that the “Id” field is empty, which would cause the error.  I did try to use the “Uid” field instead, since it appears to be a unique identifier, but the script seems to hang on the second part of the script where it vmotions to another host.

Would the “Uid” field be the right direction?  If so, what would cause it to hang on the second part?

Thanks in advance.

-mb

[LucD]

That is strange, the $Task variable should hold a TaskImpl object, not a ClientSideTaskImpl object.

That would mean that Get-Task didn't return a matching TaskImpl object after the svMotion.

But then $Task should have been empty.

Are you sure you used that last code I included ?

[mitsumaui]

Sorry to drag up this slightly older post,

Duncan - I was also trying to use your script, both original and modified and noticed the same that the -runasync returns a ClientSideTaskImpl.

I could not match this back to a Get-Task, the same as the op because there is a difference in time by 1 second on the Client Side Task and the Task in Get-Task.

However, The ClientSideTaskImpl contains the state of the task and is dynamically updated, so you can just query against this and wait for the tasks to finish.

Here's a snippet of the code I used to accomplish this. The $Serverlist variable is a CSV containing the host name, and the destination datastore.

$Tasks = $Serverlist | % {
    write-host "`n`n`n`n`n`n`n`n`n*****************************************************************************"
    Write-host "Start time" (Get-Date)
    write-host "Moving" $_.Name "to Datastore:" $_.Datastore
    #Migrate VM to new Storage
    Get-VM $_.Name | Move-VM -Datastore (Get-Datastore -Name $_.Datastore) -RunAsync
}
Write-Host "Waiting for Moves to finish..."
do {
    $TaskComplete = ($Tasks | ? { $_.State -ne "Running" }).Count
    Start-Sleep -Seconds 15
} until ($TaskComplete -eq $Tasks.Count)

Hope this helps everyone trying to run multiple tasks and waiting for completion before moving on!

[admin]

Move-VM cmdlet gave me a lot of trouble, it basically returns an object with no relationship with the VM and its task object. I experimented your method, but I realized it is very hard to match the start time of the task. Instead, I am using the method below:

By comparing the difference between tasks before and after, we can retrieve the newly created task object. It does not work 100%, but it is made the most robust as I can. If there are other people initiating svMotion at the same time, the method will consider the task with start time closer to our Move-VM task. There is a reason why I go down this path, the tasks in concern also include clone and svMotion with maximum number of operations set by user e.g. max 4 at a time. I want a generic method that is able to handle every case. So the preference is to retrieve all task objects rather than using Wait-Task, which is synchronous.

function GetNewTask($preTasks, $postTasks, $taskStartTime) {
    $collectedTasks = @()
    for($i = 0; $i -lt $postTasks.Length; $i++) {
        $thisTask = $preTasks | Where {$_.Id -eq $postTasks[$i].Id}
        if($thisTask -eq $null) {
            $collectedTasks += $postTasks[$i]
        }
    }
    $task = $null
    if($collectedTasks.Count -gt 1) {
        $task = $collectedTasks | Sort [system.math]::abs($_.StartTime - $taskStartTime) | Select -First 1
    } elseif($collectedTasks.Count -eq 1) {
        $task = $collectedTasks[0]
    } else {
        Write-Host "New Task not found through differential comparison."
    }
    return ,$task
}

$preTasks = Get-Task | Where {$_.Name -eq "RelocateVM_Task"}
$cstask = Move-VM -VM $vm.Name -Datastore $svmotionDatastore -Destination $vm.VMHost -DiskStorageFormat Thin -RunAsync
Start-Sleep -Seconds 5
$postTasks = Get-Task | Where {$_.Name -eq "RelocateVM_Task"}
$task = GetNewTask $preTasks $postTasks $cstask.StartTime

Then, I am also using your published code to get historic tasks to wait for all tasks to complete, or wait for the concurrent tasks to fall below the maximum allowed set by user. And there is another layer of vCO involved, it is better to check tasks status every N seconds, the efficiency is higher by eliminating the need to pull task status constantly.

Thank you for your hints.

[mferreira1]

I successfully figured out that there is a property called ObjectId on Get-Task that matches the VM Id.

You may keep track of the tasks by matching the VM Id and the Task ObjectId. In my case I was exporting a VM to OVA template.

PS C:\> $vm = Get-VM -Name "vm1"

PS C:\> Get-Task | Where { $_.ObjectId -eq $vm.Id }

Name                           State      % Complete Start Time   Finish Time

----                           -----      ---------- ----------   -----------

Export OVF template            Running             5 10:37:14

PS C:\>

I recorded the VM Id to the hashtable and then matched that with the task.

Function ExportOVA ($VAppName,$VMtoExport)

{

  $VMtoExp = Get-VM -Name $VMtoExport

  $null = Export-Vapp -Name $VAppName -VM $VMtoExp -Destination $OVAStaging -Format Ova -ErrorAction Stop -RunAsync

  $exportTab.Add(($VMtoExp).Id, $VAppName)

}

while($runningTasks -gt 0){

     Get-Task | % {

          If ($exportTab.ContainsKey($_.ObjectId) -And $_.State -eq "Success") {

               ...

          }

     }

}

[uswbnc53]

What if you wanted to process the storage vMotions from the array in batches of 2 or 3 at a time?

[LucD]

It would require a bit of math, but something like this should do the trick.

You would have to place the vMotion in the inner loop, and then wait till these are finished before continuing

$moveVMList = "vm1","vm2","vm3","vm4","vm5","vm6","vm7","vm8"

$total = $moveVMList.Count

$batchNumber = 3

0..([math]::Floor($total/$batchNumber)) | %{

    "Batch $_"

    $start = $_ * $batchNumber

    $start..([math]::Min(($start + $batchNumber - 1),($total - 1))) | %{

        "Moving $($moveVMList[$_])"

    }

}