Creating new VM with async fails with "A task was canceled"

[wfaulk_bw]

I am deploying a new VM from a content library OVA and it can take a while, so I am creating it with New-VM -RunAsync and then polling with Get-Task until it shows "Success" and then collecting the new VM with Wait-Task. However, sometimes the Wait-Task fails, returning $null and printing the error "A task was canceled", despite the vCenter task completing without issue.

My code is effectively:

$newvmtask = New-VM -RunAsync ...
do {
    Start-Sleep 15
    $taskstatus = Get-Task -Id $newvmtask.Uid
} while ($taskstatus.State -NE "Success")
$newvm = Wait-Task -Task $newvmtask

I've also tried it with the argument to Wait-Task as $taskstatus instead, but it produces the same problem.

Again, the problem is intermittent. It seems to coincide with the task completion taking over 5 minutes, which makes it feel like some sort of timeout. In fact, when I run New-VM without -RunAsync and it takes more than 5 minutes, I get the same error exactly 5 minutes after the cmdlet was submitted. But you'd think that using -RunAsync and polling would avoid that problem.

I'm having trouble consistently reproducing the problem, but in automated tasks it happens around half the time.

I have tried adjusting the client timeout with Set-PowerCLIConfiguration -Scope Session -WebOperationTimeoutSeconds 600 to extend the timeout to 10 minutes, but it doesn't seem to help.

This is running PowerCLI 13.0.0 under PowerShell 7.3.2 on Linux: CentOS 7.9.

[LucD]

Does $newvmtask.GetType() return a name like ClientSideTaskImpl?

[wfaulk_bw]

In the data I posted above, I just noticed that the timestamps are weird.

The task there says it ran from 5:10:39 to 5:15:39, which is exactly 5m. On vSphere, the matching task says it ran from 5:10:40 to 5:15:53. So the PowerCLI task claims that it finished earlier than vCenter says it finished, by 14 seconds.

So there's a 5 minute timeout that causes the task to report "Success"? You'd think a timeout there would at least report a failure.

But if the results from Get-Task are wrong, Wait-Task should still just block until the task is completed. So it feels like there's a timeout that's shared between Get-Task and Wait-Task.

I'm seeing some old bug reports about a shared HTTP handler under PowerShell in Linux. I wonder if that could be relevant.

[LucD]

Some Tasks started through RunAsync are converted to so-called Client Side Tasks (vs Server Side Tasks).

These Client Side Server tasks will show a different type of Task.

Also, the Get-Task cmdlet clearly states "The cmdlet retrieves information about the current or recent tasks"
That could be another reason why the Tasks shows as canceled, it is simply not there anymore (in the RecentTask property in the TaskManager object)
That was one of the reasons I wrote my Get-TaskPlus function in my Task Data Mining – An improved Get-Task post.

You might want to try that function, but be warned that it has issues with these Client Side Tasks.

[wfaulk_bw]

I ran my test again and ran Get-Task without arguments after getting back "Success" on the specific task.

The relevant tasks are definitely still in the list.

Description                                              StartTime            FinishTime             State
-----------                                              ---------            ----------             -----
                                                         3/14/2023 6:49:02 PM 3/14/2023 6:53:40 PM Success
Deploy OVF template                                      3/14/2023 6:48:50 PM                      Running
Deploy OVF package from Content Library to Resource Pool 3/14/2023 6:48:44 PM                      Running
Delete virtual machine                                   3/14/2023 6:44:05 PM 3/14/2023 6:44:05 PM Success
Fetch Content of a Library Item                          3/14/2023 6:48:39 PM 3/14/2023 6:48:41 PM Success
Parse OVF package in Content Library                     3/14/2023 6:48:38 PM 3/14/2023 6:48:44 PM Success
Fetch Content of a Library Item                          3/14/2023 6:48:44 PM 3/14/2023 6:48:46 PM Success
Fetch Content of a Library Item                          3/14/2023 6:49:00 PM 3/14/2023 6:49:01 PM Success

In the full Task Console in the vCenter web UI, there are only four tasks that were a result of this New-VM, and they're all shown here: the "Parse OVF package…", "Deploy OVF package…", "Deploy OVF template", and "Transfer File(s)" tasks, the last of which is the one that has no description in this list. (Its "Name" is "NfcCopy_Task".)

So I don't think that it's a case of the tasks falling off the "recent" list.

Can you think of a reason that this would fail other than that the cmdlet is broken? I really want to open a VMware support case about this, but they're giving me a hard time about needing to purchase SDK support, but this feels very much like "Cmdlets are not working as documented".

[LucD]

No clue why Get-Task would fail.
I always use my Get-TaskPlus function.

GSS is wrong, you don't need an SDK Support contract.
See also Solved: Re: Copy-VMGuestFile fails with error 500 when Gue... - VMware Technology Network VMTN

[wfaulk_bw]

I don't even understand how I would use your Get-TaskPlus. I have a specific task that was returned by New-VM -RunAsync. There doesn't seem to be a way to tell Get-TaskPlus about that specific task. (Ignoring, as you point out, the fact that it seems to be some virtual local task instead of any actual task on the vCenter server.) If I have to resort to searching for the task, I might as well just search for the VM.

Thanks for your help, though. I at least feel like I'm not being dense and just totally using it wrong.

[Macleud]

Hi.

I'm not sure this will help. But in my scripts I use something like this:

$New_Vm_Task = New-VM -ContentLibraryItem $VmTemplate -Name $VMName -ResourcePool $ResourcePool -Location $Location -StorageFormat $DiskFormat -Datastore $Datastore -RunAsync
Wait-Event -Timeout 20
$task = Get-Task -Status Running | where name -eq 'Deploy OVF template' | where {$_.ExtensionData.Info.EntityName -eq "$VMName"} 
if ($task) {
    $ts = Wait-Task -Task $task
}
elseif (!$task) {
    Wait-Event -Timeout 18
    $task = Get-Task -Status Running | where name -eq 'Deploy OVF template' | where {$_.ExtensionData.Info.EntityName -eq "$VMName"}
    $ts = Wait-Task -Task $task 
    if (!$task) {
        Wait-Task -Task $New_Vm_Task 
    }
}