How to "Schedule VM Compatibility Upgrade" as well as "Cancel Scheduled VM Upgrade"

[kghammond2009]

Hello,

I am working on a tool to allow our administrators to enable and disabled automatic vmhardware upgrades through an internal lifecycle management tool we have.

I have seen all the code that uses the VirtualMachineConfigSpec and that all seems to work, but when you enabled the onSoftPowerOff setting, it does something different that the Clicking "Schedule VM Compatibility Upgrade".  When you click on that button it changes the upgradepolicy back to "never".  So there must be a new method of scheduling vmware hardware upgrades.  Not sure if PowerCLI has access to the new method.

I am not sure if this might be because we enabled VMware Tools and VM Hardware in Lifecycle Manager aka updates?

Also interestingly, when I try to change the VirtualMachineConfigSpec back to never on my own, it never sets it.

Also of note, when you set the UpgradePolicy to onSoftPowerOff, the "Cancel Scheduled VM Upgrade" option is not enabled.

Can anyone point us in the right direction? I have been searching Google and asking ChatGPT and I can't find any code samples that don't use the VirtualMachineConfigSpec method.

The sample code I have been using and referencing is:

$vm = Get-VM -Name MyVM
$vm.ExtensionData.Config.ScheduledHardwareUpgradeInfo
$spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = "onSoftPowerOff"
$spec.ScheduledHardwareUpgradeInfo.VersionKey = "vmx-09"
$spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = "pending"
$vm.ExtensionData.ReconfigVM_Task($spec)

Thank You,

Kevin

[LucD]

Is the VM powered on?
Did you schedule the LCM task to do the HW upgrade at a later date/time?
If yes, you should see the Scheduled Task, it will be assigned to the VM and have a name like "MyVM - Scheduled Upgrade - VM powered on"
That task uses the com.vmware.vcIntegrity.RemediateTask method to do the HW upgrade.
Afaik these vcintegrity API are part of Update Manager and are not made public.

If the VM is powered off or suspended a similar Scheduled Task will be created, it just adds the power on.

When I used LCM to upgrade the HW version immediately, the task is executed immediately.
The HW version of the targetted VM is upgraded and the UpgradePolicy is indeed set to "never".

I'm not sure if the LCM API to create such VM HW upgrade tasks are made public.

[LucD]

And btw, if you use a VersionKey that is lower than the current HW version, nothing is changed in the ScheduledHardwareUpgradeInfo object.

Since "vmx-09" corresponds with compatibility with ESXi 5.1, I assume that was just an example of the code? 

[kghammond2009]

Yes that was just sample code, we are using vmx-19.  I also did not set the ScheduledHardwareUpgradeStatus, but when we set UpgradePolicy to onSoftPowerOff, it automatically set ScheduledHardwareUpgradeStatus to pending.

[kghammond2009]

Yes the test I was doing was with a powered on VM.  So from the vCenter LCM, I was scheduling the HW upgrade at a later time.

I didn't look at the Scheduled Tasks anywhere, not sure how to look at that in the gui  or via PowerCLI.

I think I am ok using the old method using the VirtualMachineConfigSpec, but I have tried changing UpgradePolicy to = "never" and it doesn't save it.  It just leaves it as "onSoftPowerOff" and ScheduledHardwareUpgradeStatus as "pending".

Any ideas why I cannot change UpgradePolicy back to "never" after setting it to "onSoftPowerOfff"?  Do I have to change ScheduledHardwareUpgradStatus at the same time?

Also, I got errors anytime I tired to set VersionKey to "" or $false or $null.

Thank You

[LucD]

In the GUI select the vCenter in the Inventory on the left, then select Configure, and then Scheduled Tasks.

Should the task already be completed, check in Monitor under Tasks

[kghammond2009]

For reference, when you schedule through vCenter GUI, it does not show up as a job in Scheduled Tasks.  Considering the task to upgrade VMware hardware is per virtual machine and on reboot, I would think it is still somehow tied to the virtual machine.  If it wasn't, then vCenter would have to talk to LCM on boot and ask it if there were any pending tasks before it powered it on.  I supposed that's possible, but I would be more likely that the upgrade task is somehow linked to the virtual machine and on boot, it checks the setting and upgrades its hardware if it needs to.

Under monitor tasks there is a Reconfigure virtual machine task associated with when you click Schedule VM Compatibility Upgrade via the GUI, so again, I think the task or job or setting, whatever you want to call it, it is somehow saved to the virtual machine.  Which would make you think you could modify that setting via PowerCLI.

Is there a way to look and see what objects have changed on the virtual machine before and after reconfiguring from the scheduled task?

[LucD]

Yes, see Get VM Reconfigure Details from PowerCLI - VMware Technology Network VMTN

[kghammond2009]

Getting there but your code for Auditing VM device changes doesn't seem to work anymore.  It's failing on the eCollector line.  Apparently the CreateCollectorForEvents function doesn't support 1 parameter anymore.

It is finding the correct task, but I don't think I am far enough into your code to determine what changed.

See attached screenshot

[LucD]

That code in my blogpost only looks at device changes, not changes in the HW upgrade settings.
Also, I suspect your error might be caused by a Task that doesn't have an EventChainId, which can happen if the Task failed.

This is an adapted version of the script that returns changes in the HW upgrade settings.
And it handles missing EventChainId values.

$hours = 96 # Number of hours back
$tasknumber = 999 # Windowsize for task collector
$eventnumber = 100 # Windowsize for event collector
$report = @()
$taskMgr = Get-View TaskManager
$eventMgr = Get-View eventManager
$tFilter = New-Object VMware.Vim.TaskFilterSpec
$tFilter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$tFilter.Time.beginTime = (Get-Date).AddHours(-$hours)
$tFilter.Time.timeType = "startedTime"
$tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter))
$dummy = $tCollector.RewindCollector
$tasks = $tCollector.ReadNextTasks($tasknumber)
while ($tasks) {
  $tasks | Where-Object { $_.Name -eq "ReconfigVM_Task" } | ForEach-Object {
    $task = $_
    if ($task.EventChainId) {
      $eFilter = New-Object VMware.Vim.EventFilterSpec
      $eFilter.eventChainId = $task.EventChainId
      $eCollector = Get-View ($eventMgr.CreateCollectorForEvents($eFilter))
      $events = $eCollector.ReadNextEvents($eventnumber)
      while ($events) {
        $events | ForEach-Object {
          $event = $_
          switch ($event.GetType().Name) {
            "VmReconfiguredEvent" {
              $report += New-Object PSObject -Property ([ordered]@{
                  VMname = $task.EntityName
                  Start = $task.StartTime
                  Finish = $task.CompleteTime
                  Result = $task.State
                  User = $task.Reason.UserName
                  ConfigChanges = $event.ConfigChanges.Modified
                  UpgradePolicy = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.UpgradePolicy
                  VersionKey = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.VersionKey
                  ScheduledHardwareUpgradeStatus = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus
                  Fault = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.Fault
                })
            }
            Default {}
          }
        }
        $events = $eCollector.ReadNextEvents($eventnumber)
      }
      $ecollection = $eCollector.ReadNextEvents($eventnumber)
      # By default 32 event collectors are allowed. Destroy this event collector.
      $eCollector.DestroyCollector()
    } else {
      Write-Host -Object "No EventChainId present for Task $($task.Key) with Status $($task.State)" -ForegroundColor Yellow
    }
  }
  $tasks = $tCollector.ReadNextTasks($tasknumber)
}
# By default 32 task collectors are allowed. Destroy this task collector.
$tCollector.DestroyCollector()
$report | Sort-Object -Property Start

[kghammond2009]

Thank you for the code, but it still is having the same error.  The task does have an eventChainID, but it's a negative number.  I have a feeling that is causing the error.

It is the right task though.  From debug, here is:  $task | fl

Key : task-5084215
Task : Task-task-5084215
Description :
Name : ReconfigVM_Task
DescriptionId : VirtualMachine.reconfigure
Entity : VirtualMachine-vm-1875321
EntityName : ADCONNECT11
Locked :
State : success
Cancelled : False
Cancelable : False
Error :
Result :
Progress :
Reason : VMware.Vim.TaskReasonUser
QueueTime : 3/10/2023 6:19:03 PM
StartTime : 3/10/2023 6:19:03 PM
CompleteTime : 3/10/2023 6:19:03 PM
EventChainId : -1942235932
ChangeTag :
ParentTaskKey :
RootTaskKey :
ActivationId :
LinkedView :

[LucD]

That seems to be a known issue, see Heads Up - Potential missing vCenter Server Events due to sequence ID overflow (williamlam.com)

[LucD]

This version skips negative EventChainId values, but it might be missing the tasks and events we are looking for.

$hours = 96 # Number of hours back
$tasknumber = 999 # Windowsize for task collector
$eventnumber = 100 # Windowsize for event collector
$report = @()
$taskMgr = Get-View TaskManager
$eventMgr = Get-View eventManager
$tFilter = New-Object VMware.Vim.TaskFilterSpec
$tFilter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$tFilter.Time.beginTime = (Get-Date).AddHours(-$hours)
$tFilter.Time.timeType = "startedTime"
$tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter))
$dummy = $tCollector.RewindCollector
$tasks = $tCollector.ReadNextTasks($tasknumber)
while ($tasks) {
  $tasks | Where-Object { $_.Name -eq "ReconfigVM_Task" } | ForEach-Object {
    $task = $_
    if ($task.EventChainId -and $task.EventChainId -gt 0) {
      $eFilter = New-Object VMware.Vim.EventFilterSpec
      $eFilter.eventChainId = $task.EventChainId
      $eCollector = Get-View ($eventMgr.CreateCollectorForEvents($eFilter))
      $events = $eCollector.ReadNextEvents($eventnumber)
      while ($events) {
        $events | ForEach-Object {
          $event = $_
          switch ($event.GetType().Name) {
            "VmReconfiguredEvent" {
              $report += New-Object PSObject -Property ([ordered]@{
                  VMname = $task.EntityName
                  Start = $task.StartTime
                  Finish = $task.CompleteTime
                  Result = $task.State
                  User = $task.Reason.UserName
                  ConfigChanges = $event.ConfigChanges.Modified
                  UpgradePolicy = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.UpgradePolicy
                  VersionKey = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.VersionKey
                  ScheduledHardwareUpgradeStatus = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus
                  Fault = $event.ConfigSpec.ScheduledHardwareUpgradeInfo.Fault
                })
            }
            Default {}
          }
        }
        $events = $eCollector.ReadNextEvents($eventnumber)
      }
      $ecollection = $eCollector.ReadNextEvents($eventnumber)
      # By default 32 event collectors are allowed. Destroy this event collector.
      $eCollector.DestroyCollector()
    } 
    elseif($task.EventChainId -and $task.EventChainId -lt 0){
        Write-Host -Object "Negative EventChainId present for Task $($task.Key) with Status $($task.State)" -ForegroundColor Yellow
    }
    else {
      Write-Host -Object "No EventChainId present for Task $($task.Key) with Status $($task.State)" -ForegroundColor Yellow
    }
  }
  $tasks = $tCollector.ReadNextTasks($tasknumber)
}
# By default 32 task collectors are allowed. Destroy this task collector.
$tCollector.DestroyCollector()
$report | Sort-Object -Property Start

[kghammond2009]

In case anyone else reads this thread.  It turns out I was not using the ScheduledHardwareUpgradeInfo VirtualMachineConfigSpec properly.

It seems you need to specify the ScheduledHardwareUpgradeStatus flag at the same time, in additions to UpgradePolicy and VersionKey, in order to get the vCenter GUI to recognize the scheduled hardware upgrade change.  I had been following other samples on the Internet that only specified the UpgradePolicy and the VersionKey.

To disable the scheduled upgrade you have to specify BOTH "never" and "none".  All the references I had found previously just said to specify "never".  I am not sure if this has changed over time and the second parameter is now needed as well or if the code samples where incomplete.

Thank You for all the troubleshooting assistance.

$myVM = "TESTVM"
$vm = Get-VM -Name $myVM

#Enable vmhardware upgrade on next reboot	
$spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = "always"
$spec.ScheduledHardwareUpgradeInfo.VersionKey = "vmx-19"
$spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = "pending"
$vm.ExtensionData.ReconfigVM_Task($spec)


#Enable vmhardware upgrade on next clean reboot
$spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = "onSoftPowerOff"
$spec.ScheduledHardwareUpgradeInfo.VersionKey = "vmx-19"
$spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = "pending"
$vm.ExtensionData.ReconfigVM_Task($spec)


#Disable vmhardware upgrade
$spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = "never"
$spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = "none"
$vm.ExtensionData.ReconfigVM_Task($spec)

[LucD]

I'm pretty sure that this is a Web GUI issue if the ScheduledHardwareUpgradeStatus doesn't say 'none' when the UpgradePolicy says 'never'.

To disable the planned upgrade it is sufficient to only set the UpgradePolicy to 'never', I just tested this again on several VMs.
The ScheduledHardwareUpgradeStatus is a property that receives its value from the system, not something that is set by the user.