To enable the "Schedule VM Compatibility Upgrade..." tick box for a virtual machine called VM1 you can use the next PowerCLI script:
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.scheduledHardwareUpgradeStatus = 'pending'
$spec.ScheduledHardwareUpgradeInfo.upgradePolicy = 'onSoftPowerOff'
$spec.ScheduledHardwareUpgradeInfo.versionKey = 'vmx-09'
Get-VM -Name VM1 | ForEach-Object {
$_.ExtensionData.ReconfigVM_Task($spec)
}
The script sets the compatibility to "ESX 5.1 and later" and enables the "Only upgrade after normal guest OS shutdown" tick box.