Complete Infra Maintenance - Power Off and Power On

[ganapa2000]

Hi,

We have a complete Infra maintenance scheduled this weekend. We need to shutdown ~2000 VMs. I have below scripts to Shutdown and PowerOn. Does the Runasync works on Power Off and Power On correctly ? Let me know, if any changes needed or good to execute 🙂

Shutdown VMs

$VMList = Get-VMHost "MyESXHost" | Get-VM
#$VMList = Get-Folder MyTest | Get-VM
$report = @()
foreach($vm in $VMList){
if($vm.PowerState -eq 'PoweredOn'){
if($vm.Guest.State -eq "Running"){
Shutdown-VMGuest -VM $vm -Confirm:$false -RunAsync
} else {
Stop-VM -VM $vm -confirm:$false -RunAsync
}
Write-Host "Shutting Down $vm at $(get-date)"
Start-Sleep 10
$report += New-Object PSObject -Property @{
'VM_Name' = $vm
'Power_Off_Time' = Get-Date -Format 'MM/dd/yyyy HH:mm:ss'
}
}
}
$report | ft -auto

Power On VMs

$VMList = Get-VMHost "MyESXHost" | Get-VM
#$VMList = Get-Folder MyVMs | Get-VM
$report = @()
foreach($vm in $VMList){
if($vm.PowerState -eq 'PoweredOff'){
if($vm.Guest.State -eq "NotRunning"){
Start-VM -VM $vm -Confirm:$false
}
else{
Write-Host "Unable to Power On $vmName..."
}
Write-Host "Powering On $vm at $(get-date)"
Start-Sleep 10
$report += New-Object PSObject -Property @{
'VM_Name' = $vm
'Power_On_Time' = Get-Date -Format 'MM/dd/yyyy HH:mm:ss'
}
}
}
$report | ft -auto

[LucD]

Afaik the Shutdown-VMGuest cmdlet does not have a RunAsync switch.

And even without the RunAsync switch, you will have to check the PowerState of the VM in a loop.
The Shutdown-VMGuest cmdlet sends a shutdown command to the Guest OS and then returns.
Meaning the VM is not shutdown after the cmdlet returns

[ganapa2000]

LucD,

you mean like this ?

$maxTime = 10
Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |
ForEach-Object -Process {
$triedStop = $false
$triedGuest = $false
$now = Get-Date
while ($_.ExtensionData.Runtime.PowerState -ne [Vmware.Vim.VirtualMachinePowerState]::poweredOff -and -not $triedStop) {
if ($_.ExtensionData.Guest.ToolsRunningStatus -eq "GuestToolsRunning" -and -not $triedGuest) {
Shutdown-VMGuest -VM $_ -Confirm:$false
$triedGuest = $true
} else {
if (($triedGuest -and (New-TimeSpan -Start $now -End (Get-Date)).TotalSeconds -gt $maxTime) -or -not $triedStop) {
Stop-VM -VM $_ -Confirm:$false -RunAsync
$triedStop = $true
}
}
Start-Sleep 5
$_.ExtensionData.UpdateViewData()
}
}
Write-Host "all vms shutdown completed"

[LucD]

Without actually testing that code, yes, that looks about right.

I would still advise to do a small test (a few VMs) with that code before the actual intervention.

One remark, this will shutdown all the VMs (that have VMware Tools running) in sequence, with no parallelism.
One the other hand, introducing parallelism will require a more c omplex script

[ganapa2000]

LucD,

Currently, I dont have lab to test. Hence wanted to ensure, what I got works without any issues during the day of maintenance.

So If I want to shutdown VMs based on Particular Host or based on Folder, Does the below looks good ?

Get-VMHost "MyESXHost" | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |

OR

Get-Folder MyTest | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |

$maxTime = 10
Get-VMHost "MyESXHost" | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |
ForEach-Object -Process {
$triedStop = $false
$triedGuest = $false
$now = Get-Date
while ($_.ExtensionData.Runtime.PowerState -ne [Vmware.Vim.VirtualMachinePowerState]::poweredOff -and -not $triedStop) {
if ($_.ExtensionData.Guest.ToolsRunningStatus -eq "GuestToolsRunning" -and -not $triedGuest) {
Shutdown-VMGuest -VM $_ -Confirm:$false
$triedGuest = $true
} else {
if (($triedGuest -and (New-TimeSpan -Start $now -End (Get-Date)).TotalSeconds -gt $maxTime) -or -not $triedStop) {
Stop-VM -VM $_ -Confirm:$false -RunAsync
$triedStop = $true
}
}
Start-Sleep 5
$_.ExtensionData.UpdateViewData()
}
}
Write-Host "all vms shutdown completed"

[LucD]

Yes, that will limit the VMs to those specific locations.

[ganapa2000]

Perfect.

One last thing, how can I limit the number of VMs to be executed in a batch of 5 or 10 VMs at a VM.

$maxTime = 10
Get-VMHost "MyESXHost" | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |
ForEach-Object -Process {
$triedStop = $false
$triedGuest = $false
$now = Get-Date
while ($_.ExtensionData.Runtime.PowerState -ne [Vmware.Vim.VirtualMachinePowerState]::poweredOff -and -not $triedStop) {
if ($_.ExtensionData.Guest.ToolsRunningStatus -eq "GuestToolsRunning" -and -not $triedGuest) {
Shutdown-VMGuest -VM $_ -Confirm:$false
$triedGuest = $true
} else {
if (($triedGuest -and (New-TimeSpan -Start $now -End (Get-Date)).TotalSeconds -gt $maxTime) -or -not $triedStop) {
Stop-VM -VM $_ -Confirm:$false -RunAsync
$triedStop = $true
}
}
Start-Sleep 5
$_.ExtensionData.UpdateViewData()
}
}
Write-Host "all vms shutdown completed"

[LucD · Accepted solution]

You could try something like this

$maxTime = 10
$maxRun = 5
$vms = [System.Collections.ArrayList]@()

Get-VMHost "MyESXHost" | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } |
    ForEach-Object -Process {
        $vms.Add($_.Name)

        $triedStop = $false
        $triedGuest = $false
        $now = Get-Date
        while ($_.ExtensionData.Runtime.PowerState -ne [Vmware.Vim.VirtualMachinePowerState]::poweredOff -and -not $triedStop) {
            if ($_.ExtensionData.Guest.ToolsRunningStatus -eq "GuestToolsRunning" -and -not $triedGuest) {
                Shutdown-VMGuest -VM $_ -Confirm:$false
                $triedGuest = $true
            } else {
                if (($triedGuest -and (New-TimeSpan -Start $now -End (Get-Date)).TotalSeconds -gt $maxTime) -or -not $triedStop) {
                    Stop-VM -VM $_ -Confirm:$false -RunAsync
                    $triedStop = $true
                }
            }
            Start-Sleep 5
            $_.ExtensionData.UpdateViewData()
        }
        while ($vms.Count -ge $maxRun) {
            Get-VM -Name $vms | where { $_.PowerState -eq 'PoweredOff' } | ForEach-Object {
                $vms.Remove($_.Name)
            }
            Start-Sleep 5
        }
    }
Write-Host "all vms shutdown completed"

[ganapa2000]

That worked perfectly. Thank you very much.

But It would be nice, if I can get the details of the VM count when powering Off like,

Powering Off VM 1 out of VM 50

Powering Off VM 2 out of VM 50

Powering Off VM 2 out of VM 50

[LucD]

You can add a counter

$maxTime = 10
$maxRun = 5
$vms = [System.Collections.ArrayList]@()
$counter = 0
$myVM = Get-VMHost "MyESXHost" | Get-VM | Where-Object { $_.powerstate -eq 'PoweredOn' -or $_.PowerState -eq 'suspended' } 

$myVM | ForEach-Object -Process {
        $vms.Add($_.Name)
        Write-Host "Powering off VM $($counter) of $($myVM.Count)"
        $counter++
        
        $triedStop = $false
        $triedGuest = $false
        $now = Get-Date
        while ($_.ExtensionData.Runtime.PowerState -ne [Vmware.Vim.VirtualMachinePowerState]::poweredOff -and -not $triedStop) {
            if ($_.ExtensionData.Guest.ToolsRunningStatus -eq "GuestToolsRunning" -and -not $triedGuest) {
                Shutdown-VMGuest -VM $_ -Confirm:$false
                $triedGuest = $true
            } else {
                if (($triedGuest -and (New-TimeSpan -Start $now -End (Get-Date)).TotalSeconds -gt $maxTime) -or -not $triedStop) {
                    Stop-VM -VM $_ -Confirm:$false -RunAsync
                    $triedStop = $true
                }
            }
            Start-Sleep 5
            $_.ExtensionData.UpdateViewData()
        }
        while ($vms.Count -ge $maxRun) {
            Get-VM -Name $vms | where { $_.PowerState -eq 'PoweredOff' } | ForEach-Object {
                $vms.Remove($_.Name)
            }
            Start-Sleep 5
        }
    }
Write-Host "all vms shutdown completed"