VMware Cloud Community
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Reboot ESXi host from a Txt file one after the other on a loop for patching

Hi Experts

Please help me with the below issue

Hi have a requirement where I need to reboot ESXi host from a txt or CSV file

1. Need to put ESXi host into maintenance

2. If the ESXi host is unable to go into maintenance in 15 minutes then cancel maintenance mode and go to the next host

3. Once the host is in Maintenance mode reboot  ESXi host

4. Wait for ESXi host is back and EXIT out of Maintenance mode

5 Go to the next one

The issue

1. If an ESXi dosent go into Maintenance mode  the script exits, but I need to cancel the Maintenance task and go to the next host

2. The Script work ok until the hosts reboots once the reboot is done, i can see the ESXi is up and running but the script will take long time to exit the host out of Maintenance mode.

This is taking very long time and sometimes even the ESXi host is online after reboot the script will break because it thinks the hosts is offline

connect-viserver VIservername  -User Administrator@vsphere.local -Password xxxxx

$ListofVMHosts = get-content C:\temp\hosts.txt

Write-host (Get-date) " Current Hosts Configuration " -ForegroundColor Magenta

##Show current ESXi status

Get-vmhost -Name $ListofVMHosts |Select-Object Name,connectionstate,

@{N='ESXi version';E={"$($_.Version) $($_.Build)"}},

 

@{N='ESXi Hardware';E={"$($_.ExtensionData.Hardware.SystemInfo.Vendor) $($_.ExtensionData.Hardware.SystemInfo.Model)"}},

@{N='ESXi CPU Type';E={$_.ProcessorType}} |ft -AutoSize

##Reboot ESXi hosts

$Endloop=$false

foreach ($vmHostname in $ListofVMHosts) {

Write-Host "Setting $vmHostname to maintenance mode" -ForegroundColor Magenta

get-VMHost $vmHostname | set-VMHost -state maintenance -confirm:$false -runasync |out-null

sleep 10

$count = 0

while ($true) {

if ( (get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

Write-Host (get-date) ": $vmHostname in maintenance mode" -foregroundcolor Green

break;

}

else {

Write-Host (get-date) ": waiting for $vmHostname to go into maintenance..." -foregroundcolor Yellow

sleep 120

$count++

}

if ($count -gt 15) {

Write-Host (get-date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red

$Endloop=$true

break;

}

}

#read-host "pause..."

if (!$Endloop) {

Write-Host (get-date) ":Rebooting $vmHostname " -foregroundcolor Yellow

#this is a failsafe in case you did something stupid to the codes and reboot is executed

if ( (get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

get-VMHost $vmHostname | restart-VMHost -confirm:$false -force   | out-null

#ensure that its goes into NotResponding state first, because it takes awhile

while ((get-VMHost $vmHostname).ConnectionState -ne "NotResponding") {

sleep 5

}

$count = 0

while ($true) {

if ((get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

Write-Host (get-date) ": $vmHostname is up and in maintenance mode" -foregroundcolor Green

break ;

}

else {

Write-Host (get-date) ": waiting for $vmHostname to be online..." -foregroundcolor Yellow

sleep 300

$count++

}

if ($count -gt 6) {

Write-Host (get-date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red

$Endloop=$true

break;

}

}

}

else {

Write-Host (get-date) ": $vmHostname is not in maintenance, cannot reboot!" -foregroundcolor Red

$Endloop=$true

break;

}

}

#read-host "pause..."

if (!$Endloop) {

Write-Host "Setting $vmHostname back online" -foregroundcolor Yellow

get-VMHost $vmHostname | set-VMHost -state connected -confirm:$false | out-null

sleep 10

If ((get-VMHost $vmHostname).ConnectionState -eq "Connected") {

Write-Host (get-date) ": $vmHostname is up and connected" -foregroundcolor Green

}

else {

Write-Host (get-date) ": Host did not get back online...quiting!" -foregroundcolor Red

$Endloop=$true

}

}

if ($Endloop) {break;}

}

Tags (2)
Reply
0 Kudos
1 Solution

Accepted Solutions
LucD
Leadership
Leadership
Jump to solution

You have break commands at 2 locations in an If-Then-Else construct which only have the ForEach loop above them.

Remember "... a statement you can use to immediately exit foreach, for, while, do, switch, or trap statements."

When you 'break' there, the ForEach loop is ended.

See the updated (with comments) script

#########################################################################################################################################

Clear-Host

$selectedCLuster = Get-Cluster | Select-Object Name, DRSEnabled, DrsAutomationLevel | Sort-Object name | Out-GridView -OutputMode Single -Title "Select a Single cluster for patching "

## Check if the Cluster is DRS Enabled and if the cluster is fully Automated

if (Get-Cluster -name $selectedCLuster.Name | Where-Object { $_.DrsEnabled -eq 'true' -and $_.DrsAutomationLevel -eq 'FullyAutomated' } ) {

    Clear-Host

    Write-Host " Your Cluster " $selectedCLuster.name "is Ready for Patching"

    Start-Sleep 5

} Else {

    Clear-Host

    Write-Host " Cluster" $selectedCLuster.Name -ForegroundColor Yellow   "'DRS IS NOT ENABLED or NOT FULLY AUTOMATED'. Unable to Patch the cluster"

    Write-Host " Check with Cluster owner and Enable DRS & Set cluster Automation 'Fully Automated' "

    break

}


## Select the ESXi host for Patching

$selectVMhost = Get-Cluster -Name $selectedCLuster.Name | Get-VMHost | Sort-Object name | Out-GridView -OutputMode Multiple -Title "You can select single or multiple ESXi host for patching "


## Save the Current ESXi configuration to a Variable


$ESXiconfigbeforePatching = Get-VMHost -Name $selectVMhost | Select-Object Name, connectionstate,

@{N = 'ESXi version'; E = { "$($_.Version) $($_.Build)" } },

@{N = 'ESXi Hardware'; E = { "$($_.ExtensionData.Hardware.SystemInfo.Vendor) $($_.ExtensionData.Hardware.SystemInfo.Model)" } },

@{N = 'ESXi CPU Type'; E = { $_.ProcessorType } } | Out-File -FilePath c:\temp\out.txt

Write-Host " You selected the following hosts for patching "


## Display the current ESXI configuration which is taken before patching


$ESXiconfigbeforePatching

$title = 'It is Important to confirm that you really want to patch this cluster'

$question = 'Are you sure you want to proceed with ESXI reboots during patching ?'

$choices = New-Object Collections.ObjectModel.Collection[Management.Automation.Host.ChoiceDescription]

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&Yes'))

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&No'))

$decision = $Host.UI.PromptForChoice($title, $question, $choices, 1)


if ($decision -eq 0) {

    Write-Host 'Preparing for patching activity'

} else {

    Write-Host 'Cancelling Patching job'

    Write-Host 'Disconnecting from Virtual Centre'

    Disconnect-VIServer -Server * -Confirm:$false

    Break

}


Write-Host "We care collecting ESXi datastore information "


###########################################################################################################################


## Mount and Unmount CD


$mountedCDdrives = Get-VMHost $selectVMhost | Get-VM | Where-Object { $_ | Get-CDDrive | Where-Object { $_.ConnectionState.Connected -eq "True" } }

IF ($mountedCDdrives.Count -gt "0") {

    Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";

    $mountedCDdrives.Name;

    $unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";

    IF ($unmountDrives -eq "Y") { Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) { Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False } }ELSE { Write-Host "Skipping ISO unmounting..." -foreground "Yellow" }

} ELSE { Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow" }

##############################################################################################################################


foreach ($vmHostname in $selectVMhost) {

    $Endloop = $false

    Write-Host "Setting $vmHostname to maintenance mode" -ForegroundColor Magenta

    Get-VMHost $vmHostname | Set-VMHost -state maintenance -confirm:$false -runasync | Out-Null

    Start-Sleep 10

    $count = 0


    while ($true) {

        if ( (Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

            Write-Host (Get-Date) ": $vmHostname in maintenance mode" -foregroundcolor Green

            break     # Exit while($true) block

        }

        else {

            if($count -le 15){

                Write-Host (Get-Date) ": waiting for $vmHostname to go into maintenance..." -foregroundcolor Yellow

                Start-Sleep 20

                $count++

            }

            else {

                Write-Host (Get-Date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red

                Write-Host (Get-Date) ": Cancelling Maintenance task of Host $vmHostname" -foregroundcolor Red

                $task = Get-Task -Status Queued, Running | Where-Object { $_.Name -eq 'EnterMaintenanceMode_Task' }

                if ($task) {

                    Write-Host 'True' -ForegroundColor Yellow

                    Stop-Task -Task $task -Confirm:$false

                }

                Else {

                    Write-Host 'False' -ForegroundColor Cyan

                }

                $Endloop = $true

                break     # Exit while($true) block

            }

        }

    }

    if (!$Endloop) {

        Write-Host (Get-Date) ":Rebooting $vmHostname..........!!!" -foregroundcolor Red -BackgroundColor White

        if ( (Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

            Get-VMHost $vmHostname | Restart-VMHost -confirm:$false -force | Out-Null

            while ((Get-VMHost $vmHostname).ConnectionState -ne "NotResponding") {

                Start-Sleep 5

            }

            $count = 0

            while ($true) {

                if ((Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

                    Write-Host (Get-Date) ": $vmHostname is up and in maintenance mode" -foregroundcolor Green

                    break  # End while ($true) { loop

                }

                else {

                    Write-Host (Get-Date) ": waiting for $vmHostname to be online..." -foregroundcolor Yellow

                    Start-Sleep 50

                    $count++

                }


                if ($count -gt 6) {

                    Write-Host (Get-Date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red

                    $Endloop = $true

                    break  # End while ($true) { loop

                }

            }

        }

        else {

            Write-Host (Get-Date) ": $vmHostname is not in maintenance, cannot reboot!" -foregroundcolor Red

            # If you want to continue with the next ESXi node, the following break is not needed

            # break    # End foreach ($vmHostname in $selectVMhost) { loop

        }

    }


    if (!$Endloop) {

        Write-Host "Setting $vmHostname back online" -foregroundcolor Yellow

        Get-VMHost $vmHostname | Set-VMHost -state connected -confirm:$false | Out-Null

        Start-Sleep 10

        If ((Get-VMHost $vmHostname).ConnectionState -eq "Connected") {

            Write-Host (Get-Date) ": $vmHostname is up and connected" -foregroundcolor Green

        }

        else {

            Write-Host (Get-Date) ": Host did not get back online...quiting!" -foregroundcolor Red

            # If you want to continue with the next ESXi node, the following break is not needed

            # break    # Exit foreach ($vmHostname in $selectVMhost)

        }

    }

}

#######################################################################################################


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

View solution in original post

Reply
0 Kudos
21 Replies
LucD
Leadership
Leadership
Jump to solution

1. I'm not sure where the break would cause the script to exit.

A break continues after the while block when it is called.

2. To investigate why a reboot takes long, you will need to to look in the logs of the ESXi node.

Could this delay be caused by the HA Agent reestablishing the HA setup?

For further problem solving, I would use the Verbose switch and use a transcript file.
That might shed some light on what is happening.

You can cancel a background Task with the CancelTask method.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

Thanks for the reply

I think i didn't put the question correctly

My ESXi reboots are fine and I can ping the ESXi host also on the VC i can see the ESXi host is back on line

But the script takes long time to identify the ESXi host is back on line

I need to script to quickly identify the host is back on line and go to the next host

Its the script taking  a long time to check if the host is up and online

how can that be corrected please ?

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You didn't answer any of my questions


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

1. I'm not sure where the break would cause the script to exit.

A break continues after the while block when it is called.

Thought the Break would exit the loop

2. To investigate why a reboot takes long, you will need to to look in the logs of the ESXi node.

Could this delay be caused by the HA Agent reestablishing the HA setup?

No the ESXi host reboots in minutes which is very fast no issues with the Actual hardware , its the script dosent understand that the ESXi host is online and this is causing delays

Hope this makes sense.

All i want is is for the script to exit if a Host takes longer than 15 minuets to go into Maintenance mode

Also when the host is rebooted and back on line I want the script to exit out of maintenance mode and proceed with the next host

Please help

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

1. I know what break does.

I just asked which break (you have more than one) caused the script to exit.

2. If the ESXi node shows in the Web CLient that is connected again, a Get-VMHost should reflect that in the State property.
You are saying that the State content is not correct, until after x minutes?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

1. I know what break does.

I just asked which break (you have more than one) caused the script to exit.

Sorry you are powcli guru, i was trying to say, my intention was to break the script, not sure how to break inside a loop.

2. If the ESXi node shows in the Web CLient that is connected again, a Get-VMHost should reflect that in the State property.

You are saying that the State content is not correct, until after x minutes?

Here what I am saying is after the ESXi is back on line I see it in the WebClient as online, however the script dosent recognise its up and it can take upto 10 minutes for the script to understand the ESXi is up. Sometimes it times out and the script breaks

Here is a picture to show whats happening

As you can see the ESXi host is up I can ping also its connected on the VC, but if you look at the screen snip of the script its waiting for the ESXi host

Please help

Many thanks

RXJ

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

pastedImage_0.png

pastedImage_1.png

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Being able to ping an ESXi node only says that the network is up.
That doesn't say that is fully connected to the vCenter yet.

You would have to look at the events for that ESXi node in the vCenter.

What events and which timestamps do you see there?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

Ok let me explain it, It's not about the ping after the reboot the ESXi host is fully functional

I can manually set the host out of Maintenance mode and migrate VM's, access datastores and its functional.

Its the Script which is not detecting the ESXi host after the reboot

I tried this on few clusters and all doing the same

not sure why the script is waiting for the host to be online when its online and fully functional

Can you help me to modify that part please

pastedImage_0.png

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You are waiting 5 minutes (sleep 300) between each test if the host is in maintenance mode.

Which is confirmed by the timestamp on the messages.

I would suggest lowering that interval.

    while ($true) {

        if ((Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

            Write-Host (Get-Date) ": $vmHostname is up and in maintenance mode" -foregroundcolor Green

            break ;

        } else {

            Write-Host (Get-Date) ": waiting for $vmHostname to be online..." -foregroundcolor Yellow

            Start-Sleep 300    # 5 minutes !!!

            $count++

        }


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

Thank you so much, after lowering the time now the script identifies the host quickly

That helped

I have one more humble request

When a host is not going int Maintenance mode, i have added the task to be cancelled and thats working

However its going out of the loop and ends the script

I need something like this when a host cant get to maintenance mode after 5 minutes it needs to cancel the Maintenance mode task and move to the next ESXi host

Cancelling the task works and moving to the next host dosent work

Please see the text in bold

Help me please

#########################################################################################################################################

cls

$selectedCLuster = Get-Cluster  | Select Name,DRSEnabled,DrsAutomationLevel|sort name  | Out-GridView -OutputMode Single -Title "Select a Single cluster for patching "

## Check if the Cluster is DRS Enabled and if the cluster is fully Automated

if (Get-Cluster -name $selectedCLuster.Name |where {$_.DrsEnabled -eq 'true' -and $_.DrsAutomationLevel -eq 'FullyAutomated'} )

{cls

Write-Host " Your Cluster " $selectedCLuster.name "is Ready for Patching"

sleep 5

  }

Else

{

cls

Write-Host " Cluster" $selectedCLuster.Name -ForegroundColor Yellow   "'DRS IS NOT ENABLED or NOT FULLY AUTOMATED'. Unable to Patch the cluster" 

Write-Host " Check with Cluster owner and Enable DRS & Set cluster Automation 'Fully Automated' "

break

  }

## Select the ESXi host for Patching

$selectVMhost= Get-Cluster -Name $selectedCLuster.Name | Get-VMHost |sort name | Out-GridView -OutputMode Multiple -Title "You can select single or multiple ESXi host for patching "

## Save the Current ESXi configuration to a Variable

$ESXiconfigbeforePatching= Get-vmhost -Name $selectVMhost |Select-Object Name,connectionstate,

         @{N='ESXi version';E={"$($_.Version) $($_.Build)"}},

        

         @{N='ESXi Hardware';E={"$($_.ExtensionData.Hardware.SystemInfo.Vendor) $($_.ExtensionData.Hardware.SystemInfo.Model)"}},

         @{N='ESXi CPU Type';E={$_.ProcessorType}} |Out-File -FilePath c:\temp\out.txt

Write-host " You selected the following hosts for patching "

## Display the current ESXI configuration which is taken before patching

$ESXiconfigbeforePatching

$title    = 'It is Important to confirm that you really want to patch this cluster'

$question = 'Are you sure you want to proceed with ESXI reboots during patching ?'

$choices = New-Object Collections.ObjectModel.Collection[Management.Automation.Host.ChoiceDescription]

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&Yes'))

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&No'))

$decision = $Host.UI.PromptForChoice($title, $question, $choices, 1)

if ($decision -eq 0) {

    Write-Host 'Preparing for patching activity'

} else {

    Write-Host 'Cancelling Patching job'

    Write-Host 'Disconnecting from Virtual Centre'

    Disconnect-VIServer -Server * -Confirm:$false

    Break

}

Write-host "We care collecting ESXi datastore information "

###########################################################################################################################

## Mount and Unmount CD

$mountedCDdrives = Get-VMHost $selectVMhost | Get-VM | Where { $_ | Get-CdDrive | Where { $_.ConnectionState.Connected -eq "True" } }

IF ($mountedCDdrives.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";

$mountedCDdrives.Name;

$unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";

IF ($unmountDrives -eq "Y") {Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) {Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False}}ELSE{Write-Host "Skipping ISO unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow"}

##############################################################################################################################

$Endloop=$false

foreach ($vmHostname in $selectVMhost) {

Write-Host "Setting $vmHostname to maintenance mode" -ForegroundColor Magenta

get-VMHost $vmHostname | set-VMHost -state maintenance -confirm:$false -runasync |out-null

sleep 10

$count = 0

while ($true) {

if ( (get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

Write-Host (get-date) ": $vmHostname in maintenance mode" -foregroundcolor Green

break;

}

else {

Write-Host (get-date) ": waiting for $vmHostname to go into maintenance..." -foregroundcolor Yellow

## Sleeping for 30 Minutes initial value

#sleep 120

## Sleeping for 5 Minutes

Sleep 20

$count++

}

if ($count -gt 15) {

Write-Host (get-date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red

Write-Host (get-date) ": Cancelling Maintenance task of Host $vmHostname" -foregroundcolor Red

$task= get-task |Select-Object Name,id |where {$_.Name -eq 'EnterMaintenanceMode_Task'}

if ($cancelTask=Get-Task |where {$_.Name -eq 'EnterMaintenanceMode_Task' -or $_.id -eq $task  } )

{

Write-Host 'True' -ForegroundColor Yellow

$cancelTask|Stop-Task -Confirm:$false

}

Else

{

Write-Host 'False' -ForegroundColor Cyan

}

## Comment End loop

#$Endloop=$true

#break;

}

}

#read-host "pause..."

if (!$Endloop) {

Write-Host (get-date) ":Rebooting $vmHostname..........!!!" -foregroundcolor Red -BackgroundColor White

#this is a failsafe in case you did something stupid to the codes and reboot is executed

if ( (get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

get-VMHost $vmHostname | restart-VMHost -confirm:$false -force   | out-null

#ensure that its goes into NotResponding state first, because it takes awhile

while ((get-VMHost $vmHostname).ConnectionState -ne "NotResponding") {

sleep 5

}

$count = 0

while ($true) {

if ((get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

Write-Host (get-date) ": $vmHostname is up and in maintenance mode" -foregroundcolor Green

break ;

}

else {

Write-Host (get-date) ": waiting for $vmHostname to be online..." -foregroundcolor Yellow

## Adding sleep from 300 to 50 i.e from 30 minutes to 5 minutes

sleep 50

$count++

}

if ($count -gt 6) {

Write-Host (get-date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red

$Endloop=$true

break;

}

}

}

else {

Write-Host (get-date) ": $vmHostname is not in maintenance, cannot reboot!" -foregroundcolor Red

## Commenting this entry to keep it inside the loop

#$Endloop=$true

break;

}

}

#read-host "pause..."

if (!$Endloop) {

Write-Host "Setting $vmHostname back online" -foregroundcolor Yellow

get-VMHost $vmHostname | set-VMHost -state connected -confirm:$false | out-null

sleep 10

If ((get-VMHost $vmHostname).ConnectionState -eq "Connected") {

Write-Host (get-date) ": $vmHostname is up and connected" -foregroundcolor Green

}

else {

Write-Host (get-date) ": Host did not get back online...quiting!" -foregroundcolor Red

$Endloop=$true

}

}

if ($Endloop) {break;}

}

#######################################################################################################

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Try replacing that part with this.
Since you don't check the Status of the Tasks, you are also getting Tasks that already ended

Write-Host (Get-Date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red

Write-Host (Get-Date) ": Cancelling Maintenance task of Host $vmHostname" -foregroundcolor Red

$task = Get-Task -Status Queued, Running | Where-Object { $_.Name -eq 'EnterMaintenanceMode_Task' }

if ($task) {

    Write-Host 'True' -ForegroundColor Yellow

    Stop-Task -Task $task -Confirm:$false

} Else {

    Write-Host 'False' -ForegroundColor Cyan

}


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

I replaced the script with what you provided and still its going in the same loop waiting for the host to go into Maintenance mode

pastedImage_0.png

Please advise

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Looks like you left out a 'break'.
Try with this one

#########################################################################################################################################

Clear-Host

$selectedCLuster = Get-Cluster | Select-Object Name, DRSEnabled, DrsAutomationLevel | Sort-Object name | Out-GridView -OutputMode Single -Title "Select a Single cluster for patching "


## Check if the Cluster is DRS Enabled and if the cluster is fully Automated

if (Get-Cluster -name $selectedCLuster.Name | Where-Object { $_.DrsEnabled -eq 'true' -and $_.DrsAutomationLevel -eq 'FullyAutomated' } ) {

    Clear-Host

    Write-Host " Your Cluster " $selectedCLuster.name "is Ready for Patching"

    Start-Sleep 5

} Else {

    Clear-Host

    Write-Host " Cluster" $selectedCLuster.Name -ForegroundColor Yellow   "'DRS IS NOT ENABLED or NOT FULLY AUTOMATED'. Unable to Patch the cluster"

    Write-Host " Check with Cluster owner and Enable DRS & Set cluster Automation 'Fully Automated' "

    break

}


## Select the ESXi host for Patching

$selectVMhost = Get-Cluster -Name $selectedCLuster.Name | Get-VMHost | Sort-Object name | Out-GridView -OutputMode Multiple -Title "You can select single or multiple ESXi host for patching "


## Save the Current ESXi configuration to a Variable


$ESXiconfigbeforePatching = Get-VMHost -Name $selectVMhost | Select-Object Name, connectionstate,

@{N = 'ESXi version'; E = { "$($_.Version) $($_.Build)" } },

@{N = 'ESXi Hardware'; E = { "$($_.ExtensionData.Hardware.SystemInfo.Vendor) $($_.ExtensionData.Hardware.SystemInfo.Model)" } },

@{N = 'ESXi CPU Type'; E = { $_.ProcessorType } } | Out-File -FilePath c:\temp\out.txt

Write-Host " You selected the following hosts for patching "

## Display the current ESXI configuration which is taken before patching

$ESXiconfigbeforePatching

$title = 'It is Important to confirm that you really want to patch this cluster'

$question = 'Are you sure you want to proceed with ESXI reboots during patching ?'

$choices = New-Object Collections.ObjectModel.Collection[Management.Automation.Host.ChoiceDescription]

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&Yes'))

$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&No'))

$decision = $Host.UI.PromptForChoice($title, $question, $choices, 1)


if ($decision -eq 0) {

    Write-Host 'Preparing for patching activity'

} else {

    Write-Host 'Cancelling Patching job'

    Write-Host 'Disconnecting from Virtual Centre'

    Disconnect-VIServer -Server * -Confirm:$false

    Break

}


Write-Host "We care collecting ESXi datastore information "

###########################################################################################################################


## Mount and Unmount CD


$mountedCDdrives = Get-VMHost $selectVMhost | Get-VM | Where-Object { $_ | Get-CDDrive | Where-Object { $_.ConnectionState.Connected -eq "True" } }

IF ($mountedCDdrives.Count -gt "0") {

    Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";

    $mountedCDdrives.Name;

    $unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";

    IF ($unmountDrives -eq "Y") { Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) { Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False } }ELSE { Write-Host "Skipping ISO unmounting..." -foreground "Yellow" }

} ELSE { Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow" }

##############################################################################################################################


$Endloop = $false


foreach ($vmHostname in $selectVMhost) {

    Write-Host "Setting $vmHostname to maintenance mode" -ForegroundColor Magenta

    Get-VMHost $vmHostname | Set-VMHost -state maintenance -confirm:$false -runasync | Out-Null

    Start-Sleep 10

    $count = 0


    while ($true) {

        if ( (Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

            Write-Host (Get-Date) ": $vmHostname in maintenance mode" -foregroundcolor Green

            break     # Exit while($true) block

        }

        else {

            if($count -le 15){

                Write-Host (Get-Date) ": waiting for $vmHostname to go into maintenance..." -foregroundcolor Yellow

                Start-Sleep 20

                $count++

            }

            else {

                Write-Host (Get-Date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red

                Write-Host (Get-Date) ": Cancelling Maintenance task of Host $vmHostname" -foregroundcolor Red

                $task = Get-Task -Status Queued, Running | Where-Object { $_.Name -eq 'EnterMaintenanceMode_Task' }

                if ($task) {

                    Write-Host 'True' -ForegroundColor Yellow

                    Stop-Task -Task $task -Confirm:$false

                }

                Else {

                    Write-Host 'False' -ForegroundColor Cyan

                }

                $Endloop = $true

                break     # Exit while($true) block

            }

        }

    }


    if (!$Endloop) {

        Write-Host (Get-Date) ":Rebooting $vmHostname..........!!!" -foregroundcolor Red -BackgroundColor White

        if ( (Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

            Get-VMHost $vmHostname | Restart-VMHost -confirm:$false -force | Out-Null

            while ((Get-VMHost $vmHostname).ConnectionState -ne "NotResponding") {

                Start-Sleep 5

            }


            $count = 0

            while ($true) {

                if ((Get-VMHost $vmHostname).ConnectionState -eq "Maintenance") {

                    Write-Host (Get-Date) ": $vmHostname is up and in maintenance mode" -foregroundcolor Green

                    break ;

                }

                else {

                    Write-Host (Get-Date) ": waiting for $vmHostname to be online..." -foregroundcolor Yellow

                    Start-Sleep 50

                    $count++

                }


                if ($count -gt 6) {

                    Write-Host (Get-Date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red

                    $Endloop = $true

                    break;

                }

            }

        }

        else {

            Write-Host (Get-Date) ": $vmHostname is not in maintenance, cannot reboot!" -foregroundcolor Red

            break;

        }

    }


    if (!$Endloop) {

        Write-Host "Setting $vmHostname back online" -foregroundcolor Yellow

        Get-VMHost $vmHostname | Set-VMHost -state connected -confirm:$false | Out-Null

        Start-Sleep 10

        If ((Get-VMHost $vmHostname).ConnectionState -eq "Connected") {

            Write-Host (Get-Date) ": $vmHostname is up and connected" -foregroundcolor Green

        }

        else {

            Write-Host (Get-Date) ": Host did not get back online...quiting!" -foregroundcolor Red

            break    # Exit foreach ($vmHostname in $selectVMhost)

        }

    }

}

#######################################################################################################


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

Firstly thank you so much for formatting and correcting the script

This time it  is cancelling the task and going to the second server but once its put the other host into Maintenance mode the script breaks

Please advise

pastedImage_0.png

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

I suspect that this is caused by the location of the line

$Endloop = $false

Try placing that inside the ForEach loop

foreach ($vmHostname in $selectVMhost) {

    $Endloop = $false


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

As you suggested tried replacing

$Endloop = $false

foreach ($vmHostname in $selectVMhost) {

    $Endloop = $true

Its still the same, it' sets the next host into Maintenance and exit

pastedImage_3.png

Please help

Many thanks

RXJ

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Why did you change $false to $true?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
rxjoseph
Enthusiast
Enthusiast
Jump to solution

Hi LucD

I tried both True and false, goes out of the loop not sure why this is happening

$Endloop = $false

foreach ($vmHostname in $selectVMhost) {

    $Endloop = $false

    Write-Host "Setting $vmHostname to maintenance mode" -ForegroundColor Magenta

    Get-VMHost $vmHostname | Set-VMHost -state maintenance -confirm:$false -runasync | Out-Null

    pastedImage_9.png

Many thanks for all your help

RXJ

Reply
0 Kudos