VSAN stretched cluster failover script

[Cipo800]

Hi guys, just implemented Vxrail/Vsan six nodes stretched between two sites.

Our UPS doesn't support Vsan, only shutting down the VM's and hosts in case of power failure, alternative can launch a script.

Better then nothing, but in case of a power failure in one site with UPS software configured the result is half infrastructure down.

Without UPS software, HA reboot the VM's in the second site (uncontrolled) and Hardware power down abnormaly.

The idea is to make two scripts to launch specifically in SIte A and B:

1) Disable HA.

2) Setting DRS in manual mode.

3) Move VM from host esx4 to host esx1, then from esx5 to esx2 and finally to esx6 to esx3.(and a second script vice-versa)

4) Put hosts esx4 esx5 and esx6 in maintenance mode with "ensure accessibility".

5) Shutting down esx4,esx5 and esx6.

6) Enable DRS.

The same script will be usable also to plan a controlled DR test.

I'm not a "black belt" script man:)

Thanks for any help and greetings from Italy!

[LucD]

There aren't any "special" tricks involved as far as I can see.

1) and 2) can be done with the Set-Cluster cmdlet, respectively the HAEnabled and the DrsAutomationLevel parameters.
3) is done with the Move-VM cmdlet and the Destination parameter.

4) Is with the Set-VMHost and the State parameter.

5) The Stop-VMHost cmdlet

6) See 2) above.

[Cipo800 · Accepted solution]

@LucDthank you for the tips, after further research I've create the script and it's working, two questions if you can help to improve the result:

1) Is it possible to filter the "vCLS" VM's to they don't move?

2) I received this warning during the script execution: "WARNING: The 'state' property of VMHost type is deprecated. USe the 'ConnectionState' property instead.

Thanks!

param(
    [string]$Server = "vcenter",
    [string]$Username = "vcenteruser",
    [string]$Password = "vcenterpwd",
    [string]$clusterName = "Clustername"
    )
      
    $conn = Connect-VIServer $Server -Username $Username -Password $Password -Force

    set-cluster $clusterName -HAEnabled $false -Confirm:$false

    set-cluster $clusterName -drsautomationlevel manual -confirm:$false

    Get-VM -Location esx4 | Move-VM -Destination (Get-Vmhost esx3)
    Get-VM -Location esx5 | Move-VM -Destination (Get-Vmhost esx2)
    Get-VM -Location esx6 | Move-VM -Destination (Get-Vmhost esx1)
      
    Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    Set-VMHost esx5 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    Set-VMHost esx6 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false

    Stop-VMHost esx4 -Confirm:$false
    Stop-VMHost esx5 -Confirm:$false
    Stop-VMHost esx6 -Confirm:$false

    set-cluster $clusterName -drsautomationlevel fullyautomated -confirm:$false

    set-cluster $clusterName -HAEnabled $true -Confirm:$false


    Disconnect-VIServer $conn -Force -Confirm:$false

[LucD · Accepted solution]

You can exclude the vCLS VMs with a Where-clause

    Get-VM -Location esx4 | Where{$_.Name -notmatch '^vCLS'} | Move-VM -Destination (Get-Vmhost esx3)
    Get-VM -Location esx5 | Where{$_.Name -notmatch '^vCLS'} | Move-VM -Destination (Get-Vmhost esx2)
    Get-VM -Location esx6 | Where{$_.Name -notmatch '^vCLS'} | Move-VM -Destination (Get-Vmhost esx1)

That is just a warning, until that parameter is actually dropped in favour of ConnectionState, you can keep using State.

[Cipo800]

@LucDthank you, working great!

The last question, is it possible to write a log execution file?

Have a good day

[LucD]

You can add one or more lines containing the Out-File cmdlet.
It allows you send text to an external file.

[Cipo800]

Hi @LucD ,

during the tests one of three node don't enter in maintenance node before shutting down.

Actually I've implemented some "sleeps" and two cycle of the maintenance mode command.

Is it possible to check the maintenance mode state after send the command and if maintenance mode isn't reach send the command more times?

Thank you in adavnce and happy Easter!

param(
    $Server = "vcenter",
    $Username = "user",
    $Password = "pwd",
    $clusterName = "Clustername",
    $VMexclusionName = "^vCLS",
    $LogPath = "C:\log.log"
    )

    Start-Transcript -Path $LogPath
    $conn = Connect-VIServer $Server -Username $Username -Password $Password -Force

    set-cluster $clusterName -HAEnabled $false -Confirm:$false
    set-cluster $clusterName -drsautomationlevel manual -confirm:$false

    Get-VM -Location esx4 | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx3)
    Get-VM -Location esx5t | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx2)
    Get-VM -Location esx6 | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx1t)

    Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5
    Set-VMHost esx5 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5
    Set-VMHost esx6 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5
    Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5
    Set-VMHost esx5 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5
    Set-VMHost esx6 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
    sleep 5

    Stop-VMHost esx4 -Confirm:$false
    Stop-VMHost esx5 -Confirm:$false
    Stop-VMHost esx6 -Confirm:$false

    set-cluster $clusterName -drsautomationlevel fullyautomated -confirm:$false
    set-cluster $clusterName -HAEnabled $true -Confirm:$false

    Disconnect-VIServer $conn -Force -Confirm:$false

    Stop-Transcript

[LucD]

You can check the State by doing a Get-VMHost.
I don't think that sending the command multiple times will help.
To make sure your command for Maintenance Mode is accepted by the ESXi node, you can also check the vents for an EnteringMaintenanceModeEvent
That way you know the command was received/accepted by the ESXi node.

[Cipo800]

@LucDthank you for the reply, I ask you the last favour, please, can you write the necessary commands?

Thank you

[LucD]

You can replace the sleep with a While-block.

Something like this

Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
while ((Get-VMHost esx4).State -ne 'Maintenance'){
    sleep 5
}

When one of the nodes fails to go into maintenance mode, you might end up in an endless loop.
You could use a timeout for that.

Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
while ((Get-VMHost esx4).State -ne 'Maintenance' -and $interval -le $timeOut){
    sleep 5
    $interval = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
}
if($interval -gt $timeOut){
    Write-Host "Timeout"
    exit
}

[Cipo800]

Thank you again @LucD I've just replace the set-maintenace and sleep rows only with your second "block" of commands, is it correct?

I've try and seems to working good, below the entire script:

param(
    $Server = "vcenter",
    $Username = "user",
    $Password = "pwd",
    $clusterName = "Clustername",
    $VMexclusionName = "^vCLS",
    $LogPath = "C:\log.log"
    )

    Start-Transcript -Path $LogPath
    $conn = Connect-VIServer $Server -Username $Username -Password $Password -Force

    set-cluster $clusterName -HAEnabled $false -Confirm:$false
    set-cluster $clusterName -drsautomationlevel manual -confirm:$false

    Get-VM -Location esx4 | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx3)
    Get-VM -Location esx5t | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx2)
    Get-VM -Location esx6 | Where{$_.Name -notmatch $VMexclusionName} | Move-VM -Destination (Get-Vmhost esx1t)

Set-VMHost esx4 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
while ((Get-VMHost esx4).State -ne 'Maintenance' -and $interval -le $timeOut){
    sleep 5
    $interval = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
}
if($interval -gt $timeOut){
    Write-Host "Timeout"
    exit
}
Set-VMHost esx5 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
while ((Get-VMHost esx5).State -ne 'Maintenance' -and $interval -le $timeOut){
    sleep 5
    $interval = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
}
if($interval -gt $timeOut){
    Write-Host "Timeout"
    exit
}
Set-VMHost esx6 -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
while ((Get-VMHost esx6).State -ne 'Maintenance' -and $interval -le $timeOut){
    sleep 5
    $interval = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
}
if($interval -gt $timeOut){
    Write-Host "Timeout"
    exit
}
    Stop-VMHost esx4 -Confirm:$false
    Stop-VMHost esx5 -Confirm:$false
    Stop-VMHost esx6 -Confirm:$false

    set-cluster $clusterName -drsautomationlevel fullyautomated -confirm:$false
    set-cluster $clusterName -HAEnabled $true -Confirm:$false

    Disconnect-VIServer $conn -Force -Confirm:$false

    Stop-Transcript

[LucD]

Yes, that is correct.

You could store all ESXi names in an array and then loop through that array.
That way you avoid having that part x-times in your script

[Cipo800]

Great!! Can I send you a beer!?:)

I've no idea how to collect the esxi array names, if you want to give another help write the optimization, but is just "cosmetic", right?

[LucD]

You could do something like this

param(
  $Server = "vcenter",
  $Username = "user",
  $Password = "pwd",
  $clusterName = "Clustername",
  $VMexclusionName = "^vCLS",
  $LogPath = "C:\log.log"
)

Start-Transcript -Path $LogPath
$conn = Connect-VIServer $Server -Username $Username -Password $Password -Force

Set-Cluster $clusterName -HAEnabled $false -Confirm:$false
Set-Cluster $clusterName -DrsAutomationLevel manual -Confirm:$false

Get-VM -Location esx4 | where { $_.Name -notmatch $VMexclusionName } | Move-VM -Destination (Get-VMHost esx3)
Get-VM -Location esx5 | where { $_.Name -notmatch $VMexclusionName } | Move-VM -Destination (Get-VMHost esx2)
Get-VM -Location esx6 | where { $_.Name -notmatch $VMexclusionName } | Move-VM -Destination (Get-VMHost esx1t)

$esxNames = 'esx4', 'esx5', 'esx6'

foreach ($esx in $esxName) {
  Set-VMHost $esx -State Maintenance -VsanDataMigrationMode EnsureAccessibility -Confirm:$false
  while ((Get-VMHost $esx).State -ne 'Maintenance' -and $interval -le $timeOut) {
    sleep 5
    $interval = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
  }
  if ($interval -gt $timeOut) {
    Write-Host "Timeout"
    exit
  }

  Stop-VMHost $esx -Confirm:$false
}

Set-Cluster $clusterName -DrsAutomationLevel fullyautomated -Confirm:$false
Set-Cluster $clusterName -HAEnabled $true -Confirm:$false

Disconnect-VIServer $conn -Force -Confirm:$false

Stop-Transcript