VMware Cloud Community
StephenMoll
Expert
Expert

Detecting a VM starting to perform a Guest Shutdown

What is the earliest point in the shutdown process a PowerCLI script could detect that a Guest has initiated a controlled shutdown?

I'm asking because I am looking at a system where tight IOPS limits are applied to all VMs. Don't ask, it is simply a requirement we have to meet. Some of these are VERY low, which seriously hampers the VM startup and shutdown processes.

Startup is a trivial thing to deal with:

When VMs are off, have the IOPS limits unset. When a VM starts, apply the IOPS limits, when VMTools reports the guest is running, with perhaps an additional time delay to allow a limit headroom. Or more simply, apply the limits a set time after VM power-on.

The tricky one is when a VM is shutting down. If the process is initiated from within the guest, I would like to be able to detect this has happened and remove the IOPS limits, so the VM can shutdown swiftly, and be in the correct state for the next power-on.

Any ideas out there?

18 Replies
LucD
Leadership
Leadership

You could look for an occurrence of the VmStoppingEvent.

This event is triggered when the shutdown starts, before it is actually shutdown.


You could look for example every minute of such an event occurred in the last minute.

$start = (Get-Date).AddMinutes(-1)

Get-VIEvent -Start $start |

where{$_ -is [VMware.Vim.VmStoppingEvent]}

As an alternative, you could create an Alarm based on that event.

And then trigger for example a Trap to a SNMP server, which could then take appropriate action via a script.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

StephenMoll
Expert
Expert

I shall get someone to take a look at that. Thanks LucD.

Reply
0 Kudos
StephenMoll
Expert
Expert

I've had a play with this, and it didn't work.

Examining the full output of Get-VIEvent in gridview I see tasks that indicate the "{VM Name} on {Host} in {Cluster} is powered off". The output when "where {$_ -is [VMware.Vim.VmStoppingEvent]}" is added is completely empty.

I am guessing that the events I see (as described above) are a bit late. What I am trying to do is detect when a VM is shutting down and remove any IOPS limits from it so it I can speed up the shutdown, and massively speed up the next startup. Start up is not a problem, the VMs are started by a script, and are started and then IOPS limits applied a minute or so later. When the script is used to shut them down, likewise the limits are removed and a guest shutdown initiated. The problem is what to do when the shutdown is commanded from within the guest.

The entries I am seeing don't appear to have any fields matching the value your snippet looks for. I see "Vm = VMware.Vim.VMEventArgument" and others related to Host, Compute Resource and Datacenter.

Reply
0 Kudos
LucD
Leadership
Leadership

Can you check what kind of events are raised for the VM that was stopped in the last X minutes (example goes for 15 mins).

$vmName = 'MyVM'

$vm = Get-VM -Name $vmName


Get-VIEvent -Entity $vm -MaxSamples ([int]::MaxValue) -Start (Get-Date).AddMinutes(-15) |

Group-Object -Property { $_.GetType().Name }


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

Startup produces:

Count Name                       Group

----- ----                       -----

    1 VmAcquiredTicketEvent      {VMware.Vim.VmAcquiredTicketEvent}

    1 VmPoweredOnEvent           {VMware.Vim.VmPoweredOnEvent}

    1 VmStartingEvent            {VMware.Vim.VmStaringEvent}

    1 TaskEvent                  {VMware.Vim.TaskEvent}

Shutdown using in Guest Windows Shutdown from Start Menu produces:

Count Name                       Group

----- ----                       -----

    4 AlarmStatusChangedEvent    {VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent}

    1 VmPoweredOffEvent          {VMware.Vim.VmPoweredOffEvent}

A restart from within Windows produces:

Count Name                       Group

----- ----                       -----

    2 AlarmStatusChangedEvent    {VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent}

I'm not seeing anything that looks convincingly early enough in the shutdown process to make a real difference. Some of the VMs are required to have stupidly low IOPS limits set, which as you can imagine really hampers VM power-on, but also affects guest-shutdown as well.

Reply
0 Kudos
LucD
Leadership
Leadership

Do these VMs have the VMware Tools installed and running?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

Yes they do. We rely on VMtools in order to harvest the VMs for performance test results data.

Reply
0 Kudos
LucD
Leadership
Leadership

Does this give sufficient time?

$vm = Get-VM MyVM

Get-Date

Shutdown-VMGuest -VM $vm -Confirm:$false

while($vm.ExtensionData.Guest.GuestOperationsReady){

  sleep 2

   $vm.ExtensionData.UpdateViewData('Guest.GuestOperationsReady')

}

Write-Host "VM is powering off"

Get-Date

$vm.ExtensionData.UpdateViewData('Runtime.PowerState')

while($vm.ExtensionData.Runtime.PowerState -ne 'poweredoff'){

  sleep 2

   $vm.ExtensionData.UpdateViewData('Runtime.PowerState')

}

Get-Date

If it doesn't (and that is what I think will turn out), then that is because the VMware Tools are one of the last processes to be stopped

An alternative which you could use, since you have the VMware Tools installed and running, is to use the ListProcessesInGuest method.
Depending on which guest OS you have running inside the VM, you could check if one or more processes are stopped.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

I'll have a play with that and see what it does. I suspect that getting an early enough signal that a VM is shutting down due to an internal action is going to be tricky. I think I will just suggest that when using the test VMs that all shutdown be done using the external command scripts.

Basically we have a VM template that has some performance measuring tools on board. We have a set of PowerCLI scripts that role out multiple VMs from this template that is used to simulate a final whole system across the cluster. We use this to simulate workloads for CPU, memory and storage IO testing across the whole hardware suite. A test is configured, started and stopped through these scripts and the results harvested from all the VMs as well.

With the CPU and memory testing there were no issues, but with storage IO it is different. The final design requires us to set IOPS limit based on the final design. All system resources are accounted for and no overprovisioning is permitted as a strict requirement. So where some VMs are specified with very low run-time IOPS requirement, the VMs take an age to boot up, so we really only want to apply the limits after the VM has booted up. We then would like to remove the limits before shutting the VMs down. This is easy when people stick to using the control scripts to do this, but that isn't always the case. So I'll probably just end up telling people that manual shutdown performance of a test VM may not be great.

Reply
0 Kudos
StephenMoll
Expert
Expert

Just ran:

18 April 2019 14:45:06

OSFullName   : Microsoft Windows 8 (64-bit)

:

GuestFamily  : windowsguest

VM is powering off

18 April 2019 14:45:06

18 April 2019 14:45:24

So time for this VM to power of was about 18 seconds. But of course in this test we knew when the shutdown started, because it was initiated by PowerCLI.

I wonder if there is anyway to use VMTools to place a small PowerShell script inside the test VM which will trigger a detectable event as soon as someone clicks shutdown or restart inside the guest?

Reply
0 Kudos
StephenMoll
Expert
Expert

What would be really useful I think would be the ability to define further parameters for VM limits and reservations, but especially for IOPS. These parameters would be to specify when the limits apply, for example:

IOPS limit : nnn IOPS

Activation time : Immediate | sss Seconds after power on | When "toolsOK"

Deactivation time : At start of guest shutdown | After poweroff | Never

Reply
0 Kudos
LucD
Leadership
Leadership

That is exactly what I suggested with the use of the ListProcessesInGuest method.

The problem might be what to look for.
It will depend on what kind of guest OS you have running inside the VM.

Another option is to run a script inside the VM (with Invoek-VMScript) and let that script fire for example send an event with the LogUserEvent method.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

That sounds intriguing. What would that look like. I've not used the LogUserEvent method before.

Does that trigger an event record in vCenter that we can detect like the earlier attempts?

Reply
0 Kudos
LucD
Leadership
Leadership

Yes, and it is quite simple to do.

$vmName = 'MyVM'

$si = Get-View ServiceInstance

$evtMgr = Get-View -Id $si.Content.EventManager


$vm = Get-VM -Name $vmName

$msg = "$(Get-Date -Format 'HH:mm:ss.ffff') Sending a user event"

$evtMgr.LogUserEvent($vm.ExtensionData.MoRef, $msg)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

That would require installing PowerCLI on all the test VMs?

Reply
0 Kudos
LucD
Leadership
Leadership

Yes, if you want to run such a script, you will need the PowerCLI framework to be present.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StephenMoll
Expert
Expert

That's getting quite involved.

I'll need to check after the Easter Bank Holiday weekend now, but hopefully the test VMs have the framework in place, but I suspect they don't.

Reply
0 Kudos
LucD
Leadership
Leadership

If you don't have PowerCLI on those target stations, you can still run a script in the guest OS.
You just need to think of another kind of semaphore to pass to the monitoring station.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos