1 2 Previous Next 18 Replies Latest reply on Apr 18, 2019 1:08 PM by LucD

    Detecting a VM starting to perform a Guest Shutdown

    StephenMoll Hot Shot

      What is the earliest point in the shutdown process a PowerCLI script could detect that a Guest has initiated a controlled shutdown?

       

      I'm asking because I am looking at a system where tight IOPS limits are applied to all VMs. Don't ask, it is simply a requirement we have to meet. Some of these are VERY low, which seriously hampers the VM startup and shutdown processes.

       

      Startup is a trivial thing to deal with:

       

      When VMs are off, have the IOPS limits unset. When a VM starts, apply the IOPS limits, when VMTools reports the guest is running, with perhaps an additional time delay to allow a limit headroom. Or more simply, apply the limits a set time after VM power-on.

       

      The tricky one is when a VM is shutting down. If the process is initiated from within the guest, I would like to be able to detect this has happened and remove the IOPS limits, so the VM can shutdown swiftly, and be in the correct state for the next power-on.

       

      Any ideas out there?

        • 1. Re: Detecting a VM starting to perform a Guest Shutdown
          LucD Guru
          vExpertCommunity WarriorsUser Moderators

          You could look for an occurrence of the VmStoppingEvent.

          This event is triggered when the shutdown starts, before it is actually shutdown.


          You could look for example every minute of such an event occurred in the last minute.

           

          $start = (Get-Date).AddMinutes(-1)

          Get-VIEvent -Start $start |

          where{$_ -is [VMware.Vim.VmStoppingEvent]}

           

          As an alternative, you could create an Alarm based on that event.

          And then trigger for example a Trap to a SNMP server, which could then take appropriate action via a script.

          • 2. Re: Detecting a VM starting to perform a Guest Shutdown
            StephenMoll Hot Shot

            I shall get someone to take a look at that. Thanks LucD.

            • 3. Re: Detecting a VM starting to perform a Guest Shutdown
              StephenMoll Hot Shot

              I've had a play with this, and it didn't work.

               

              Examining the full output of Get-VIEvent in gridview I see tasks that indicate the "{VM Name} on {Host} in {Cluster} is powered off". The output when "where {$_ -is [VMware.Vim.VmStoppingEvent]}" is added is completely empty.

               

              I am guessing that the events I see (as described above) are a bit late. What I am trying to do is detect when a VM is shutting down and remove any IOPS limits from it so it I can speed up the shutdown, and massively speed up the next startup. Start up is not a problem, the VMs are started by a script, and are started and then IOPS limits applied a minute or so later. When the script is used to shut them down, likewise the limits are removed and a guest shutdown initiated. The problem is what to do when the shutdown is commanded from within the guest.

               

              The entries I am seeing don't appear to have any fields matching the value your snippet looks for. I see "Vm = VMware.Vim.VMEventArgument" and others related to Host, Compute Resource and Datacenter.

              • 4. Re: Detecting a VM starting to perform a Guest Shutdown
                LucD Guru
                Community WarriorsvExpertUser Moderators

                Can you check what kind of events are raised for the VM that was stopped in the last X minutes (example goes for 15 mins).

                 

                $vmName = 'MyVM'

                 

                $vm = Get-VM -Name $vmName


                Get-VIEvent -Entity $vm -MaxSamples ([int]::MaxValue) -Start (Get-Date).AddMinutes(-15) |

                Group-Object -Property { $_.GetType().Name }

                • 5. Re: Detecting a VM starting to perform a Guest Shutdown
                  StephenMoll Hot Shot

                  Startup produces:

                   

                  Count Name                       Group

                  ----- ----                       -----

                      1 VmAcquiredTicketEvent      {VMware.Vim.VmAcquiredTicketEvent}

                      1 VmPoweredOnEvent           {VMware.Vim.VmPoweredOnEvent}

                      1 VmStartingEvent            {VMware.Vim.VmStaringEvent}

                      1 TaskEvent                  {VMware.Vim.TaskEvent}

                   

                   

                  Shutdown using in Guest Windows Shutdown from Start Menu produces:

                   

                  Count Name                       Group

                  ----- ----                       -----

                      4 AlarmStatusChangedEvent    {VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent}

                      1 VmPoweredOffEvent          {VMware.Vim.VmPoweredOffEvent}

                   

                   

                  A restart from within Windows produces:

                   

                  Count Name                       Group

                  ----- ----                       -----

                      2 AlarmStatusChangedEvent    {VMware.Vim.AlarmStatusChangedEvent, VMware.Vim.AlarmStatusChangedEvent}

                   

                   

                  I'm not seeing anything that looks convincingly early enough in the shutdown process to make a real difference. Some of the VMs are required to have stupidly low IOPS limits set, which as you can imagine really hampers VM power-on, but also affects guest-shutdown as well.

                   

                   

                  • 6. Re: Detecting a VM starting to perform a Guest Shutdown
                    LucD Guru
                    Community WarriorsvExpertUser Moderators

                    Do these VMs have the VMware Tools installed and running?

                    • 7. Re: Detecting a VM starting to perform a Guest Shutdown
                      StephenMoll Hot Shot

                      Yes they do. We rely on VMtools in order to harvest the VMs for performance test results data.

                      • 8. Re: Detecting a VM starting to perform a Guest Shutdown
                        LucD Guru
                        vExpertCommunity WarriorsUser Moderators

                        Does this give sufficient time?

                         

                        $vm = Get-VM MyVM

                        Get-Date

                        Shutdown-VMGuest -VM $vm -Confirm:$false

                        while($vm.ExtensionData.Guest.GuestOperationsReady){

                          sleep 2

                           $vm.ExtensionData.UpdateViewData('Guest.GuestOperationsReady')

                        }

                        Write-Host "VM is powering off"

                        Get-Date

                        $vm.ExtensionData.UpdateViewData('Runtime.PowerState')

                        while($vm.ExtensionData.Runtime.PowerState -ne 'poweredoff'){

                          sleep 2

                           $vm.ExtensionData.UpdateViewData('Runtime.PowerState')

                        }

                        Get-Date

                         

                         

                        If it doesn't (and that is what I think will turn out), then that is because the VMware Tools are one of the last processes to be stopped

                         

                        An alternative which you could use, since you have the VMware Tools installed and running, is to use the ListProcessesInGuest method.
                        Depending on which guest OS you have running inside the VM, you could check if one or more processes are stopped.

                        • 9. Re: Detecting a VM starting to perform a Guest Shutdown
                          StephenMoll Hot Shot

                          I'll have a play with that and see what it does. I suspect that getting an early enough signal that a VM is shutting down due to an internal action is going to be tricky. I think I will just suggest that when using the test VMs that all shutdown be done using the external command scripts.

                           

                          Basically we have a VM template that has some performance measuring tools on board. We have a set of PowerCLI scripts that role out multiple VMs from this template that is used to simulate a final whole system across the cluster. We use this to simulate workloads for CPU, memory and storage IO testing across the whole hardware suite. A test is configured, started and stopped through these scripts and the results harvested from all the VMs as well.

                           

                          With the CPU and memory testing there were no issues, but with storage IO it is different. The final design requires us to set IOPS limit based on the final design. All system resources are accounted for and no overprovisioning is permitted as a strict requirement. So where some VMs are specified with very low run-time IOPS requirement, the VMs take an age to boot up, so we really only want to apply the limits after the VM has booted up. We then would like to remove the limits before shutting the VMs down. This is easy when people stick to using the control scripts to do this, but that isn't always the case. So I'll probably just end up telling people that manual shutdown performance of a test VM may not be great.

                          • 10. Re: Detecting a VM starting to perform a Guest Shutdown
                            StephenMoll Hot Shot

                            Just ran:

                             

                            18 April 2019 14:45:06

                             

                            OSFullName   : Microsoft Windows 8 (64-bit)

                            :

                            GuestFamily  : windowsguest

                             

                            VM is powering off

                            18 April 2019 14:45:06

                            18 April 2019 14:45:24

                             

                             

                            So time for this VM to power of was about 18 seconds. But of course in this test we knew when the shutdown started, because it was initiated by PowerCLI.

                             

                            I wonder if there is anyway to use VMTools to place a small PowerShell script inside the test VM which will trigger a detectable event as soon as someone clicks shutdown or restart inside the guest?

                            • 11. Re: Detecting a VM starting to perform a Guest Shutdown
                              StephenMoll Hot Shot

                              What would be really useful I think would be the ability to define further parameters for VM limits and reservations, but especially for IOPS. These parameters would be to specify when the limits apply, for example:

                               

                              IOPS limit : nnn IOPS

                              Activation time : Immediate | sss Seconds after power on | When "toolsOK"

                              Deactivation time : At start of guest shutdown | After poweroff | Never

                              • 12. Re: Detecting a VM starting to perform a Guest Shutdown
                                LucD Guru
                                User ModeratorsCommunity WarriorsvExpert

                                That is exactly what I suggested with the use of the ListProcessesInGuest method.

                                The problem might be what to look for.
                                It will depend on what kind of guest OS you have running inside the VM.

                                 

                                Another option is to run a script inside the VM (with Invoek-VMScript) and let that script fire for example send an event with the LogUserEvent method.

                                • 13. Re: Detecting a VM starting to perform a Guest Shutdown
                                  StephenMoll Hot Shot

                                  That sounds intriguing. What would that look like. I've not used the LogUserEvent method before.

                                  Does that trigger an event record in vCenter that we can detect like the earlier attempts?

                                  • 14. Re: Detecting a VM starting to perform a Guest Shutdown
                                    LucD Guru
                                    User ModeratorsvExpertCommunity Warriors

                                    Yes, and it is quite simple to do.

                                     

                                    $vmName = 'MyVM'

                                    $si = Get-View ServiceInstance

                                    $evtMgr = Get-View -Id $si.Content.EventManager


                                    $vm = Get-VM -Name $vmName

                                    $msg = "$(Get-Date -Format 'HH:mm:ss.ffff') Sending a user event"

                                    $evtMgr.LogUserEvent($vm.ExtensionData.MoRef, $msg)

                                    1 2 Previous Next