10 Replies Latest reply on Jan 29, 2018 9:19 AM by andreaspa

    Automating patching of Windows Templates - strange error on 2012 R2..

    andreaspa Enthusiast

      So, last friday I was about to patch templates in different systems, and this time it needed some special care for Spectre/Meltdown patches and workarounds. Figured I may as well finally start scripting this procedure as it needed to be repeated on several templates. Most of this is done with Invoke-VMScript and the use of different powershell modules, and utilize some pre-standardisation of our templates (same DVD drive letter, same user/password, script in c:\install etc.).

       

      In broad terms, the script (in it's current revision) does the following:

      - Checks for powered on VMs in folders named "Templates" that has guestID matching "windows"

      - Adds registry value so January security patches gets installed (No AV in base image, except Windows Defender)

      - Adds NuGet package provider

      - Installs PSwindowsupdate PS Module

      - Downloads all windows updates (filtered if needed)

      - Install all Windows Updates and reboots the machine

      - Runs DISM with some parameters to ensure a small image

      - Maps the ISO for VMware Tools and runs the installation silently, then dismounts the CD

      - Shutting down the VM when complete

      .. everything is logged to c:\temp\<name of VM>.txt

       

      I have been trying to figure out why some VMs work, and some don't. Often the failure is due to problems with powershell, and I'm guessing that is because the machine is not logged on (or have some intrusive patch installed).

       

      The errors I get are these:

      Join-Path : Cannot bind argument to parameter 'Path' because it is null.
      At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:47 char:77
      + ... osoft.PowerShell.Management\Join-Path -Path $env:LOCALAPPDATA -ChildP ...
      +                                                 ~~~~~~~~~~~~~~~~~
          + CategoryInfo          : InvalidData: (:) [Join-Path], ParameterBindingValidationException
          + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.JoinPathCommand
      
      The variable '$script:PSGetAppLocalPath' cannot be retrieved because it has not been set.
      At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:48 char:86
      + ... werShell.Management\Join-Path -Path $script:PSGetAppLocalPath -ChildP ...
      +                                         ~~~~~~~~~~~~~~~~~~~~~~~~~
          + CategoryInfo          : InvalidOperation: (script:PSGetAppLocalPath:String) [], RuntimeException
          + FullyQualifiedErrorId : VariableIsUndefined
      
      The variable '$script:PSGetAppLocalPath' cannot be retrieved because it has not been set.
      At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:51 char:81
      + ... werShell.Management\Join-Path -Path $script:PSGetAppLocalPath -ChildP ...
      +                                         ~~~~~~~~~~~~~~~~~~~~~~~~~
          + CategoryInfo          : InvalidOperation: (script:PSGetAppLocalPath:String) [], RuntimeException
          + FullyQualifiedErrorId : VariableIsUndefined
      
      WARNING: The property 'Values' cannot be found on this object. Verify that the property exists.
      WARNING: The property 'Keys' cannot be found on this object. Verify that the property exists.
      WARNING: The variable '$script:PSGetModuleSourcesFilePath' cannot be retrieved because it has not been set.
      PackageManagement\Install-Package : No match was found for the specified search criteria and module name 'PSWindowsUpdate'. Try Get-PSRepository to see all available registered module repositories.
      At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1772 char:21
      + ...          $null = PackageManagement\Install-Package @PSBoundParameters
      +                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Exception
          + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
      
      

       

      So far, it seems to be working if the console is logged on, or if the OS is 2008 R2.

      Here's the script (it has no error handling or much intelligense yet, but it does the job..)

       

      $guestuser = "administrator"
      $guestpass = "password"
      $skipupdates = "KB4033369,KB4033342"
      $toolspath = "[Isofiles] VMware/VMware Tools/10.2.0/vmtools/windows.iso"
      
      if( $Host -and $Host.UI -and $Host.UI.RawUI ) {
          $rawUI = $Host.UI.RawUI
          $oldSize = $rawUI.BufferSize
          $typeName = $oldSize.GetType( ).FullName
          $newSize = New-Object $typeName (120, $oldSize.Height)
          $rawUI.BufferSize = $newSize
        }
      
      Clear-Host
      
      foreach ($vm in (Get-Folder Templates | Get-VM | Sort-Object Name | Where-Object {$_.PowerState -eq "PoweredOn" -and $_.guestid -match "windows" })) {
          Write-Host Processing: $vm.name
          $file = "c:\temp\$($vm.name).txt"
          $now = get-date
      
          add-content $file "Actions started on $now"
      
          Write-host " - Adding registry values for Spectre/Meltdown patches"
          add-content $file "Adding registry values for Spectre/Meltdown patches"
          $output = Invoke-VMScript -vm $vm -ScriptText "reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0 /f" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat
          add-content $file $output
      
          Write-host " - Installing NuGet package provider"
          add-content $file "Installing NuGet package provider"
          $output = Invoke-VMScript -vm $vm -ScriptText "Install-PackageProvider nuget -force" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
          add-content $file $output
      
          Write-host " - Installing PSWindowsUpdate module"
          add-content $file "Installing PSWindowsUpdate module"
          $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PowerShellGet; Start-Sleep -Seconds 10; Install-Module -Name PSWindowsUpdate -force -scope AllUsers" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
          add-content $file $output
      
          if ($skipupdates -eq "") {
              Write-host " - Downloading all Windows Updates"
              add-content $file "Downloading all Windows Updates"
              $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Download -Verbose -AcceptAll" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
              add-content $file $output
      
              Write-host " - Installing all Windows Updates"
              add-content $file "Installing all Windows Updates"
              $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -MicrosoftUpdate -AcceptAll -Install -AutoReboot -Verbose" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
              add-content $file $output
          } else {
              Write-host " - Downloading all Windows Updates, except $skipupdates"
              add-content $file "Downloading all Windows Updates, except $skipupdates"
              $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Download -Verbose -AcceptAll -NotKBArticleID $skipupdates" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
              add-content $file $output
      
              Write-host " - Installing all Windows Updates, except $skipupdates"
              add-content $file "Installing all Windows Updates, except $skipupdates"
              $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -MicrosoftUpdate -AcceptAll -Install -AutoReboot -NotKBArticleID $skipupdates -Verbose" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell
              add-content $file $output
          }
      
          Write-Host " - Waiting 180 seconds for reboot to complete"
          Start-Sleep -Seconds 180
      
          Write-host " - Running cleanup script"
          add-content $file "Running cleanup script"
          $output = Invoke-VMScript -ToolsWaitSecs 360 -vm $vm -ScriptText "c:\install\cleanup.cmd" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat
          add-content $file $output
      
          Write-Host " - Upgrading VMware Tools"
          add-content $file "Upgrading VMware Tools"
          Get-CDDrive -vm $vm | Set-CDDrive -IsoPath $toolspath -Connected:$true -Confirm:$false | Out-Null
          Write-Host "  - Waiting for ISO to be mounted"
          Start-Sleep -Seconds 15
          $output = Invoke-VMScript -ToolsWaitSecs 360 -vm $vm -ScriptText "Z:\setup64.exe /s /v ""/qn REBOOT=Force ADDLOCAL=ALL REMOVE=Hgfs""" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat
          add-content $file $output
          #sleep added due to vmware tools upgrade trashing the connection
          Write-Host "  - Waiting five minutes for VMware Tools installation to complete"
          Start-Sleep -Seconds 300
          Write-Host "  - Dismounting ISO file"
          Get-CDDrive -vm $vm | Set-CDDrive -NoMedia -Connected:$false -Confirm:$false | Out-Null
      
      
          Write-host " - Shutting down the VM"
          add-content $file "Shutting down the VM"
          $output = Invoke-VMScript -vm $vm -ScriptText "shutdown /s /f /t 10 /d p:0:0" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType Bat
          add-content $file $output
      }
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      

       

      Have anyone else encountered problems with Invoke-VMScript and powershell?

      (Note, I could prepare a BAT/CMD that does all the above, but this can be used for other purposes as well when automating customer patching later on)

       

      Message was edited by: Andreas Cederlund Forgot to edit out password :-)

        • 1. Re: Automating patching of Windows Templates - strange error on 2012 R2..
          LucD Guru
          User ModeratorsCommunity WarriorsvExpert

          Looks like  the environment variable returns $null.

          Has the guest user ever logged on to the OS?

          Does the account have a profile?

          Can you try sending the CMD "set localappdata" to the guest OS through Invoke-VMScript (with that same account)?

          • 2. Re: Automating patching of Windows Templates - strange error on 2012 R2..
            andreaspa Enthusiast

            Ok, this is strange :-)

             

            I shut down and started the VM again, waited some time (last time I waited two days, since time ran out..) and now this is the result:

            Downloading all Windows Updates, except KB4033369,KB4033342
            VERBOSE: WIN-E5A6VCSRL0I: Connecting to Microsoft Update server. Please wait...
            VERBOSE: Found [1] Updates in pre search criteria
            VERBOSE: Found [0] Updates in post search criteria
            VERBOSE: Accepted [0] Updates ready to Download
            VERBOSE: Downloaded [0] Updates ready to Install
            
            Installing all Windows Updates, except KB4033369,KB4033342
            Import-Module : Could not load file or assembly 'file:///C:\Program Files\Windo
            wsPowerShell\Modules\PSWindowsUpdate\2.0.0.3\PSWindowsUpdate.dll' or one of its
             dependencies. Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEX
            PECTED))
            At line:1 char:64
            + ... rocess -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start- ...
            +                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                + CategoryInfo          : NotSpecified: (:) [Import-Module], FileLoadExcep 
               tion
                + FullyQualifiedErrorId : System.IO.FileLoadException,Microsoft.PowerShell 
               .Commands.ImportModuleCommand
            
            Get-WindowsUpdate : The 'Get-WindowsUpdate' command was found in the module 'PS
            WindowsUpdate', but the module could not be loaded. For more information, run '
            Import-Module PSWindowsUpdate'.
            At line:1 char:120
            + ... e PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Micros ...
            +                                                 ~~~~~~~~~~~~~~~~~
                + CategoryInfo          : ObjectNotFound: (Get-WindowsUpdate:String) [], C 
               ommandNotFoundException
                + FullyQualifiedErrorId : CouldNotAutoloadMatchingModule
            
            
            

             

            So, the first time the PSwindowsupdate module is loaded and executed correctly, but the second run it can't load the module properly. Could it be that I execute commands too close to the last command execution, so that processes hasn't finished their work?

             

            I think that doing all this while the VM is logged on doesn't give any errors..

            • 3. Re: Automating patching of Windows Templates - strange error on 2012 R2..
              andreaspa Enthusiast

              Didn't see your reply before writing my own, sorry about that LucD!

              Yes, the user has logged on multiple times with this account.

               

              I can run the "Install-PackageProvider nuget -force" command successfully each time, so powershell is correctly started and can add this provider, it's more like subsequent runs of powershell fails. Will tinker some more tomorrow and have a user logged on to the VM while running the command (which I did during development of the script, wanted to make sure things were happening).

              • 4. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                LucD Guru
                User ModeratorsvExpertCommunity Warriors

                Could it be that your hitting this Paesecto.A thing that is doing the rounds?

                I experience the same, Defender removes some files from modules, and hence you can't load them anymore

                • 5. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                  andreaspa Enthusiast

                  Well, on the 2012 R2 there is no Windows Defender, which is even more odd.

                  I'm leaning more towards that there is some cleanup that's not being made properly between my Invoke-VMScript calls..

                   

                  The first Invoke-VMScript with -ScriptType PowerShell _always_ works, but subsequent calls have issues on some machines.

                  Adding "Start-Sleep -Seconds 10" between them to see if that helps

                  • 6. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                    LucD Guru
                    User ModeratorsCommunity WarriorsvExpert

                    Something I experienced in the past, are you sure that the guest OS deployment is completed?
                    And it's not too easy to determine when sysprep actually completes, short-off checking the registry.

                    • 7. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                      andreaspa Enthusiast

                      Yes, no guest os deployment is done at all.

                      Our templates are regular VMs, we just mark them as VM and run this script. We entrust VMware with running the sysprep while doing guest customization

                       

                      So basically, these are just regular VMs. This run I tried with them not being logged in, will try to run this while logged on to all VMs.

                      • 8. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                        andreaspa Enthusiast

                        Alright, I have some results.

                         

                        If I am logged on to the console of the VM, everything works out nicely.

                         

                        Conclusion - there are issues running powershell command if the console is not logged on, and it applies on all OS'es I've tested (2008 R2, 2012, 2012 R2, 2016). Known bug?

                        • 9. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                          LucD Guru
                          vExpertUser ModeratorsCommunity Warriors

                          Just one more question, was PowerShell ever started on these boxes?
                          And if yes, under which account? SYSTEM?

                           

                          You can schedule a PS script to run under SYSTEM as a scheduled task.
                          I wonder if that would see the same issues?

                           

                          And yes, creating a Scheduled Task with a PS script, when there is a possible PS issue, sounds like a chicken and egg thing

                          • 10. Re: Automating patching of Windows Templates - strange error on 2012 R2..
                            andreaspa Enthusiast

                            Just one more question, was PowerShell ever started on these boxes?
                            And if yes, under which account? SYSTEM?

                             

                            Yes, under the "Administrator" account, and as whatever account Invoke-VMScript runs as.

                             

                            You can schedule a PS script to run under SYSTEM as a scheduled task.
                            I wonder if that would see the same issues?

                             

                            And yes, creating a Scheduled Task with a PS script, when there is a possible PS issue, sounds like a chicken and egg thing

                            Good questions!

                             

                            To clarify how I tested things, I have four VMs that are templates. I marked them as regular VMs, powered them on, ran the script on them, and noticed the result. I then powered the same VMs on again, and logged on to the console on them and ran the script, and the third time I powered them on again and ran the script without logging on to the console.

                             

                            Only conclusion I could take away is that when the VMs have their console logged on, everything works. When not being logged on, there are some strange errors.