VMware Cloud Community
andreaspa
Hot Shot
Hot Shot

Automating patching of Windows Templates - strange error on 2012 R2..

So, last friday I was about to patch templates in different systems, and this time it needed some special care for Spectre/Meltdown patches and workarounds. Figured I may as well finally start scripting this procedure as it needed to be repeated on several templates. Most of this is done with Invoke-VMScript and the use of different powershell modules, and utilize some pre-standardisation of our templates (same DVD drive letter, same user/password, script in c:\install etc.).

In broad terms, the script (in it's current revision) does the following:

- Checks for powered on VMs in folders named "Templates" that has guestID matching "windows"

- Adds registry value so January security patches gets installed (No AV in base image, except Windows Defender)

- Adds NuGet package provider

- Installs PSwindowsupdate PS Module

- Downloads all windows updates (filtered if needed)

- Install all Windows Updates and reboots the machine

- Runs DISM with some parameters to ensure a small image

- Maps the ISO for VMware Tools and runs the installation silently, then dismounts the CD

- Shutting down the VM when complete

.. everything is logged to c:\temp\<name of VM>.txt

I have been trying to figure out why some VMs work, and some don't. Often the failure is due to problems with powershell, and I'm guessing that is because the machine is not logged on (or have some intrusive patch installed).

The errors I get are these:

Join-Path : Cannot bind argument to parameter 'Path' because it is null.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:47 char:77

+ ... osoft.PowerShell.Management\Join-Path -Path $env:LOCALAPPDATA -ChildP ...

+                                                 ~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidData: (:) [Join-Path], ParameterBindingValidationException

    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.JoinPathCommand

The variable '$script:PSGetAppLocalPath' cannot be retrieved because it has not been set.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:48 char:86

+ ... werShell.Management\Join-Path -Path $script:PSGetAppLocalPath -ChildP ...

+                                         ~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (script:PSGetAppLocalPath:String) [], RuntimeException

    + FullyQualifiedErrorId : VariableIsUndefined

The variable '$script:PSGetAppLocalPath' cannot be retrieved because it has not been set.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:51 char:81

+ ... werShell.Management\Join-Path -Path $script:PSGetAppLocalPath -ChildP ...

+                                         ~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (script:PSGetAppLocalPath:String) [], RuntimeException

    + FullyQualifiedErrorId : VariableIsUndefined

WARNING: The property 'Values' cannot be found on this object. Verify that the property exists.

WARNING: The property 'Keys' cannot be found on this object. Verify that the property exists.

WARNING: The variable '$script:PSGetModuleSourcesFilePath' cannot be retrieved because it has not been set.

PackageManagement\Install-Package : No match was found for the specified search criteria and module name 'PSWindowsUpdate'. Try Get-PSRepository to see all available registered module repositories.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1772 char:21

+ ...          $null = PackageManagement\Install-Package @PSBoundParameters

+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Exception

    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

So far, it seems to be working if the console is logged on, or if the OS is 2008 R2.

Here's the script (it has no error handling or much intelligense yet, but it does the job..)

$guestuser = "administrator"

$guestpass = "password"

$skipupdates = "KB4033369,KB4033342"

$toolspath = "[Isofiles] VMware/VMware Tools/10.2.0/vmtools/windows.iso"

if( $Host -and $Host.UI -and $Host.UI.RawUI ) {

    $rawUI = $Host.UI.RawUI

    $oldSize = $rawUI.BufferSize

    $typeName = $oldSize.GetType( ).FullName

    $newSize = New-Object $typeName (120, $oldSize.Height)

    $rawUI.BufferSize = $newSize

  }

Clear-Host

foreach ($vm in (Get-Folder Templates | Get-VM | Sort-Object Name | Where-Object {$_.PowerState -eq "PoweredOn" -and $_.guestid -match "windows" })) {

    Write-Host Processing: $vm.name

    $file = "c:\temp\$($vm.name).txt"

    $now = get-date

    add-content $file "Actions started on $now"

    Write-host " - Adding registry values for Spectre/Meltdown patches"

    add-content $file "Adding registry values for Spectre/Meltdown patches"

    $output = Invoke-VMScript -vm $vm -ScriptText "reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0 /f" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat

    add-content $file $output

    Write-host " - Installing NuGet package provider"

    add-content $file "Installing NuGet package provider"

    $output = Invoke-VMScript -vm $vm -ScriptText "Install-PackageProvider nuget -force" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

    add-content $file $output

    Write-host " - Installing PSWindowsUpdate module"

    add-content $file "Installing PSWindowsUpdate module"

    $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PowerShellGet; Start-Sleep -Seconds 10; Install-Module -Name PSWindowsUpdate -force -scope AllUsers" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

    add-content $file $output

    if ($skipupdates -eq "") {

        Write-host " - Downloading all Windows Updates"

        add-content $file "Downloading all Windows Updates"

        $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Download -Verbose -AcceptAll" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

        add-content $file $output

        Write-host " - Installing all Windows Updates"

        add-content $file "Installing all Windows Updates"

        $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -MicrosoftUpdate -AcceptAll -Install -AutoReboot -Verbose" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

        add-content $file $output

    } else {

        Write-host " - Downloading all Windows Updates, except $skipupdates"

        add-content $file "Downloading all Windows Updates, except $skipupdates"

        $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Download -Verbose -AcceptAll -NotKBArticleID $skipupdates" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

        add-content $file $output

        Write-host " - Installing all Windows Updates, except $skipupdates"

        add-content $file "Installing all Windows Updates, except $skipupdates"

        $output = Invoke-VMScript -vm $vm -ScriptText "Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -MicrosoftUpdate -AcceptAll -Install -AutoReboot -NotKBArticleID $skipupdates -Verbose" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType PowerShell

        add-content $file $output

    }

    Write-Host " - Waiting 180 seconds for reboot to complete"

    Start-Sleep -Seconds 180

    Write-host " - Running cleanup script"

    add-content $file "Running cleanup script"

    $output = Invoke-VMScript -ToolsWaitSecs 360 -vm $vm -ScriptText "c:\install\cleanup.cmd" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat

    add-content $file $output

    Write-Host " - Upgrading VMware Tools"

    add-content $file "Upgrading VMware Tools"

    Get-CDDrive -vm $vm | Set-CDDrive -IsoPath $toolspath -Connected:$true -Confirm:$false | Out-Null

    Write-Host "  - Waiting for ISO to be mounted"

    Start-Sleep -Seconds 15

    $output = Invoke-VMScript -ToolsWaitSecs 360 -vm $vm -ScriptText "Z:\setup64.exe /s /v ""/qn REBOOT=Force ADDLOCAL=ALL REMOVE=Hgfs""" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType bat

    add-content $file $output

    #sleep added due to vmware tools upgrade trashing the connection

    Write-Host "  - Waiting five minutes for VMware Tools installation to complete"

    Start-Sleep -Seconds 300

    Write-Host "  - Dismounting ISO file"

    Get-CDDrive -vm $vm | Set-CDDrive -NoMedia -Connected:$false -Confirm:$false | Out-Null

    Write-host " - Shutting down the VM"

    add-content $file "Shutting down the VM"

    $output = Invoke-VMScript -vm $vm -ScriptText "shutdown /s /f /t 10 /d p:0:0" -GuestUser $guestuser -GuestPassword $guestpass -ScriptType Bat

    add-content $file $output

}

Have anyone else encountered problems with Invoke-VMScript and powershell?

(Note, I could prepare a BAT/CMD that does all the above, but this can be used for other purposes as well when automating customer patching later on)

Message was edited by: Andreas Cederlund Forgot to edit out password 🙂

Tags (1)
Reply
0 Kudos
10 Replies
LucD
Leadership
Leadership

Looks like  the environment variable returns $null.

Has the guest user ever logged on to the OS?

Does the account have a profile?

Can you try sending the CMD "set localappdata" to the guest OS through Invoke-VMScript (with that same account)?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Ok, this is strange 🙂

I shut down and started the VM again, waited some time (last time I waited two days, since time ran out..) and now this is the result:

Downloading all Windows Updates, except KB4033369,KB4033342

VERBOSE: WIN-E5A6VCSRL0I: Connecting to Microsoft Update server. Please wait...

VERBOSE: Found [1] Updates in pre search criteria

VERBOSE: Found [0] Updates in post search criteria

VERBOSE: Accepted [0] Updates ready to Download

VERBOSE: Downloaded [0] Updates ready to Install

Installing all Windows Updates, except KB4033369,KB4033342

Import-Module : Could not load file or assembly 'file:///C:\Program Files\Windo

wsPowerShell\Modules\PSWindowsUpdate\2.0.0.3\PSWindowsUpdate.dll' or one of its

dependencies. Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEX

PECTED))

At line:1 char:64

+ ... rocess -ExecutionPolicy Bypass; Import-Module PSWindowsUpdate; Start- ...

+                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileLoadExcep

   tion

    + FullyQualifiedErrorId : System.IO.FileLoadException,Microsoft.PowerShell

   .Commands.ImportModuleCommand

Get-WindowsUpdate : The 'Get-WindowsUpdate' command was found in the module 'PS

WindowsUpdate', but the module could not be loaded. For more information, run '

Import-Module PSWindowsUpdate'.

At line:1 char:120

+ ... e PSWindowsUpdate; Start-Sleep -seconds 10; Get-WindowsUpdate -Micros ...

+                                                 ~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ObjectNotFound: (Get-WindowsUpdate:String) [], C

   ommandNotFoundException

    + FullyQualifiedErrorId : CouldNotAutoloadMatchingModule

So, the first time the PSwindowsupdate module is loaded and executed correctly, but the second run it can't load the module properly. Could it be that I execute commands too close to the last command execution, so that processes hasn't finished their work?

I think that doing all this while the VM is logged on doesn't give any errors..

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Didn't see your reply before writing my own, sorry about that LucD​!

Yes, the user has logged on multiple times with this account.

I can run the "Install-PackageProvider nuget -force" command successfully each time, so powershell is correctly started and can add this provider, it's more like subsequent runs of powershell fails. Will tinker some more tomorrow and have a user logged on to the VM while running the command (which I did during development of the script, wanted to make sure things were happening).

Reply
0 Kudos
LucD
Leadership
Leadership

Could it be that your hitting this Paesecto.A thing that is doing the rounds?

I experience the same, Defender removes some files from modules, and hence you can't load them anymore


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Well, on the 2012 R2 there is no Windows Defender, which is even more odd.

I'm leaning more towards that there is some cleanup that's not being made properly between my Invoke-VMScript calls..

The first Invoke-VMScript with -ScriptType PowerShell _always_ works, but subsequent calls have issues on some machines.

Adding "Start-Sleep -Seconds 10" between them to see if that helps Smiley Happy

Reply
0 Kudos
LucD
Leadership
Leadership

Something I experienced in the past, are you sure that the guest OS deployment is completed?
And it's not too easy to determine when sysprep actually completes, short-off checking the registry.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Yes, no guest os deployment is done at all.

Our templates are regular VMs, we just mark them as VM and run this script. We entrust VMware with running the sysprep while doing guest customization Smiley Happy

So basically, these are just regular VMs. This run I tried with them not being logged in, will try to run this while logged on to all VMs.

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Alright, I have some results.

If I am logged on to the console of the VM, everything works out nicely.

Conclusion - there are issues running powershell command if the console is not logged on, and it applies on all OS'es I've tested (2008 R2, 2012, 2012 R2, 2016). Known bug?

Reply
0 Kudos
LucD
Leadership
Leadership

Just one more question, was PowerShell ever started on these boxes?
And if yes, under which account? SYSTEM?

You can schedule a PS script to run under SYSTEM as a scheduled task.
I wonder if that would see the same issues?

And yes, creating a Scheduled Task with a PS script, when there is a possible PS issue, sounds like a chicken and egg thing :smileygrin:


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
andreaspa
Hot Shot
Hot Shot

Just one more question, was PowerShell ever started on these boxes?
And if yes, under which account? SYSTEM?

Yes, under the "Administrator" account, and as whatever account Invoke-VMScript runs as.

You can schedule a PS script to run under SYSTEM as a scheduled task.
I wonder if that would see the same issues?

And yes, creating a Scheduled Task with a PS script, when there is a possible PS issue, sounds like a chicken and egg thing

Good questions!

To clarify how I tested things, I have four VMs that are templates. I marked them as regular VMs, powered them on, ran the script on them, and noticed the result. I then powered the same VMs on again, and logged on to the console on them and ran the script, and the third time I powered them on again and ran the script without logging on to the console.

Only conclusion I could take away is that when the VMs have their console logged on, everything works. When not being logged on, there are some strange errors.

Reply
0 Kudos