PowerCLI Update-Tools cmdlet throwing errors (GUI tools update works)... How to fix?

[GeoPerkins]

Trying to remotely run PowerShell to update the VMware Tools, but getting errors. I can update tools if I use the GUI. 

Here's my command:

Get-VMGuest $VM | Update-Tools -NoReboot -RunAsync | Wait-Task

These are the errors:

• Error upgrading VMware Tools vix error code 21009.
• ToolsISO Failed to open Tools ISO /vmfs/volumes/... /productLocker/vmtools/windows.iso (FileIOResult=2).
• Warning [autoUpgrade] RPC guest.initiateUpgradeAtShutdown failed, reply is Unknown command.

Guest VM is a Windows 2019 Server.

vCenter Version 7.0.3 Build number 22357613.

ESXi 7.0.2 Build number 18426014.

VMware Tools 12.2.0 12352.

VMware.PowerCLI 12.7.0 build 20091289

VMware Common PowerCLI Component 12.7 build 20067789

VMware Cis Core PowerCLI Component PowerCLI Component 12.6 build 19601368

VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 12.7 build 20091293

[GeoPerkins]

RTFM:

Do I need to run Mount-Tools first? Then Update-Tools. Then Dismount-Tools?

And I am looking at my original command (sorry I inherited it). Option RunAsync piped to Wait-Task seems redundant.  Would this be better:

Get-VMGuest $VM | Mount-Tools | Update-Tools -NoReboot | Dismount-Tools 

Would that form of the pipeline work properly?

[LucD]

No, you don't need to run Mount-Tools if the VMware Tools are already installed in the VM's Guest OS.

But Vix error 21009 is a known issue.
Unfortunately, it can have multiple causes.
I would start by checking KB2129927 and KB1037405 and I would check the vmware.log file.

Also, check the VM's advanced setting

isolation.tools.autoInstall.disable = FALSE

[GeoPerkins]

KB2129927 and KB1037405 do not apply in this case. I had already searched and viewed those old solutions.

The relevant error message posted is from the vmware.log,

ToolsISO: Failed to open Tools ISO /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/windows.iso (FileIOResult=2).

the complete area of that log contains:

2024-02-27T22:20:18.044Z| vmx| | I005: VigorTransportProcessClientPayload: opID=30a7c8a7-ee-53f5 seq=1612: Receiving ToolsInstallManager.Install request.
2024-02-27T22:20:18.044Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL entering BEGINNING state.
2024-02-27T22:20:18.044Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL updating Rpc handlers registration.
2024-02-27T22:20:18.044Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL toolInstall.guestStoreEnabled=0
2024-02-27T22:20:18.044Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Refreshing imageName for 'windows9srv-64' (refreshCount=1, lastCount=1).
2024-02-27T22:20:18.048Z| vmx| opID=30a7c8a7-ee-53f5| I005: FILE:open error on /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/isoimages_manifest.txt: Operation not permitted
2024-02-27T22:20:18.048Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: open of /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/isoimages_manifest.txt failed: Error
2024-02-27T22:20:18.048Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Updated cached value for imageName to 'windows.iso'.
2024-02-27T22:20:18.048Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Selected Tools ISO 'windows.iso' for 'windows9srv-64' guest.
2024-02-27T22:20:18.048Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL Attempting auto-upgrade (GuestStoreUpgrade=false).
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS installed legacy version 11265, available legacy version 11334
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS manifest update status is 2
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS using cached value 1 for isoImageExists (refreshCount=1, lastCount=1).
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS can be autoupgraded.
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL Launching auto-upgrader with args "/s /v"/qn REBOOT=ReallySuppress"".
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL Register required Rpc handlers
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL updating Rpc handlers registration.
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL autoupgrading old tools.
2024-02-27T22:20:18.049Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Refreshing imageName for 'windows9srv-64' (refreshCount=1, lastCount=1).
2024-02-27T22:20:18.051Z| vmx| opID=30a7c8a7-ee-53f5| I005: FILE:open error on /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/isoimages_manifest.txt: Operation not permitted
2024-02-27T22:20:18.051Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: open of /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/isoimages_manifest.txt failed: Error
2024-02-27T22:20:18.051Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Updated cached value for imageName to 'windows.iso'.
2024-02-27T22:20:18.051Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Selected Tools ISO 'windows.iso' for 'windows9srv-64' guest.
2024-02-27T22:20:18.054Z| vmx| opID=30a7c8a7-ee-53f5| I005: FILE:open error on /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/windows.iso: Operation not permitted
2024-02-27T22:20:18.054Z| vmx| opID=30a7c8a7-ee-53f5| I005: ToolsISO: Failed to open Tools ISO /vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools/windows.iso (FileIOResult=2).
2024-02-27T22:20:18.054Z| vmx| opID=30a7c8a7-ee-53f5| I005: Cannot open ISO image with upgrader binaries.
2024-02-27T22:20:18.054Z| vmx| opID=30a7c8a7-ee-53f5| I005: TOOLS INSTALL entering IDLE state.
2024-02-27T22:20:18.054Z| vmx| opID=30a7c8a7-ee-53f5| A000: ConfigDB: Setting toolsInstallManager.updateCounter = "10"
2024-02-27T22:20:18.062Z| vmx| opID=30a7c8a7-ee-53f5| A000: ConfigDB: Setting extendedConfigFile = "ASiCEStst.vmxf"
2024-02-27T22:20:18.062Z| vmx| opID=30a7c8a7-ee-53f5| A000: ConfigDB: Setting toolsInstallManager.lastInstallError = "21009"
2024-02-27T22:20:18.062Z| vmx| opID=30a7c8a7-ee-53f5| I005: Vix: [vigorCommands.c:1004]: VigorToolsInstallManagerCommandCallback: toolInstallErr = 12
2024-02-27T22:20:18.063Z| vmx| opID=30a7c8a7-ee-53f5| I005: VigorTransport_ServerSendResponse opID=30a7c8a7-ee-53f5 seq=1612: Completed ToolsInstallManager request with messages.
2024-02-27T22:20:18.063Z| vmx| | A000: ConfigDB: Setting toolsInstallManager.lastInstallError = "21009"
2024-02-27T22:20:18.063Z| vmx| | I005: TOOLS INSTALL entering IDLE state.

I am opening a ticket with VMware support, but they do not support PowerCLI. Catch-22.

As far as the error message indicating "Operation not permitted" I do not know what that means. I am able to list the contents of the directory:  

[root@OneNeck01-5:~] cd vmfs
[root@OneNeck01-5:/vmfs] cd volumes
[root@OneNeck01-5:/vmfs/volumes] cd ebc504d1-d7bffd90
[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90] cd productLocker
[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90/productLocker] cd vmtools
[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools] ls
buildNumber.txt               version.txt                   windows.iso.sig
isoimages_manifest.txt        windows.iso                   windows_avr_manifest.txt
isoimages_manifest.txt.sig    windows.iso.sha               windows_avr_manifest.txt.sig

[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools] 
cat buildNumber.txt
21223074

[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools] cat isoimages_manifest.txt
.encoding = "utf-8"
almaLinux-64 = "linux.iso"
arm-windows10-64 = "windows.iso"
arm-windows11-64 = "windows.iso"
asianux3 = "linuxPreGlibc25.iso"
asianux3-64 = "linuxPreGlibc25.iso"
asianux4 = "linuxPreGlibc25.iso"
asianux4-64 = "linuxPreGlibc25.iso"
asianux5-64 = "linux.iso"
asianux7-64 = "linux.iso"
asianux8-64 = "linux.iso"
asianux9-64 = "linux.iso"
centos = "linux.iso"
centos-64 = "linux.iso"
centos6 = "linux.iso"
centos6-64 = "linux.iso"
centos7-64 = "linux.iso"
centos8-64 = "linux.iso"
centos9-64 = "linux.iso"
darwin = "darwinPre15.iso"
darwin-64 = "darwinPre15.iso"
darwin10 = "darwinPre15.iso"
darwin10-64 = "darwinPre15.iso"
darwin11 = "darwinPre15.iso"
darwin11-64 = "darwinPre15.iso"
darwin12-64 = "darwinPre15.iso"
darwin13-64 = "darwinPre15.iso"
darwin14-64 = "darwinPre15.iso"
darwin15-64 = "darwin.iso"
darwin16-64 = "darwin.iso"
darwin17-64 = "darwin.iso"
darwin18-64 = "darwin.iso"
darwin19-64 = "darwin.iso"
darwin20-64 = "darwin.iso"
darwin21-64 = "darwin.iso"
darwin22-64 = "darwin.iso"
darwin23-64 = "darwin.iso"
debian4 = "linuxPreGlibc25.iso"
debian4-64 = "linuxPreGlibc25.iso"
debian5 = "linuxPreGlibc25.iso"
debian5-64 = "linuxPreGlibc25.iso"
debian6 = "linux.iso"
debian6-64 = "linux.iso"
debian7 = "linux.iso"
debian7-64 = "linux.iso"
debian8 = "linux.iso"
debian8-64 = "linux.iso"
debian9 = "linux.iso"
debian9-64 = "linux.iso"
debian10 = "linux.iso"
debian10-64 = "linux.iso"
debian11 = "linux.iso"
debian11-64 = "linux.iso"
debian12 = "linux.iso"
debian12-64 = "linux.iso"
Fedora = "linux.iso"
Fedora-64 = "linux.iso"
flatcar-64 = "linux.iso"
freeBSD = "freebsd.iso"
freeBSD-64 = "freebsd.iso"
freeBSD11 = "freebsd.iso"
freeBSD11-64 = "freebsd.iso"
freeBSD12 = "freebsd.iso"
freeBSD12-64 = "freebsd.iso"
freeBSD13 = "freebsd.iso"
freeBSD13-64 = "freebsd.iso"
freeBSD14 = "freebsd.iso"
freeBSD14-64 = "freebsd.iso"
linux = "linuxPreGlibc25.iso"
longhorn = "windows.iso"
longhorn-64 = "windows.iso"
mandrake = "linuxPreGlibc25.iso"
mandrake-64 = "linuxPreGlibc25.iso"
mandriva = "linuxPreGlibc25.iso"
mandriva-64 = "linuxPreGlibc25.iso"
netware4 = "netware.iso"
netware5 = "netware.iso"
netware6 = "netware.iso"
nld9 = "linuxPreGlibc25.iso"
nt4 = "winPre2k.iso"
oes = "linuxPreGlibc25.iso"
opensuse = "linux.iso"
opensuse-64 = "linux.iso"
oraclelinux = "linux.iso"
oraclelinux-64 = "linux.iso"
oraclelinux6 = "linux.iso"
oraclelinux6-64 = "linux.iso"
oraclelinux7-64 = "linux.iso"
oraclelinux8-64 = "linux.iso"
oraclelinux9-64 = "linux.iso"
other24xlinux = "linuxPreGlibc25.iso"
other24xlinux-64 = "linuxPreGlibc25.iso"
other26xlinux = "linux.iso"
other26xlinux-64 = "linux.iso"
other3xlinux = "linux.iso"
other3xlinux-64 = "linux.iso"
other4xlinux = "linux.iso"
other4xlinux-64 = "linux.iso"
other5xlinux = "linux.iso"
other5xlinux-64 = "linux.iso"
other6xlinux = "linux.iso"
other6xlinux-64 = "linux.iso"
redhat = "linuxPreGlibc25.iso"
rhel2 = "linuxPreGlibc25.iso"
rhel3 = "linuxPreGlibc25.iso"
rhel3-64 = "linuxPreGlibc25.iso"
rhel4 = "linuxPreGlibc25.iso"
rhel4-64 = "linuxPreGlibc25.iso"
rhel5 = "linux.iso"
rhel5-64 = "linux.iso"
rhel6 = "linux.iso"
rhel6-64 = "linux.iso"
rhel7 = "linux.iso"
rhel7-64 = "linux.iso"
rhel8-64 = "linux.iso"
rhel9-64 = "linux.iso"
rockyLinux-64 = "linux.iso"
sjds = "linuxPreGlibc25.iso"
sles = "linuxPreGlibc25.iso"
sles-64 = "linuxPreGlibc25.iso"
sles10 = "linuxPreGlibc25.iso"
sles10-64 = "linuxPreGlibc25.iso"
sles11 = "linux.iso"
sles11-64 = "linux.iso"
sles12 = "linux.iso"
sles12-64 = "linux.iso"
sles15-64 = "linux.iso"
sles16-64 = "linux.iso"
solaris6 = "solaris.iso"
solaris7 = "solaris.iso"
solaris8 = "solaris.iso"
solaris9 = "solaris.iso"
solaris10 = "solaris.iso"
solaris10-64 = "solaris.iso"
solaris11-64 = "solaris.iso"
suse = "linuxPreGlibc25.iso"
suse-64 = "linuxPreGlibc25.iso"
turbolinux = "linuxPreGlibc25.iso"
turbolinux-64 = "linuxPreGlibc25.iso"
ubuntu = "linux.iso"
ubuntu-64 = "linux.iso"
whistler = "winPreVista.iso"
win2000 = "winPreVista.iso"
win2000AdvServ = "winPreVista.iso"
win2000Pro = "winPreVista.iso"
win2000Serv = "winPreVista.iso"
win31 = "winPre2k.iso"
win95 = "winPre2k.iso"
win98 = "winPre2k.iso"
windows7 = "windows.iso"
windows7-64 = "windows.iso"
windows8 = "windows.iso"
windows8-64 = "windows.iso"
windows9 = "windows.iso"
windows9-64 = "windows.iso"
windows11-64 = "windows.iso"
windows12-64 = "windows.iso"
windows7srv-64 = "windows.iso"
windows8srv-64 = "windows.iso"
windows9srv-64 = "windows.iso"
windows2019srv-64 = "windows.iso"
windows2019srvNext-64 = "windows.iso"
windows2022srvNext-64 = "windows.iso"
winMe = "winPre2k.iso"
winNetBusiness = "winPreVista.iso"
winNetDatacenter = "winPreVista.iso"
winNetDatacenter-64 = "winPreVista.iso"
winNetEnterprise = "winPreVista.iso"
winNetEnterprise-64 = "winPreVista.iso"
winNetStandard = "winPreVista.iso"
winNetStandard-64 = "winPreVista.iso"
winNetWeb = "winPreVista.iso"
winNT = "winPre2k.iso"
winServer2008Cluster-32 = "windows.iso"
winServer2008Cluster-64 = "windows.iso"
winServer2008Datacenter-32 = "windows.iso"
winServer2008Datacenter-64 = "windows.iso"
winServer2008DatacenterCore-32 = "windows.iso"
winServer2008DatacenterCore-64 = "windows.iso"
winServer2008Enterprise-32 = "windows.iso"
winServer2008Enterprise-64 = "windows.iso"
winServer2008EnterpriseCore-32 = "windows.iso"
winServer2008EnterpriseCore-64 = "windows.iso"
winServer2008SmallBusiness-32 = "windows.iso"
winServer2008SmallBusiness-64 = "windows.iso"
winServer2008SmallBusinessPremium-32 = "windows.iso"
winServer2008SmallBusinessPremium-64 = "windows.iso"
winServer2008Standard-32 = "windows.iso"
winServer2008Standard-64 = "windows.iso"
winServer2008StandardCore-32 = "windows.iso"
winServer2008StandardCore-64 = "windows.iso"
winServer2008Web-32 = "windows.iso"
winServer2008Web-64 = "windows.iso"
winVista = "windows.iso"
winVista-64 = "windows.iso"
winXPHome = "winPreVista.iso"
winXPPro = "winPreVista.iso"
winXPPro-64 = "winPreVista.iso"
[root@OneNeck01-5:/vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools]

[LucD]

"... they do not support PowerCLI."

What GSS has told you is not true. Here is the PowerCLI support information from the current PowerCLI user guide: https://developer.vmware.com/docs/15315//GUID-0F835722-E667-4177-A8EF-51AB15FA837E.html. 

[GeoPerkins]

If I perform the same procedure manually in the vCenter web GUI, VMware Tools are updated normally. Seems only to be a PowerCLI problem?

Before:

Update Tools Steps:

After power off/on and automated Tools update (which does a Windows restart):

[LucD]

That is not the same procedure as what the PowerCLI cmdlet does.
Clicking "Upgrade VMware Tools" on that first screen would do the same as the cmdlet.

Since these files are symbolic links, did you also do an 'ls -l' of that folder?
Just to check that the link is correct and that the permissions on the link/file are correct.

[GeoPerkins]

Thanks for the tip on listing symbolic links of files actually elsewhere. When I do that it works fine, so it is not (apparently) a file system permission problem.

[root@OneNeck01-6:/vmfs/volumes/ebc504d1-d7bffd90/productLocker/vmtools] ls -l
total 117228
-rwx------    1 root     root             8 Jun  1  2023 buildNumber.txt
-rwx------    1 root     root          5895 Jun  1  2023 isoimages_manifest.txt
-rwx------    1 root     root          3378 Jun  1  2023 isoimages_manifest.txt.sig
-rwx------    1 root     root             6 Jun  1  2023 version.txt
-rwx------    1 root     root     130721792 Jun  1  2023 windows.iso
-rwx------    1 root     root       2043040 Jun  1  2023 windows.iso.sha
-rwx------    1 root     root          3378 Jun  1  2023 windows.iso.sig
-rwx------    1 root     root          1669 Jun  1  2023 windows_avr_manifest.txt
-rwx------    1 root     root          3378 Jun  1  2023 windows_avr_manifest.txt.sig

[LucD]

Did you check the symbolic links with secpolicytools as described in KB1037405 ?

Also, does the "Upgrade VMware Tools..." button in the GUI work and upgrades the tools?

[GeoPerkins]

I had no reason to expect symbolic links (KB1037405) to be broken, but since you asked, I did follow the first step to confirm they seem intact:

[root@OneNeck01-6:~] ls -la /usr/lib/vmware/isoimages
lrwxrwxrwx    1 root     root            23 Aug  4  2021 /usr/lib/vmware/isoimages -> /productLocker/vmtools/

Yes the "Upgrade VMware Tools..." button in the GUI works.  You mentioned (several posts upward) that the GUI update is the same as what the PowerCLI "Update-Tools" cmdlet is doing? What if the guest VM does not have Tools installed already, then I need to mount the Tools DVD ISO first; but if there are Tools already installed, then I don't need to mount the Tools DVD ISO? Can I just mount the Tools DVD ISO anyway for all cases, so I don't need to program for a null condition?

[LucD]

The Update VMware Tools button will only be there if there are VMware Tools installed.
You can easily check if VMware Tools are installed in a Guest OS.
Based on that test you then use the cmdlet to mount the ISO.

[GeoPerkins · Accepted solution]

I updated a cluster to newer VMware Tools 12.3.5 and I am unable to reproduce the error. The original error may have been unique to a specific guest VM or perhaps unique to the syntax I used in the Update-Tools cmdlet.

Original error Tools 12.2.0 and cmdlet:

Update-Tools -Guest [object]

Working Tools 12.3.5 and cmdlet:

Update-Tools -VM [string]

I am unable to reproduce the error, so closing this thread. Thanks for your help.