Invoke-VMScript failed when guestcredential is not provided

[zealyahweh]

We have some PowerShell script to manage servers,

recently some of them do not work on newly deployed servers.

Here is any sample:

$script = "hostname"

Invoke-VMScript -vm MyServer -ScriptText $script

this used to work good and still working on old servers, the script will use current windows user credential.

but when running on newly deployed servers. We got following error:

Invoke-VMScript Error occurred while executing script on guest OS in VM 'MyServer'. Could not locate "Powershell" script interpreter in any of the expected locations. Probably

you do not have enough permissions to execute command within guest.

We have a workaround like this, but it needs user input and we always try to make things fully automatic

$Creds = Get-Credential "$env:USERDNSDOMAIN/$env:USERNAME"

$script = "hostname"

Invoke-VMScript -vm MyServer -ScriptText $script  -GuestCredential $Creds

please help, thank you

Here is some log:

[2019-09-03T07:44:28.017Z] [   debug] [vmsvc] RpcIn: received 624 bytes, content:"Vix_1_Relayed_Command "56e4594565a63349"\00\01\00\0d衆05\00G\02\00\003\00\00\00郳01\00\004\00\00\00\0d筡00\00\00\00\00\00\00..."

[2019-09-03T07:44:28.017Z] [ message] [vix] VixTools_ProcessVixCommand: command 185

[2019-09-03T07:44:28.017Z] [   debug] [vix] VixToolsImpersonateUser: successfully impersonated user MyName

[2019-09-03T07:44:28.017Z] [   debug] [vix] VixTools_StartProgram: User:  MyName args: progamPath: 'cmd.exe', arguments: '/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAaABvAHMAdABuAGEAbQBlAH0AJwAgAD4AIAAiAEMAOgBcAFUAcwBlAHIAcwBcAHoAaABnAG8AbgBnAC0AYQBcAEEAcABwAEQAYQB0AGEAXABMAG8AYwBhAGwAXABUAGUAbQBwAFwAcABvAHcAZQByAGMAbABpAHYAbQB3AGEAcgBlADIANwAiADsAIABlAHgAaQB0ACAAJABsAGEAcwB0AGUAeABpAHQAYwBvAGQAZQA=', workingDir: ''

[2019-09-03T07:44:28.017Z] [ message] [VCGA] Pref_Init: Using 'C:\ProgramData\VMware\VMware VGAuth\vgauth.conf' as preferences filepath

[2019-09-03T07:44:28.017Z] [   debug] [VCGA] Pref_GetBool: Pref_GetBool(auditSuccessEvents) failed: Key file does not have group 'auditing'

[2019-09-03T07:44:28.017Z] [   debug] [VCGA] I18n_BindTextDomain: user locale=en_US

[2019-09-03T07:44:28.017Z] [ message] [VCGA] VGAuth 'build-12385171' initialized for application 'vmtoolsd'.  Context created at 000001F59C31E6B0

[2019-09-03T07:44:28.017Z] [   debug] [vix] VixToolsTranslateVGAuthError: translated VGAuth err 0x2 to Vix err 3

[2019-09-03T07:44:28.017Z] [ warning] [vix] VixToolsStartProgramImpl: Failed to get user token

[2019-09-03T07:44:28.017Z] [   debug] [vix] GuestAuthUnloadUserProfileAndToken: special-case profile unload 0000000000000000

[2019-09-03T07:44:28.017Z] [ warning] [vix] GuestAuthUnloadUserProfileAndToken: UnloadUserProfile() failed 6

[2019-09-03T07:44:28.017Z] [   debug] [vix] VixTools_StartProgram: returning '-1'

[2019-09-03T07:44:28.017Z] [ message] [vix] VixTools_StartProgram: opcode 185 returning 3

[2019-09-03T07:44:28.017Z] [   debug] [vix] ToolsDaemonTcloReceiveVixCommand: command 185, additionalError = 0

[2019-09-03T07:44:28.017Z] [   debug] [vmsvc] RpcIn: sending 10 bytes

[2019-09-03T07:44:28.314Z] [   debug] [vmsvc] RpcIn: received 206 bytes, content:"Vix_1_Relayed_Command "56e45946ba7551cf"\00\01\00\0d衆05\0000\00\003\00\00\00>\00\00\004\00\00\00\05耚00\00\00\00\01\00\00..."

[2019-09-03T07:44:28.314Z] [ message] [vix] VixTools_ProcessVixCommand: command 194

[2019-09-03T07:44:28.314Z] [   debug] [vix] VixToolsImpersonateUser: successfully impersonated user  MyName

[2019-09-03T07:44:28.314Z] [   debug] [vix] VixToolsDeleteObject: User: MyName path: 'C:\Users\ MyName\AppData\Local\Temp\powerclivmware27'

[2019-09-03T07:44:28.314Z] [ message] [vix] VixToolsDeleteObject: opcode 194 returning 0

[2019-09-03T07:44:28.314Z] [   debug] [vix] ToolsDaemonTcloReceiveVixCommand: command 194, additionalError = 0

[2019-09-03T07:44:28.314Z] [   debug] [vmsvc] RpcIn: sending 8 bytes

[LucD]

I assume the guest OS in those VM is a Windows OS, but which specific version?

And which PowerCLI version are you using?

Or is the guest OS a Linux distro?

[zealyahweh]

Thank you for the quick reply,

Guest OS is Windows, from 2008R2 to 2016, all with latest windows security patch.

I can find the issue on both 2012 and 2016, but only new server have the issue.

I don't have too much test but right now I can see all server with issue have ToolsVersion : 10.3.10, when no issue servers are 10.3.5.

This may could explain why only new servers are affected, we only upgrade VM Tools in template.

Powershell version is 5.1.14409.1018.

Will go to try upgrade a server from 10.3.5 to 10.3.10

[zealyahweh]

Now I quite believe this is a VMWare Tools BUG(or by design for security?)

I ran the script against an old server,

then upgrade VMware tools to 10.3.10 and run again.

Here is the result:

PS C:\User\Desktop> C:\Users\MyName\Desktop\test.ps1

Name                           Port  User                        

----                           ----  ----                        

VC                           443   MyDomain\MyName         

VM           : TestServer

ExitCode     : 0

ScriptOutput : TESTSERVER

Uid          : /VIServer=MyDomain\MyName@TestServer:443/VirtualMachine=VirtualMachine-vm-34/VMScriptResult=-157373385_0/

Length       : 17

Name         : TestServer

ToolsVersion : 10.3.5

PS C:\Users\MyName\Desktop> C:\Users\MyName\Desktop\test.ps1

Name                           Port  User                        

----                           ----  ----                        

VC                           443   MyDomain\MyName    

Invoke-VMScript : 9/4/2019 11:03:36 AM Invoke-VMScript Error occured while executing script on guest OS in VM 'TestServer'. Could not locate "Powershell" script

interpreter in any of the expected locations. Probably you do not have enough permissions to execute command within guest.

At C:\Users\MyName\Desktop\test.ps1:10 char:1

+ Invoke-VMScript -vm $a -ScriptText $script  #-GuestCredential $oCreds

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ResourceUnavailable: (TestServer:UniversalVirtualMachineImpl) [Invoke-VMScript], VimException

    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_RunScriptCore_ExeLookupFailed,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

Name         : TestServer

ToolsVersion : 10.3.10

Code:

$a = Get-VM -Name Testserver

#$oCreds = Get-Credential

$script = "hostname"

Invoke-VMScript -vm $a -ScriptText $script  #-GuestCredential $oCreds

$a | Select-Object -Property Name,@{Name='ToolsVersion';Expression={$_.Guest.ToolsVersion}}

[LucD]

Thanks for sharing that info.

Did you open an SR?

Could you try adding the following line, before calling Invoke-VMScript?

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls11,[Net.SecurityProtocolType]::Tls12,[Net.SecurityProtocolType]::Tls13

[zealyahweh]

It doesn't work, in fact, I don't think our system is supporting TLS1.3, at least, I don't see it in SCHANNEL.

For SR, do you mean service request?

[LucD]

Yes, a service request.

[GG81BDC]

hi - any resolution on this? I hit the same issue. Anything under 10.3.5 - invoke-vmscript works with no explicit credentials (it uses the credentials that "runs" the script - like an SSO/pass-through), but any VM that has 10.3.10+ (11.0.0 behaves the same) needs explicit credentials

[Goatie]

We've had the same issue here too. Using passthrough authentication when running scripts inside VMs now fails when using the latest releases of 6.5 and 6.7 ESXi. Specifying the same/different credentials using -GuestCredential works fine inside Invoke-VMScript.

Has anyone came across a cause for this or a way to turn it off?

Edit:

I've done some more testing and Vmware Tools 10.3.5 works fine (both ESXi 6.5 and 6.7), but when upgrading to both VMware Tools 11.0.1 or 11.0.5. We've tested using VMware PowerCLI 11.3.0 and 11.5.0 and Server 2008R2, 2012R2 and 2016 with the same results. Also tested were PowerShell and Bat commands, through the -ScriptType parameter.

if you provide credentials via -GuestCredential it works fine, but if you use pass-through authentication it breaks. Using pass-through authentication is good as the script runs as SYSTEM and is able to do low-level OS changes that are not possible when running the script whilst providing credentials.

Does anyone know if this issue is being worked on?

[ThomasBiesmans]

Same here as well... Reproducibly stopped working after an upgrade from build 10246 to build 11265, unless explicit credentials are used. Such a lovely regression...

[Goatie]

Does anyone have any news on this issue? Does anyone know if it been resolved in any subsequent releases of ESXi/PowerCLI?

[Goatie]

We've fully resolved the issue by upgrading to VMware Tools v11.1.0.

VMware Tools 11.1.0 Release Notes

You'll need to set up a VMware Tools repository to deploy it as 11.1.0 isn't shipping with ESXi releases yet.

https://blogs.vmware.com/vsphere/2019/01/configure-a-vmware-tools-repo-in-vsphere-6-7u1.html

This repository is only available in vSphere 6.7u1+, so you'll need to use Update Manager or some other means to deploy the updated tools in earlier versions.