Invoke-VMScript returning 500 (Internal Server Error) , on VM which is attached to domain

[MSpriya]

Hi @LucD , 

I am running into an issue where Invoke-VMScript command is returning 500 (Internal Server Error) on VM that is attached to domain. On the guest VM which is not attached to domain, the command executes successfully. 

Can you please point me to the domain policies which may cause this issue? 

I have verified that firewalls are disabled and port 902 is not blocked. 

PowerCLI version used is 12.0.

Thanks,

Priya 

[LucD]

How do you call Invoke-VMScript?
Which vSphere version?
Do the ESXi certificates contain the FQDN and the IP address of the ESXi node?

[MSpriya]

Thanks @LucD for the quick response. 

Invoke-VMScript is invoked through PowerShell as follows: 

Invoke-VMScript -VM $vm -ScriptText "mkdir c:\temp_Dir" -ScriptType Bat -GuestUser <user> -GuestPassword <password>

vSphere version is 6.7

I shall get back to you on certificates query. 

[LucD]

Are you getting that error with any command in ScriptText or only with that 'mkdir'?

[IT_pilot]

And when you go under this Account, the folder does not say that you need to have administrator rights?

[MSpriya]

Hi @LucD , the same error is seen with any command execution, not specific to mkdir. 

[MSpriya]

@IT_pilot , I am running the command as Administrator only. As I mentioned in previous comment, the issue is seen with any command executed on the guest VM using Invoke-VMScript.

[LucD]

You might want to have a look in the hostd.log on the ESXi node where the targetted VM is running.
There might be some more info available there.

You can also have a look in the VMware Tools logs ("%windir%\temp\vmware-<service>.log") on the VM itself.

[MSpriya]

Hi @LucD ,

I am still facing this issue. Invoke-VMScript command still returns 500 (Internal Server Error).

The command execution is successful inside the guest VM, but the results are not propagated back to the caller. I have verified this by providing a simple mkdir command to create a directory and directory creation is successful inside the guest VM.

The ESXi host which is hosting this VM, has other VMs hosted. Invoke-VMScript is executed without any errors on another Windows 7 guest VM in the same ESXi host. So I dont think there is any issue with port 902 in the caller machine and ESXi host.

The hostd.log on the ESXi host contains the following messages at the time of Invoke-VMScript failure:

2021-08-03T12:38:35.417Z info hostd[2099813] [Originator@6876 sub=Guestsvc.GuestFileTransfer] StoreAndReturnToken: Generated token: 52635d25-3ed0-c42f-61ee-b2132abd0ae5269

2021-08-03T12:38:35.418Z info hostd[2099878] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 141716 : Guest operation Initiate File Transfer From Guest performed on Virtual machine _recovery.

2021-08-03T12:38:35.418Z info hostd[2099878] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/e95cb7b7-d7759c98//.vmx] State Transition (VM_STATE_GUEST_OPERATION -> VM_STATE_ON)

2021-08-03T12:38:35.418Z info hostd[2099891] [Originator@6876 sub=Guestsvc.GuestFileTransfer] Entered VmPowerStateListener 2021-08-03T12:38:35.418Z info hostd[2099891] [Originator@6876 sub=Guestsvc.GuestFileTransfer] VmPowerStateListener succeeded 2021-08-03T12:38:35.418Z info hostd[2099891] [Originator@6876 sub=Hbrsvc] Replicator: powerstate change VM: 269 Old: 1 New: 1 2021-08-03T12:38:35.818Z info hostd[2099253] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters, token is 52635d25-3ed0-c42f-61ee-b2132abd0ae5269

2021-08-03T12:38:35.818Z info hostd[2099253] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters: vmmoid is 269

2021-08-03T12:38:35.818Z info hostd[2099253] [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is C:\Users\ADMINI~1\AppData\Local\Temp\powerclivmware167

2021-08-03T12:38:35.818Z info hostd[2099876] [Originator@6876 sub=Libs] HGFileCopy_TransferFileUsingReader: Error, cannot convert C:\Users\ADMINI~1\AppData\Local\Temp\powerclivmware167 to CPName

2021-08-03T12:38:35.819Z info hostd[2099876] [Originator@6876 sub=Libs] Vix: [vmGuestOps.cpp:5657]: Error VIX_E_FAIL in CopyFileDoneCallback(): Unknown error

2021-08-03T12:38:35.819Z error hostd[2099876] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() job failed with 22

2021-08-03T12:38:35.819Z error hostd[2099876] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() throwing InternalServerError

VMware Tools log contains the following and I dont see any errors in it:

[2021-07-21T10:14:18.633Z] [ message] [vix] VixTools_ProcessVixCommand: command 188

[2021-07-21T10:14:18.664Z] [ message] [vix] VixToolsInitiateFileTransferFromGuest: opcode 188 returning 0

...

[2021-07-21T10:14:20.199Z] [ message] [vix] VixToolsInitiateFileTransferFromGuest: opcode 188 returning 4

[2021-07-21T10:14:20.246Z] [ message] [vix] VixTools_ProcessVixCommand: command 188

With your Invoke-VMScriptPlus script (with SkipCertificateCheck), it fails as follows:

VERBOSE: 03/08/2021 14:53:27 Get-View Finished execution

VERBOSE: 03/08/2021 14:53:27 Get-View Finished execution

VERBOSE: 03/08/2021 14:53:27 Get-View Finished execution

VERBOSE: 03/08/2021 14:53:27 Get-View Finished execution

VERBOSE: No GuestOSType value provided. Trying to determine now.

VERBOSE: It's a Windows guest OS

VERBOSE: Created temp folder in guest OS C:\Users\ADMINI~1\AppData\Local\Temp\Administrator_5616vmware152

VERBOSE: Created temp script file in guest OS C:\Users\ADMINI~1\AppData\Local\Temp\Administrator_5616vmware152\Administrator_5616vmware5.cmd

VERBOSE: Created temp output file in guest OS C:\Users\ADMINI~1\AppData\Local\Temp\Administrator_5616vmware152\Administrator_5616_outputvmware75

VERBOSE: Created Data to C:\Users\ADMINI~1\AppData\Local\Temp\Administrator_5616vmware152\Administrator_5616vmware5.cmd VERBOSE: PUT https://<host-ip>/guestFile?id=<id> with -1-byte payload

Invoke-WebRequest : 22 The file name is not valid

At char: C:\Users\Administrator.Domain\Invoke-VMScriptPlus.ps1:240 : 15

+ $copyResult = Invoke-WebRequest @sWeb

+

+ CategoryInfo : InvalidOperation : (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException

+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

ScriptText copy failed!Status

At char: C:\Users\Administrator.Domain\Invoke-VMScriptPlus.ps1:242 : 1

I have also tried the following:

1. Reinstalled VMware Tools on the guest VM

2. Guest OS and Guest Family are set correctly on the guest VM

3. Firewall is disabled on the Guest VM

4. The error is the same when guest VM is attached to domain or not. 

5. UAC is set to the lowest level

Any idea on root cause of this issue?

[LucD]

One option is to use the KeepFiles switch with my Invoke-VMScriptPlus function.
It will not delete the files inside the target VM's Guest OS.
This allows you to check these files, if they exist and what is their content.

[MSpriya]

@LucD , ok will try with that option. 

However, with Invoke-VMScript I could confirm that command is getting executed within the Guest VM. Still the caller gets 500 Internal Server Error and the results of command execution are not received by the caller. 

Any idea why this may happen? 

Also Invoke-VMScript command execution is successful on another Windows 7 VM hosted in the same ESXi server. 

[LucD]

Not really, that is why I suggested keeping the temporary files, and checking if their existence and content can shed any light.

[MSpriya]

ok, thank you.