If you check the link in the answer marked as "correct" you will find what cause the problem for me and somebody else.
Essentially, when the invoke-vmscript has to download the result, despite you are connected to the vcenter ,it is trying to connect to the ESX host using the IP address (not the FQDN) and in my case was failing because the certificate doesn't contain the ip address in the subject alternative name.
So I got two options: re-issue the certificate including the FQDN or disable the check of the certificate. You can find the workaround in the other thread.
Thank you for your response. If it is the issue with certificate and the getting the output to the caller of the invoke script, then why I don't see any activity in the VMWare logs? We should have something logged there, isn't it?
Why would you expect something in any of the VMware logs?
The ESXi node just passes the certificate along.
When the caller (your script) can't verify that certificate with the CA, the problem is encountered on the client side.
You can try to do the following to one of the ESXi nodes that demonstrates the error.
You should get a message saying something along the lines of "Could not establish trust relationship for the SSL/TLS secure channel."
Invoke-WebRequest -Method Get -Uri https://ESXi-node-FQDN
When you do the same from PowerShell v6, the message is clearer.
It says "The remote certificate is invalid according to the validation procedure."
And in v6 you have the option SkipCertificateCheck, which will make the call work.
Invoke-WebRequest -Method Get -Uri https://ESXi-node-FQDN -SkipCertificateCheck
I may be missing some thing here. The solution says that we have an issue while downloading the results, which means the command should have got executed by then, so I expect that there must be some activity in the VMWare logs on the guest machine. Isnt it?
Sorry but you said in your previous comment that you run into the same issue so I guess Lucd was explaining you that the log doesn't matter....
Also I think to see something in the log about the execution of the invoke-vmscript you have to enable some more verbose level.
Now, here the question:
Despite the error you got, does your command being executed? If you run something like "echo hello >c:\temp\test.txt does it work?
If not you probably have a different problem that need to be investigate.
The Invoke-VMScript cmdlet comprises several steps: simplified it consists of creating temp files in the guest OS, running the command in the guest OS from one of those file and storing the output of the command in another temp file, and finally downloading the result.
It is in that last stage, the download of the temporary file, with output from the command, that things go wrong.
This operation fails due the invalid certificate, and that is due to the fact because the internals need to use a web function Get from the ESXi node to receive that file.
Contrary to most of the other steps which use some of the GuestOperations methods.
You can simulate this retrieval from the ESXi node by doing a Get-WebRequest, with the errors I have shown before.
An that kind of failure is not recorded in any of the VMware logs afaik.
I can confidently say that I am getting the same error as follows,:
Invoke-VMScript - An error occurred while sending the request
One thing I observed was it takes sonetiso , around 5 minutes to come out of the command execution and error out.
I suspect that my command is not getting executef.
I am not just mentioning about failures in the log file.
I assumed that there will be some logging activity if a command is issued using invoke-vmscript.
To check if your command is executed in the guest, you could use the method Chris just proposed.
The availability of log entries depends on where things go wrong.
If there is an issue with any of the methods in GuestOperations, you could activate debug logging for the VMware Tools.
That it takes 5 minutes could probably be due to some timeout somewhere.
Could be the cmdlet waiting for the output file to be downloaded.
A network trace should be able to tell you at what stage the problem occurs.
As a general remark; you seem to forget that this is a forum where users try to help other users, this is not VMware Global Support.
If you feel that there is a bug or a problem in the cmdlet, you should open a SR.
That is, after all, why you pay for a Support Contract.
Right now I am not in a position to say if it is a bug.
I am trying to figure out where I am going wrong with the command execution.
As I mentioned in my first comment,
I have two ESXi clusters managed by one VCenter. The invoke-vmscript is working well with one cluster where I was able to execute and get output. And I have the problem with VMs hosted on the other ESXi cluster, and recieve this error. I turned on the debug logging, but I don't see anything obvious in the logs when I run invoke-vmscript.
Hence my confusion and not sure where to look at.
The tcpdump or network trace does indicate that the command is going outside the client machine to VCenter,. As you can see I was able to successfully execute this command for one ESXi cluster.
In my opinion, this may not be a bug with vMWare. Maybe an issue with configuration. I am trying to see what else I can do to identify the issue.
Did you try to run a test command as in my previous post? that would tell us if the command has been executed or not and where to look.
Anyway a part from the vmware.log file when you use the invoke-vmscript you can see some events in the vcenter ( select the VM, go to monitor and event ) like "Guest Operation..." so you can also check there. Also another test you can do is if the command copy-vmguestfile gives you a similar error or not. This command works very similar to invoke-vmscript for downloading or uploading files to the guest vm. Again you can see with a process monitor if the command is doing something or not.
Then last bit I would like to suggest is to you Lucd Invoke-VMScriptPlus that in verbose mode give you a better understing of what is going on and where the process fails.
Do all these try and let us know.
Sorry. We took down the ESXi setup for general maintenance. Trying to bring it up. We will try as you suggested as soon as possible and share the results.
There was a firewall issue between our primary site and secondary one. After fixing those issues, we are able to invoke the script successfully.
Thanks for all your help.