PowerCLI Invoke-VMScript Sporadic Execution

[levans01]

Issue Details: 

1. We use PowerCLI to get values from VMs.  There are instances where the values fail to return.  It is our belief that this worked flawlessly before and we can show that it still does in some instances.  We developed a test script to reproduce the problem as well as show instances where things appear to work correctly.  This script connects to a machine we chose.  We provide administrative credentials and the script returns the value 12345.  It does ten iterations of this task.  We expect 10/10. 

$creds = Get-Credential

$successes = 0; while ($true) {

$output = Invoke-VMScript -VM "MyVM" -ScriptText "return '12345'" -GuestCredential $creds

$output

if ($output.ScriptOutput) {$successes++}

if ($successes -eq 10) {break}

}

When we run this script against some servers it works flawlessly.  That is the return is 10/10.  When we look at the VM > Tasks & Events > Events  we see the following successive events which are normal.

(1) Guest operation Create Temporary File performed.

(2) Guest operation Start program performed.

(3) Guest operation List processes Performed.

(4) Guest operation List processes Performed.

(5) Guest operation Initiate File Transfer from Guest Perform.

(6) Guest operations Delete File performed.

The system repeats these events for each successful iteration.

2.  Our test script will make as many iterations as possible to get 10 successful results.   Ideally, it would only have to make 10 attempts to get 10 results.  But when there are failures, we have to make more than 10 attempts to get the desired results. I have seen it take as many as 19 attempts to get 10 successes.

3. As an additional data point, When running this script against the domain controller in an environment it works flawlessly.  However, when running against a domain joined machine in the domain, the problem happens. If I watch the creation of the Powershell## file in the temp directory on the system.  The ones that copy with 0 bytes are the ones that fail. 

Any ideas?

Thanks!

[LucD]

You didn't mention which PowerShell (on both sides) and which PowerCLI version you are using.

I haven't seem that behaviour yet.
On a side note, I would be very much interested to know if you see similar when you use my Invoke-VMScriptPlus function.

[levans01]

Good Morning,

My PowerCLI Version is: PowerCLI 6.5 Release 1 build 4624819   the output of $PsVersionTable on the target VMs is:

PSVersion 4.0

WSManStackVersion 3.0

SerializationVersion 1.1.0.1

CLRVersion 4.0.30319.42000

BuildVersion 6.3.9600.19170

PSCapatibleVersion {1.0, 2.0, 3.0, 4.0}

RSremotingProtocolVersion 2.2

[LucD]

Is there a specific reason why your are using (somewhat) older PowerShell and PowerCLI versions?

[levans01]

I manage a very large environment and have change controls in place.  If we think upgrading to the latest version of PowerCLI and and Powershell will permit an defined system function to work correctly it is worth a shot.  However, that would suggest to me that there is a bug in the software.  I have situations when I run the commands against a domain controllers and it works flawlessly.  But when I run the commands against domain joined machines the results get sporadic. (The problem appears to be particularly specific).

[LucD]

I reran your test several times against a DC and a member server, and I always get 10/10.

Admittedly, I ran the tests with PowerCLI 11.0.0 and PowerShell 5.1.

I'm not questioning that this potentially could be a PowerCLI bug in 6.5 R1, but to make sure, I would suggest you open a SR.

On a side note, I'm still interested to know if you see the same behaviour with my Invoke-VMScriptPlus function.

[levans01]

Understood.  Thanks for the feedback.  I can try different versions in my test environment provided I can reproduce the the problem with the older versions that is, the ones I am using in my production.  How can I use Invoke-VMScriptPlus​?  What do have to do to use it on my system?

Thanks! 

[LucD]

Well, the part on my blog that is listed under Code (the Class and the function) can be placed in a .ps1 file.

Since I made the parameters of my function compatible with the original Invoke-VMScript cmdlet, you should be able to just call the function with the same code you have been using.

Just replace Invoke-VMScript with Invoke-VMScriptPlus.

Place your lines also in that .ps1 file.

Then just run the .ps1 file.

There are a couple of sample calls in the blog post, including one for a Windows guest OS.

Let me know if you get it work, if not, just post any issues you might have in this thread.

[levans01]

LucD,

Thanks for all of your help.  I was able to work with a team member to get your script to run inside of Powershell on my machine.  I can report that is pretty much works flawlessly.  My PowerCLI script has a much higher failure rate when getting results from non-domain controllers. Your script failed a couple of times when running iterations against member servers.  The error returned was File : <Path.PS1> cannot be loaded.  The file <FileName> is not digitally signed.  You cannot run this script on the current system.  For more information about running scripts and settings execution policy, see about_Execution-Policies at http://go.microsoft.com/fwlink/?LinkID=135170.  - This only happened twice during multiple attempts.  Have you ever seen this before?  I do not think the problem is a script error but something in the guest perhaps.  I have screenshots.

[LucD]

Thanks for the feedback.

Most probably the PowerShell ExecutionPolicy inside that guest OS has a setting of AllSigned or RemoteSigned.

The function can sign the script that is being created inside the guest OS, but it would need a Certificate for that.
I'll check how I can add a parameter to allow for a Certificate to sign the script.

Can you check with your Windows admins if my assumption about the ExecutionPolicy is correct?

And if yes, do they allow you to use a certificate for signing scripts?

You would need to have public access to the certificate.

[levans01]

LucD,

In our troubleshooting, we disabled the policy.  It did not change the results for our script.  And to be clear,  your script was able to grab several results over multiple runs against the same servers and only sporadically missed two.  We are running our test script that attempts to get 10 sets of 12345 from the server.  In one of our test runs, your script missed on 4th attempt out of 10 with the error previously noted against the same server.  And in another instance, on the 10th attempt. Subsequent runs were without error.

[LucD]

Perhaps you could enable Script Tracing and Logging?

It does require a GPO setting I'm afraid.

And then after a failure you could check if there is anything in the eventlog.

You can do that with submitting something like this after a failure (or of course remotely by using the ComputerName parameter on Get-WinEvent).

$eventFilter = @{

    Logname = 'Microsoft-Windows-PowerShell/Operational'

    StartTime = (Get-Date).AddHours(-1)

}

Get-WinEvent -FilterHashtable $eventFilter