Freeze when running PowerCLI cmdlets via invoke-command

[Craig_Baltzer]

I'm trying to run a bit of PowerCLI on a remote server using invoke-command. I'm hitting a wall where the execution "hangs" after the connect-viserver cmdlet executes.
I've increased the memory available for remote processes on that server to 2GB (i.e. I've run set-item wsman:localhost\Shell\MaxMemoryPerShellMB 2048 on the remote server and restarted).

Remote execution works fine as I can do

invoke-command -ComputerName server.bob.com -Scriptblock { get-help}

this returns the output from "get-help" as expected

Likewise I can load the PowerCLI snap-in and connect to a vSphere host

invoke-command -ComputerName server.bob.com -Scriptblock { add-pssnapin VMware.VimAutomation.Core; connect-viserver vsserver.bob.com -User root -Password pwd; get-help}

this returns the output from connect-viserver showing that I connected to the host, then the "get-help" output follows as expected

The burn comes when I try to run another VMware cmdlet

invoke-command -ComputerName server.bob.com -Scriptblock { add-pssnapin VMware.VimAutomation.Core; connect-viserver vsserver.bob.com -User root -Password pwd; get-vmhost}

this returns the output from connect-viserver and then just hangs (I have to close the PowerShell instance to get control back). The cmdlet doesn't matter (i.e. get-vm will also hang)

The underlying OS is Windows 2008 R2 and I've tried both 5.0 (build 435427) as well as 5.5 PowerCLI (build 1295336) with the same results. Any thoughts?

[LucD]

Try suppressing warning messages before you run the Get-VMHost.

Add this line

Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false

[Craig_Baltzer]

Thanks Luc. With that change I now get

Processing data from remote server failed with the following error message: The WSMan provider host process did not return a proper response. A provider in the host process may have behaved improperly.

+CategoryInfo: OperationStopped: (System.Manageme...pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException

+FullyQualifiedErrorID: JobFailure

That got me thinking about whether the PowerCLI returned object(s) were an issue. So a bit more testing

invoke-command -ComputerName server.bob.com -Scriptblock { add-pssnapin VMware.VimAutomation.Core; Set-PowerCLIConfiguration -DisplayDeprecationWarnings $False -Confirm:$False; connect-viserver vsserver.bob.com -User root -Password pwd; get-vmhost}

fails with the error above complaining about an improper response as noted above

invoke-command -ComputerName server.bob.com -Scriptblock { add-pssnapin VMware.VimAutomation.Core; Set-PowerCLIConfiguration -DisplayDeprecationWarnings $False -Confirm:$False; connect-viserver vsserver.bob.com -User root -Password pwd; get-vmhost vssam}

works and returns a standard cmdlet error message saying the host "vssam" doesn't exist

invoke-command -ComputerName server.bob.com -Scriptblock { add-pssnapin VMware.VimAutomation.Core; Set-PowerCLIConfiguration -DisplayDeprecationWarnings $False -Confirm:$False; connect-viserver vsserver.bob.com -User root -Password pwd; get-vmhost | select-object Name, PowerState }

works and returns the host name and its power state

So it looks like the objects being returned by the cmdlets appear invalid to the WSMan provider. Is this a "server-side vs client-side" objects thing? Any thoughts?

[LucD]

Doesn't ring a bell.

Could you include the complete error with the "vssam" mention in it ?

Btw, can you include "Get-ChildItem wsman:\localhost\Shell" in the script you submit this way ?

Just to make sure the MaxMemoryPerShellMB is set.

[Craig_Baltzer]

Thanks Luc. Complete error is

Processing data for a remote command failed with the following error message: The WSMan provider host process did not return a proper response.  A provider in the host process may have behaved improperly. For more information, see the about_Remote_Troubleshooting Help topic.   

+ CategoryInfo          : OperationStopped: (System.Manageme...pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException   

+ FullyQualifiedErrorId : JobFailure

The MaxMemoryPerShellMB was run on the remote system directly (i.e. I RDP'd in and ran the command locally). It can't be included in the invoke-command stream as the remote session doesn't have enough privilege to execute it (gives an "Access Denied"). Doing a "get-item" on it shows that its set to 2048...

[LucD]

And do you see the new value inside the remote session (which you start implicitly with the Invoke-Command cmdlet).

Add the line I mentioned above, just to make sure.

[Craig_Baltzer]

It doesn't look like that can be queried remotely...

Invoke-Command -ComputerName server.bob.com -ScriptBlock { Get-ChildItem wsman:\localhost\Shell } returns

Cannot find path 'localhost\Shell' because it does not exist.   

+ CategoryInfo          : ObjectNotFound: (localhost\Shell:String) [Get-ChildItem], ItemNotFoundException    

+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

Putting quotes around the item (i.e. "wsman:\localhost\Shell") doesn't change the behaviour. Perhaps a permissions issue?

Running the command directly on server.bob.com returns   

WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell

Name                      Value                                                             Type

----                      -----                                                             ----

AllowRemoteShellAccess    true                                                              System.String

IdleTimeout               180000                                                            System.String

MaxConcurrentUsers        5                                                                 System.String

MaxShellRunTime           2147483647                                                        System.String

MaxProcessesPerShell      15                                                                System.String

MaxMemoryPerShellMB       2048                                                              System.String

MaxShellsPerUser          5                                                                 System.String

[LucD]

That seems to work for me. Did you try to another server ?

[Craig_Baltzer]

I've tried a few of our servers. If I do a

invoke-command -ComputerName server2.bob.com { get-childitem wsman:localhost }

I get

Access is denied.

+ CategoryInfo          : NotSpecified: (:) [Get-ChildItem], InvalidOperationException 

+ FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.GetChildItemCommand

so I suspect the remote shell doesn't have enough permissions to read anything down the wsman:localhost path

Are you able to run

Invoke-Command -ComputerName server.bob.com -ScriptBlock { add-pssnapin VMware.VIMAutomation.Core; Set-PowerCLIConfiguration -DisplayDeprecationWarnings $False -Confirm:$False; connect-viserver vsserver.bob.com -User root -Password pwd; get-vmhost }

and get the results back correctly from "get-vmhost"?

[GuilhermeAlves]

That seems to work for me too.. Maybe a permission problem.

Can you connect remotely to your servers using powershell and run the script you want?

[Craig_Baltzer]

Thanks Guilherme. Just to confirm were you able to get the full command (i.e. get results back from get-vmhost run remotely via invoke-command) successfully?

I've seen this problem in two different clients now (prob. looked at 5-6 servers). Remote Powershell works fine (i.e. no issues remotely running other PS scripts (use it to do things like reconfigure disks, setup networks, copy files, import ovas, etc.)), nor is there an issue with PowerCLI cmdlets that don't return PowerCLI objects (i.e. see the previous example where I put "select-object" on the end to avoid returning a PowerCLI object). Its just the PowerCLI cmdlets that return PowerCLI objects that result in the wsman error saying the execution didn't return a proper response.

Which version of Powershell/PowerCLI/OS are you running? Domain joined servers? What privs does the user have that's logged in (i.e. local administrator on the source and remote servers, a domain admin, or just a regular user?). We're on Windows 2008 R2, Powershell 2, and PowerCLI 5.1 Release 2. The boxes are AD domain joined and I'm logged in as a user that is in the local Administrators group (but not a domain admin in AD)...

[GuilhermeAlves]

i'm at the same environment you've mentioned but my powercli version is 5.5 and i'm a domain admin...can you try to update the powercli and get a doman admin to run the commands? only test 😃

[Craig_Baltzer]

I've built a fresh server from the Windows 2008 R2 media, patched it up-to-date, then

• Ran "winrm quickconfig" to configure WinRM
• Started PowerShell and ran "set-item wsman:localhost\Shell\MaxMemoryPerShellMB 2048" to set the available remote session memory to 2GB
• Installed PowerCLI from VMware-PowerCLI-5.5.0-1295336.exe
• Rebooted the server

Note that this server is NOT domain joined so there is no possibility that GPOs are changing any security settings

When I attempt to run

invoke-command -ComputerName 1.1.1.1 -ScriptBlock { add-pssnapin Vmware.VimAutomation.Core; connect-viserver v1.bob.com -User root -Password pwd; set-powercliconfiguration -DisplayDeprecationWarnings $False -Confirm:$False; get-vmhost v1.bob.com } -Credential newserver\administrator

I get back

PSComputerName     : 1.1.1.1
RunspaceId         : 8fb6cc34-00d4-435a-9b8e-c9f3b5f84720
PSShowComputerName : True
IsConnected        : True
Id                 : /VIServer=root@v1.bob.com:443/
ServiceUri         : https://v1.bob.com/sdk
SessionSecret      : 52d10847-1892-d9d3-0d69-5c121571f2c7
Name               : v1.bob.com
Port               : 443
SessionId          : 52d10847-1892-d9d3-0d69-5c121571f2c7
User               : root
Uid                : /VIServer=root@v1.bob.com:443/
Version            : 5.5
Build              : 1474528
ProductLine        : embeddedEsx
InstanceUuid       :
RefCount           : 1
ExtensionData      : VMware.Vim.ServiceInstance

DefaultVIServerMode        : Multiple
ProxyPolicy                : UseSystemProxy
DisplayDeprecationWarnings : False
InvalidCertificateAction   : Unset
WebOperationTimeoutSeconds : 300
VMConsoleWindowBrowser     :
Scope                      : Session

DefaultVIServerMode        :
ProxyPolicy                :
DisplayDeprecationWarnings :
InvalidCertificateAction   :
WebOperationTimeoutSeconds :
VMConsoleWindowBrowser     :
Scope                      : User

DefaultVIServerMode        :
ProxyPolicy                :
DisplayDeprecationWarnings : False
InvalidCertificateAction   :
WebOperationTimeoutSeconds :
VMConsoleWindowBrowser     :
Scope                      : AllUsers

Processing data from remote server failed with the following error message: The WSMan provider host process did not return a proper response.  A provider in the host process may have behaved improperly. For more information, see the about_Remote_Troubleshooting Help topic.
    + CategoryInfo          : OperationStopped: (System.Manageme...pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRemotingTransportException
    + FullyQualifiedErrorId : JobFailure

This problem only occurs with certain cmdlets, notibly "get-vmhost" and "get-vm". For example

invoke-command -ComputerName 1.1.1.1 -ScriptBlock { add-pssnapin Vmware.VimAutomation.Core; connect-viserver v1.bob.com -User root -Password pwd; set-powercliconfiguration -DisplayDeprecationWarnings $False -Confirm:$False; get-harddisk -VM C1 } -Credential newserver\administrator

Then I get back a good set of results as shown below.

PSComputerName     : 1.1.1.1
RunspaceId         : 9dbd8d12-8a63-4629-8bf4-b7b8ad9f432a
PSShowComputerName : True
IsConnected        : True
Id                 : /VIServer=root@v1.bob.com:443/
ServiceUri         : https://v1.bob.com/sdk
SessionSecret      : 52b8ce1d-7b59-a76b-80da-97f910a7fd83
Name               : v1.bob.com
Port               : 443
SessionId          : 52b8ce1d-7b59-a76b-80da-97f910a7fd83
User               : root
Uid                : /VIServer=root@v1.bob.com:443/
Version            : 5.5
Build              : 1474528
ProductLine        : embeddedEsx
InstanceUuid       :
RefCount           : 1
ExtensionData      : VMware.Vim.ServiceInstance

DefaultVIServerMode        : Multiple
ProxyPolicy                : UseSystemProxy
DisplayDeprecationWarnings : False
InvalidCertificateAction   : Unset
WebOperationTimeoutSeconds : 300
VMConsoleWindowBrowser     :
Scope                      : Session

DefaultVIServerMode        :
ProxyPolicy                :
DisplayDeprecationWarnings :
InvalidCertificateAction   :
WebOperationTimeoutSeconds :
VMConsoleWindowBrowser     :
Scope                      : User

DefaultVIServerMode        :
ProxyPolicy                :
DisplayDeprecationWarnings : False
InvalidCertificateAction   :
WebOperationTimeoutSeconds :
VMConsoleWindowBrowser     :
Scope                      : AllUsers

PSComputerName     : 1.1.1.1
RunspaceId         : 9dbd8d12-8a63-4629-8bf4-b7b8ad9f432a
PSShowComputerName : True
StorageFormat      : Thin
Persistence        : Persistent
DiskType           : Flat
Filename           : [P2000-vd01-v001] C2/C2.vmdk
CapacityKB         : 41943040
CapacityGB         : 40
ParentId           : VirtualMachine-10
Parent             : C2
Uid                : /VIServer=root@v1.bob.com:443/VirtualMachine=VirtualMachine-10/HardDisk=2000/
ConnectionState    :
ExtensionData      : VMware.Vim.VirtualDisk
Id                 : VirtualMachine-10/2000
Name               : Hard disk 1

PSComputerName     : 1.1.1.1
RunspaceId         : 9dbd8d12-8a63-4629-8bf4-b7b8ad9f432a
PSShowComputerName : True
StorageFormat      : EagerZeroedThick
Persistence        : Persistent
DiskType           : Flat
Filename           : [P2000-vd01-v001] C1/C1_1.vmdk
CapacityKB         : 1048576
CapacityGB         : 1
ParentId           : VirtualMachine-10
Parent             : C2
Uid                : /VIServer=root@v1.bob.com:443/VirtualMachine=VirtualMachine-10/HardDisk=2016/
ConnectionState    :
ExtensionData      : VMware.Vim.VirtualDisk
Id                 : VirtualMachine-10/2016
Name               : Hard disk 2

PSComputerName     : 1.1.1.1
RunspaceId         : 9dbd8d12-8a63-4629-8bf4-b7b8ad9f432a
PSShowComputerName : True
StorageFormat      : EagerZeroedThick
Persistence        : Persistent
DiskType           : Flat
Filename           : [P2000-vd01-v001] C1/C1_2.vmdk
CapacityKB         : 5242880
CapacityGB         : 5
ParentId           : VirtualMachine-10
Parent             : C2
Uid                : /VIServer=root@v1.bob.com:443/VirtualMachine=VirtualMachine-10/HardDisk=2017/
ConnectionState    :
ExtensionData      : VMware.Vim.VirtualDisk
Id                 : VirtualMachine-10/2017
Name               : Hard disk 3

So I don't believe I'm chasing a general "configuration" issue but rather something that's a bit off in the way a few of the cmdlets return data...

[GuilhermeAlves]

Well, try to get back some other information like the observer connection as a string. That way we'll test if the invoke-command is not returning variables Like I had this problem in past.

Try returning like this:

$vc=connect-viServer ...

$vc | out-string

Or then out-variable and then try to see if your variable is properly setted

--- Mensagem Original ---

De: "Craig Baltzer" <communities-emailer@vmware.com>

Enviado: 21 de março de 2014 23:45

Para: "GuilhermeAlves" <guilhermestela@hotmail.com>

Assunto: New message: "Freeze when running PowerCLI cmdlets via invoke-command"

VMware Communities<https://communities.vmware.com/index.jspa>

Freeze when running PowerCLI cmdlets via invoke-command

created by Craig Baltzer<https://communities.vmware.com/people/Craig+Baltzer> in VMware vSphere™ PowerCLI - View the full discussion<https://communities.vmware.com/message/2360017#2360017>

[Craig_Baltzer]

I've added a bit more information to post #12. It looks like only certain cmdlets return the malformed results So far

• Causes the WSMan provider host process did not return a proper response error
  • get-vmhost
  • get-vm
  • get-template
  • get-vmhoststorage

• Return valid data (work correctly)
  • Get-harddisk
  • Get-View
  • get-vmhostnetwork
  • get-vmhostnetworkadapter
  • get-datastore
  • get-folder
  • get-virole
  • get-vipermission
  • get-vmhostservice

[Craig_Baltzer]

Thanks Guilherme. Manipulating the output from the cmdlet will return valid results (see post #2, the 3rd example where "get-vmhost | select-object Name, PowerState" was shown to work correctly and return the Name and Power state from get-vmhost. Out-String also works of course.

The puzzling thing is a large number of cmdlets to work fine with invoke-command without doing anything "extra" (get-view, get-vmhostnetwork, get-vmhostnetworkadapter, get-harddisk all return valid results as expected) its just get-vm and get-vmhost that I've found so far to be causing the errors

[GuilhermeAlves]

Very glad to know that!

At Monday I'll test a bit more to try to know why we're not receiving complete outputs.

I have experienced days before a similar problem and I solve my problem using out-string. Instead using invoke, I've used start-job, by both are similar cmdlets too.

For now let's keep testing!

--- Mensagem Original ---

De: "Craig Baltzer" <communities-emailer@vmware.com>

Enviado: 22 de março de 2014 00:24

Para: "GuilhermeAlves" <guilhermestela@hotmail.com>

Assunto: New message: "Freeze when running PowerCLI cmdlets via invoke-command"

VMware Communities<https://communities.vmware.com/index.jspa>

Freeze when running PowerCLI cmdlets via invoke-command

created by Craig Baltzer<https://communities.vmware.com/people/Craig+Baltzer> in VMware vSphere™ PowerCLI - View the full discussion<https://communities.vmware.com/message/2360128#2360128>

[GuilhermeAlves]

Did you managed your script to work Craig?

[Craig_Baltzer]

I've managed to get what I wanted to get done by using "Select-Object" at the end of the ScriptBlock pipeline to mask off the issues with the objects returned by cmdlets like Get-VM and Get-VMHost. While not the best, it does work.

When I get some time I'll open a support case on this and see if I can get support to repro the problem and and if so log it as a bug against those particular PowerCLI cmdlets.

Guilherme and Luc, thanks again for the ideas on workarounds!

[GuilhermeAlves]

Nice! Let me know if you find some answer. I'll give a try when I could. 😃

--- Mensagem Original ---

De: "Craig Baltzer" <communities-emailer@vmware.com>

Enviado: 27 de março de 2014 22:45

Para: "GuilhermeAlves" <guilhermestela@hotmail.com>

Assunto: New message: "Freeze when running PowerCLI cmdlets via invoke-command"

VMware Communities<https://communities.vmware.com/index.jspa>

Freeze when running PowerCLI cmdlets via invoke-command

created by Craig Baltzer<https://communities.vmware.com/people/Craig+Baltzer> in VMware vSphere™ PowerCLI - View the full discussion<https://communities.vmware.com/message/2362569#2362569>