VMware Cloud Community
jonathank71
Enthusiast
Enthusiast

vRO + PowerShell plugin --> Get-Tag failure

vCenter 6.5d and e (tested with single VCSA/PSCA and three linked VCSA and external PSCA)

vRO 7.2.0.4629841 (built-in service in vRA 7.2.0.4659752, PS plugin @ 1.0.11)

PS script host tested with Windows 2008R2 and 2016 (connected with HTTPS, WinRM, shared session, Kerboros, and a domain cert)

Test script as follows:

Import-Module vmware.VimAutomation.core

Connect-VIServer -Server 'vcenter.company.loc' -User 'domain\account' -Password 'password'

$TagList = Get-Tag

Which results in:

PowerShellInvocationError: Errors found while executing script

Get-Tag : 8/8/2017 9:37:46 PM Get-Tag vSphere single sign-on failed for connection

'/VIServer=domain\account@vcenter.company.loc:443/'. Future operations which require single sign-on on this

connection will fail. The underlying cause was: The requested operation cannot be completed. The computer must be

trusted for delegation and the current user account must be configured to allow delegation.

That all being said... this worked prior to our upgrade to 6.5 from 6.0.  (and still does work to a 6.0 vCenter)  If I change the server name to a 6.0 vCenter in the connection string it works as expected.  6.5 vCenter.... errors.  What changed in the upgrade?

Does anyone think an vRO update to 7.3 would help?

31 Replies
daphnissov
Immortal
Immortal

What version of PowerCLI do you have on your PS host? Also, there is v1.0.13 of the PS plug-in you can use, not that I'd expect it to fix this.

Reply
0 Kudos
jonathank71
Enthusiast
Enthusiast

PowerCLI is 6.5.0.234.  (before and after the vSphere 6.5 upgrade)

Didn't know there was an update for the PS plugin.  I'll give it a shot.  I don't have high hopes though.  Seems like a bug in the PowerCLI cmdlet and how tagging works now.  I thought I read a post that tagging changed fundamentally in the backend.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Just to be on the latest, you should probably get PowerCLI 6.5.2. You must remove 6.5 before you install 6.5.1, but you can grab that from the PowerShell Gallery (Install-Module VMware.PowerCLI).

jonathank71
Enthusiast
Enthusiast

Uninstalled PowerCLI and did a Install-Module on the scripthost.  It loaded and now reports: 6.5.2.6234650.

Still sad panda.  Same error message.  Going to try the plugin update as soon as I get a window in vRA.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Hmm. With that updated PowerCLI, can you manually connect to vCenter and do a Get-Tag?

Reply
0 Kudos
jonathank71
Enthusiast
Enthusiast

Get-Tag has always worked from a PS console.  It's just when vRO kicks off the script in the scripthost that Get-Tag errors out.

FWIW, I updated the PS plugin to .13 and the error remains.

Reply
0 Kudos
igaydajiev
VMware Employee
VMware Employee

>The requested operation cannot be completed. The computer must be trusted for delegation and the current user account must be configured to allow delegation.

Powershell Plugin is doing the call from remotely. Which enforce different security context compared to running it directly on PS host machine.

You should receive same error if you invoke the command remotely for example by using winrs or winrm client and you use KERBEROS authentication with disabled delegation.

Multi-Hop Support in WinRM (Windows)

One option to overcome this issue is to use CredSSP authentication. Note that this authentication is not supported by PS plugin directly.

Here is a blog on the topic how to overcome multi hop issue in vRO Multi-hop | Spas Kaloferov's Blog 

Uridium454
Enthusiast
Enthusiast

I have a couple of questions and a couple of links that may be of some help to you.

1) Are you able to run this script from your vRA PowerShell host using your creds?

2) If you are able to run as yourself, can you run it as a service account that would have the necessary access? (Please see link below for storing and using creds)

Storing and using creds in PS - Storing Passwords to Disk in PowerShell with Machine-key Encryption | Tome's Land of IT

#Example script for retrieving tags using the stored cred

$encrypted = Import-Clixml 'C:\temp\do_not_delete.xml'           

$key = (1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)           

$csp = New-Object System.Security.Cryptography.CspParameters

$csp.KeyContainerName = "SuperSecretProcessOnMachine"

$csp.Flags = $csp.Flags -bor [System.Security.Cryptography.CspProviderFlags]::UseMachineKeyStore

$rsa = New-Object System.Security.Cryptography.RSACryptoServiceProvider -ArgumentList 5120,$csp

$rsa.PersistKeyInCsp = $true           

$password = [char[]]$rsa.Decrypt($encrypted, $true) -join "" |ConvertTo-SecureString -Key $key

$cred = New-Object System.Management.Automation.PsCredential 'myDomain\tome',$password

if(!(Get-Module VMware.VimAutomation.Core)){

     Import-Module VMware.VimAutomation.Core

}

Connect-VIServer -Server 'vcenter.company.loc' -Credential $cred

$TagList = Get-Tag

$TagList

Retrieving the values in vRO will require a bit more work.  You could use the following to parse through the returning values.

var result = output.getRootObject();

var val = "";

if( Object.prototype.toString.call( result ) === '[object Array]' ) {

    for each(var r in result){

          val += r;

     }

}else{

     val = result;

}

for each(var v in val){

     System.log("Tag Name: " + v);

}

As igaydajiev mentioned in previous reply, you will want to be using WINRM when you setup your PS host if you haven't already done so.  There are several blogs with the needed information out there.  The following was quite helpful to me when I setup my PS hosts - [vCO PowerShell plugin] How to set up and use Kerberos authentication - VMware vCenter Orchestrator ...

Hope this helps a bit, or at least gets you pointed in the correct direction.

Reply
0 Kudos
jonathank71
Enthusiast
Enthusiast

My biggest red flag for this is that it was working in vCenter 6.0 and then stopped working in 6.5.  All things being the same, something changed in vCenter and how tagging works now.  And it's just the tagging functions in PowerCLI/vRO.  Everything else works as it did in 6.0.

I've hard coded domain credentials as well as SSO creds.  Same error.

WinRM is setup and functional and used elsewhere in the script/workflows.

Credssp is also being used elsewhere without issue.  There is no parameter in Get-Tag to use Credssp authentication.  There really shouldn't be a need for it.

The domain service account and both server AD objects are set for delegation.  (didn't need this before 6.5)

Unless someone can verify that Get-Tag is working in a Powershell script called from the PowerShell plugin in vRO, I'm considering this a bug as I'm out of ideas and things to update/upgrade.

I did manage to wrap Get-Tag in a Invoke-Command using Credssp on localhost.  And that worked.  It's a bit of a pain and a kludge, but it's working.  I'm now creating a set of functions to replace the native tag cmdlets that work with a persistent pssession and Credssp.

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

We have been looking at upgrading since update 1 was released and I have a sandbox environment where I can reproduce the same error when running get-tag and get-tagassignment, however when I run get-vmhost or get-cluster through vRO it works just fine.  It appears to be only related to tags.

igaydajiev
VMware Employee
VMware Employee

Ok! This looks like change in PowerCLI tagging functionality.

Could you try same vRO/PowerShell plugin against different PS host running older version of PowerCLI(the one that used to work before)

Also you can try to invoke tagging functionality with winrm/winrs client remotely this should be prety close to what vRO is doing

Reply
0 Kudos
Uridium454
Enthusiast
Enthusiast

Sorry for the delay.  Fly and I will give this a go tomorrow and get back to you with the results.

Reply
0 Kudos
jonathank71
Enthusiast
Enthusiast

I got my Tag replacement functions functional and am retrofitting our scheduled script tasks as well as the vRO invoked scripts.

Interesting side note, Get-Vm -Tag $TagObj works now.  At least it does in the Invoke-Command scriptblock using Credssp.

I will be putting in an SR for this eventually.  I'll update this thread if anything comes of it.  I expect a fair amount of pushback since it's vRO/PowerShell.

Thanks for everyone's input!

Jonathan

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

We've opened a ticket with VMware to dig into this. SR 17541052208

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

igaydajiev​ - I don't understand why we would see these errors only for specific types of commandlet calls.  I have only tested get-cluster, get-vmhost, get-tag and get-tagassignment.  The two tag commandlets fail with this delegated user error where the cluster and vmhost calls work just fine.

Reply
0 Kudos
igaydajiev
VMware Employee
VMware Employee

"The computer must be trusted for delegation and the current user account must be configured to allow delegation."

Usually this kind of error is related to so called multi-hop ("double-hop") issue. Not sure if you had the chance to review provided link but it boils down to that:

You open a connection to remote system (ps host) providing Kerberos token. Then from PS host you try to access another remote system (this is the second hop) for example shared network folder,  AD server , or something else requiring authentication and you use same credentials then there should be specific configuration of the user and token used so that it is allowed to be used for authenticating against second remote system.

Usually this kind of error should not appear if you create new Credentials object (from username/password) directly in ps host and provide them to second remote system

I am not sure how exactly Get-Tag is implemented so can not say how provided credentials are used. That's why I suggested to test with older version of powerCLI and so we can confirm that change that triggered the issue is coming fro change in Get-Tag.

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

The only difference here is we can use the same script host, same vRO to hit a VCSA 6.0 and a VCSA 6.5 and the 6.0 works and the 6.5 doesn't. I'm using administrator@sphere.local as the login user.

Reply
0 Kudos
igaydajiev
VMware Employee
VMware Employee

Have you tried invoking same power-shell script outside vRO

Import-Module vmware.VimAutomation.core

Connect-VIServer -Server 'vcenter.company.loc' -User 'domain\account' -Password 'password'

$TagList = Get-Tag

1. Using PS console itself.

2. If that succeed try invoking it in remote context (How to Run PowerShell Commands on Remote Computers ).

If without change in vRO/PowerCLI component but only in vCenter one leads to above error it could be change in authentication in vCenter.

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

It fails the same through the remote session as it does through vRO.  However if I am on the console of the PS Host they all work just fine.

Reply
0 Kudos