VMware Cloud Community
Paul_Knight1
Contributor
Contributor

Unstable Retrieval of Tag Assignments Introduced In 6.5?

I am observing reproducible errors when querying the tag assignments of about 1500 VMs managed by a single 6.5U1g VCSA.  I have verified that the query "Get-VM | Get-TagAssignment" produces this error.  Over repeated runs, the errors are not generated for the same VMs, nor are the number or errors the same from run to run.  The error occurs in clusters, meaning that when it does occur, it will occur for multiple VMs, then stop, then occur for another cluster.  I typically see two clusters of errors for the 1500 VMs.  The error I am seeing is

Get-TagAssignment : 8/12/2018 2:49:23 AM Get-TagAssignment Unexpected character encountered while parsing value: <.

Path '', line 0, position 0.

At line:1 char:22

+ Get-VM | Sort Name | Get-TagAssignment

+                      ~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Get-TagAssignment], CisException

    + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssign

ment.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTagAssignment

I have submitted this problem to VMware's SDK Support team.

Reply
0 Kudos
14 Replies
LucD
Leadership
Leadership

You didn't mention which PowerCLI version you are using.

In some older PowerCLI versions there were indeed issues with the Tag related cmdlets.


These issues should have disappeared in more recent PowerCLI versions, where the Tag cmdlets use the REST API.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Paul_Knight1
Contributor
Contributor

Name                                          Value

----                                               -----

PSVersion                                   5.1.14393.2395

PSEdition                                    Desktop

PSCompatibleVersions              {1.0, 2.0, 3.0, 4.0...}

BuildVersion                               10.0.14393.2395

CLRVersion                                4.0.30319.42000

WSManStackVersion                  3.0

PSRemotingProtocolVersion      2.3

SerializationVersion                    1.1.0.1

ModuleType Version                 Name                                                        ExportedCommands

----------         -------                    ----                                                             ----------------

Script            6.7.0.8250345     VMware.DeployAutomation                      {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Co...

Script            6.7.0.8250345     VMware.ImageBuilder                              {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare...

Manifest       10.1.1.8827524    VMware.PowerCLI

Script            6.7.0.8343295     VMware.Vim

Script            10.1.0.8377811    VMware.VimAutomation.Cis.Core           {Connect-CisServer, Disconnect-CisServer, Get-CisService}

Script            10.0.0.7893901    VMware.VimAutomation.Cloud               {Add-CIDatastore, Connect-CIServer, Disconnect-CIServe...

Script            10.1.0.8342134    VMware.VimAutomation.Common

Script            10.1.0.8344055    VMware.VimAutomation.Core                {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetwo...

Script            6.5.4.7567193      VMware.VimAutomation.HA                   Get-DrmInfo

Script            7.5.0.8827468      VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}

Script            10.0.0.7893904    VMware.VimAutomation.License            Get-LicenseDataManager

Script            10.1.0.8346947    VMware.VimAutomation.Nsxt                 {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtSe...

Script            10.0.0.7893924    VMware.VimAutomation.PCloud            {Connect-PIServer, Disconnect-PIServer, Get-PIComputeI...

Script            10.1.0.8342078    VMware.VimAutomation.Sdk                  {Get-PSVersion, Get-InstallPath}

Script            10.0.0.7893900    VMware.VimAutomation.Srm                 {Connect-SrmServer, Disconnect-SrmServer}

Script            10.1.0.8313015    VMware.VimAutomation.Storage           {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStora...

Script            1.2.0.0                  VMware.VimAutomation.StorageUtility   Update-VmfsDatastore

Script            10.1.0.8344219    VMware.VimAutomation.Vds                  {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHos...

Script            10.0.0.7893902    VMware.VimAutomation.Vmc                 {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-...

Script            10.0.0.7893921    VMware.VimAutomation.vROps             {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, G...

Script            6.5.1.7862888      VMware.VumAutomation                        {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Com...

Reply
0 Kudos
Paul_Knight1
Contributor
Contributor

I did a complete uninstall of PowerCLI and reinstall with, what I'm assuming is, the latest 10.1 version.

Reply
0 Kudos
Paul_Knight1
Contributor
Contributor

I started noticing this odd behavior after our recent upgrade to 6.5U1g from 6.0.  The error is non-terminating, so it was not an immediate show-stopper.  I have implemented defensive coding where possible, forcing it to terminate using "-ErrorAction Stop" in a try/catch, then throwing a custom exception when necessary.  The problem is that this introduces uncertainty and we perform unrecoverable operations based on our metadata, like removing snapshots.

Reply
0 Kudos
LucD
Leadership
Leadership

Yes, that's the latest version, PowerCLI 10.1.1.

I don't know if there is a direct link with vSPhere 6.5U1g, but you're definitely not the only experiencing this issue.
See for example PowerCLI 10 - Get-tag and get-tagassignment are not working as expected.

If this is indeed a bug, and it looks like it, then you did well creating a SR.

Note that you can open SR for PowerCLI with your regular Support Contract, no need to have a Developer Support Contract.

See PowerCLI Support Breakdown


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Paul_Knight1
Contributor
Contributor

I'm really glad you'd said that, Luc!  I tried to open an SR against PowerCLI, but Support insisted they wanted to send this to the SDK team, and they've both been dogging me for an SDK Entitlement.  I've repeatedly explained that I shouldn't need one if there is a bug in the product!  I've got our account exec looped in too.

I know that 6.5 introduced noticeable behavioral changes in the metadata retrieval mechanism, like limiting the number of returned tag-associated items to 128.  I assume that was to correct issues related to UI queries timing out, or something similar.  As a long-time coder, changing an API's behavior in that way seems to break its contract with the caller.  If, instead, the cmdlets batched queries until exhaustion, the contract would be preserved.  I have been wondering if this bug is a side effect of those mods.

Anyway, I will update as I get more info from Support.  Please let me know if there is anything else I can do to help.

Reply
0 Kudos
eric_silberberg
Enthusiast
Enthusiast

Same issue, but with Powercli 10.2 and Vsphere 6.5U2

I'll try opening a ticket next week.

PS C:\Users\r> $vms=Get-VM

PS C:\Users\> $vms.Count

1696

PS C:\Users\> $tags=get-tag

get-tag : 9/7/2018 4:19:48 PM Get-Tag vSphere single sign-on failed for connection '/VIServer=xxxyyy:443/'. Future operations which require single sign-on on this

connection will fail. The underlying cause was: There was no endpoint listening at https://xxxyyypsc:7444/sts/STSService/vsphere.local that could accept the message. This is often caused by

an incorrect address or SOAP action. See InnerException, if present, for more details.

At line:1 char:7

+ $tags=get-tag

+       ~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Get-Tag], CisException

    + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTag.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTag

Manifest   10.2.0.... VMware.PowerCLI                                                                                                                                                                      

Script     6.7.0.9... VMware.Vim                                                                                                                                                                           

Script     10.1.0.... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}                                                                                        

Script     10.0.0.... VMware.VimAutomation.Cloud          {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}                                                                         

Script     10.1.0.... VMware.VimAutomation.Common                                                                                                                                                          

Script     10.1.0.... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}                                             

Script     6.5.4.7... VMware.VimAutomation.HA             Get-DrmInfo                                                                                                                                      

Script     7.5.0.8... VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}                                                                                                          

Script     10.0.0.... VMware.VimAutomation.License        Get-LicenseDataManager                                                                                                                           

Script     10.2.0.... VMware.VimAutomation.Nsxt           {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtService}                                                                                     

Script     10.0.0.... VMware.VimAutomation.PCloud         {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}                                                                 

Script     10.1.0.... VMware.VimAutomation.Sdk            {Get-PSVersion, Get-InstallPath}                                                                                                                 

Script     10.0.0.... VMware.VimAutomation.Srm            {Connect-SrmServer, Disconnect-SrmServer}                                                                                                        

Script     10.1.0.... VMware.VimAutomation.Storage        {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStoragePolicy, Get-KeyManagementServer...}                                                      

Script     1.2.0.0    VMware.VimAutomation.StorageUtility Update-VmfsDatastore                                                                                                                             

Script     10.1.0.... VMware.VimAutomation.Vds            {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}                                                 

Script     10.0.0.... VMware.VimAutomation.Vmc            {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-VmcServer...}                                                                              

Script     10.0.0.... VMware.VimAutomation.vROps          {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}                                                                   

Script     6.5.1.7... VMware.VumAutomation                {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}                                                                                

Reply
0 Kudos
RobGoodworth
Contributor
Contributor

We are also having this issue, any response from VMware support?  Thinking i'll need to also raise a Service Request on this as it's causing us grief with our automation activities.

RobGoodworth
Reply
0 Kudos
LucD
Leadership
Leadership

And is that with the latest PowerCLI release (currently 11.0.0)?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
MarcelSwartjes
Contributor
Contributor

Same issue here, with powercli 11.0.0
we had this on vSphere 6.5 and now after upgrading to 6.7u1 it's still there

The behaviour a bit unstable, repeating the same command will normaly work fine.

Marcel
Reply
0 Kudos
LucD
Leadership
Leadership

I would also suggest to open a SR (see above).
The more they receive on this issue, the sooner it might be fixed.

Out of curiosity, are you running with an integrated PSC or a separate PSC?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
MarcelSwartjes
Contributor
Contributor

We use integrated PSC on our VCSA servers
I will open a SR on this issue.

Marcel
NelsonCandela
Enthusiast
Enthusiast

I upgraded the customer's environment from 6.0 to 6.7 two days ago, and unfortunately with that upgrade the daily reporting scripts stopped working.

Took me a while to find out this was related to the filtering by tags.

Here are my version details:

PowerCLI Version

----------------

   VMware PowerCLI 11.3.0 build 13990089

---------------

Component Versions

---------------

   VMware Cis Core PowerCLI Component PowerCLI Component 11.3 build 13964830

   VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 11.3 build 13964826

The following commands related to tagging thus fail:

Get-Tag

Get-TagAssignment

Get-TagCategory

Has anyone managed to solve this in the meantime?

MarcelSwartjes​ did you file a SR and did you ever hear back from an engineering team at VMware?

I only read about very complicated workarounds by using wrappers, vRA, vRO and other tinkering I don't manage to wrap my head around.

Thanks.

BR

NC

Reply
0 Kudos
MarcelSwartjes
Contributor
Contributor

The SR has been open for a long time and closed without a good solution.
VMware has created a best-practice white paper https://communities.vmware.com/create-advanced-comment.jspa?id=2926953&draftID=578391

This best-practice does not really solve the problem, the numbers are very low, for example 10K VMs each with 3 tag assignments is to much.

What really did help was this command on de VCSA:

cloudvm-ram-size -C 512 vmware-vapi-endpoint
Followed by restart service vmware-vapi-endpoint

service-control --stop vmware-vapi-endpoint

service-control --start vmware-vapi-endpoint

cloudvm-ram-size -l | grep vmware-vapi-endpoint shows a value of 608, the original value is 256.

note: You need to repeat this command after upgrading to a new version of VCSA.

Our TAM told me that VMware has build a paging mechanism for querying tag assignments that's part of 6.7u2 and will probably be implemented in a future version of PowerCli.

Marcel
Reply
0 Kudos