14 Replies Latest reply on Feb 17, 2020 3:47 AM by MarcelSwartjes

    Unstable Retrieval of Tag Assignments Introduced In 6.5?

    Paul.Knight Novice

      I am observing reproducible errors when querying the tag assignments of about 1500 VMs managed by a single 6.5U1g VCSA.  I have verified that the query "Get-VM | Get-TagAssignment" produces this error.  Over repeated runs, the errors are not generated for the same VMs, nor are the number or errors the same from run to run.  The error occurs in clusters, meaning that when it does occur, it will occur for multiple VMs, then stop, then occur for another cluster.  I typically see two clusters of errors for the 1500 VMs.  The error I am seeing is


      Get-TagAssignment : 8/12/2018 2:49:23 AM Get-TagAssignment Unexpected character encountered while parsing value: <.

      Path '', line 0, position 0.

      At line:1 char:22

      + Get-VM | Sort Name | Get-TagAssignment

      +                      ~~~~~~~~~~~~~~~~~

          + CategoryInfo          : NotSpecified: (:) [Get-TagAssignment], CisException

          + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssign



      I have submitted this problem to VMware's SDK Support team.

        • 1. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
          LucD Guru
          vExpertUser ModeratorsCommunity Warriors

          You didn't mention which PowerCLI version you are using.

          In some older PowerCLI versions there were indeed issues with the Tag related cmdlets.

          These issues should have disappeared in more recent PowerCLI versions, where the Tag cmdlets use the REST API.

          Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
          • 2. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
            Paul.Knight Novice

            Name                                          Value

            ----                                               -----

            PSVersion                                   5.1.14393.2395

            PSEdition                                    Desktop

            PSCompatibleVersions              {1.0, 2.0, 3.0, 4.0...}

            BuildVersion                               10.0.14393.2395

            CLRVersion                                4.0.30319.42000

            WSManStackVersion                  3.0

            PSRemotingProtocolVersion      2.3



            ModuleType Version                 Name                                                        ExportedCommands

            ----------         -------                    ----                                                             ----------------

            Script       VMware.DeployAutomation                      {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Co...

            Script       VMware.ImageBuilder                              {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare...

            Manifest    VMware.PowerCLI

            Script       VMware.Vim

            Script      VMware.VimAutomation.Cis.Core           {Connect-CisServer, Disconnect-CisServer, Get-CisService}

            Script      VMware.VimAutomation.Cloud               {Add-CIDatastore, Connect-CIServer, Disconnect-CIServe...

            Script      VMware.VimAutomation.Common

            Script      VMware.VimAutomation.Core                {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetwo...

            Script        VMware.VimAutomation.HA                   Get-DrmInfo

            Script        VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}

            Script      VMware.VimAutomation.License            Get-LicenseDataManager

            Script      VMware.VimAutomation.Nsxt                 {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtSe...

            Script      VMware.VimAutomation.PCloud            {Connect-PIServer, Disconnect-PIServer, Get-PIComputeI...

            Script      VMware.VimAutomation.Sdk                  {Get-PSVersion, Get-InstallPath}

            Script      VMware.VimAutomation.Srm                 {Connect-SrmServer, Disconnect-SrmServer}

            Script      VMware.VimAutomation.Storage           {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStora...

            Script                    VMware.VimAutomation.StorageUtility   Update-VmfsDatastore

            Script      VMware.VimAutomation.Vds                  {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHos...

            Script      VMware.VimAutomation.Vmc                 {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-...

            Script      VMware.VimAutomation.vROps             {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, G...

            Script        VMware.VumAutomation                        {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Com...

            • 3. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
              Paul.Knight Novice

              I did a complete uninstall of PowerCLI and reinstall with, what I'm assuming is, the latest 10.1 version.

              • 4. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                Paul.Knight Novice

                I started noticing this odd behavior after our recent upgrade to 6.5U1g from 6.0.  The error is non-terminating, so it was not an immediate show-stopper.  I have implemented defensive coding where possible, forcing it to terminate using "-ErrorAction Stop" in a try/catch, then throwing a custom exception when necessary.  The problem is that this introduces uncertainty and we perform unrecoverable operations based on our metadata, like removing snapshots.

                • 5. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                  LucD Guru
                  User ModeratorsCommunity WarriorsvExpert

                  Yes, that's the latest version, PowerCLI 10.1.1.


                  I don't know if there is a direct link with vSPhere 6.5U1g, but you're definitely not the only experiencing this issue.
                  See for example PowerCLI 10 - Get-tag and get-tagassignment are not working as expected.


                  If this is indeed a bug, and it looks like it, then you did well creating a SR.

                  Note that you can open SR for PowerCLI with your regular Support Contract, no need to have a Developer Support Contract.

                  See PowerCLI Support Breakdown

                  Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
                  • 6. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                    Paul.Knight Novice

                    I'm really glad you'd said that, Luc!  I tried to open an SR against PowerCLI, but Support insisted they wanted to send this to the SDK team, and they've both been dogging me for an SDK Entitlement.  I've repeatedly explained that I shouldn't need one if there is a bug in the product!  I've got our account exec looped in too.


                    I know that 6.5 introduced noticeable behavioral changes in the metadata retrieval mechanism, like limiting the number of returned tag-associated items to 128.  I assume that was to correct issues related to UI queries timing out, or something similar.  As a long-time coder, changing an API's behavior in that way seems to break its contract with the caller.  If, instead, the cmdlets batched queries until exhaustion, the contract would be preserved.  I have been wondering if this bug is a side effect of those mods.


                    Anyway, I will update as I get more info from Support.  Please let me know if there is anything else I can do to help.

                    • 7. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                      eric_silberberg Novice

                      Same issue, but with Powercli 10.2 and Vsphere 6.5U2


                      I'll try opening a ticket next week.

                      PS C:\Users\r> $vms=Get-VM



                      PS C:\Users\> $vms.Count




                      PS C:\Users\> $tags=get-tag

                      get-tag : 9/7/2018 4:19:48 PM Get-Tag vSphere single sign-on failed for connection '/VIServer=xxxyyy:443/'. Future operations which require single sign-on on this

                      connection will fail. The underlying cause was: There was no endpoint listening at https://xxxyyypsc:7444/sts/STSService/vsphere.local that could accept the message. This is often caused by

                      an incorrect address or SOAP action. See InnerException, if present, for more details.

                      At line:1 char:7

                      + $tags=get-tag

                      +       ~~~~~~~

                          + CategoryInfo          : NotSpecified: (:) [Get-Tag], CisException

                          + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTag.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTag



                      Manifest   10.2.0.... VMware.PowerCLI                                                                                                                                                                      

                      Script VMware.Vim                                                                                                                                                                           

                      Script     10.1.0.... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}                                                                                        

                      Script     10.0.0.... VMware.VimAutomation.Cloud          {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}                                                                         

                      Script     10.1.0.... VMware.VimAutomation.Common                                                                                                                                                          

                      Script     10.1.0.... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}                                             

                      Script VMware.VimAutomation.HA             Get-DrmInfo                                                                                                                                      

                      Script VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}                                                                                                          

                      Script     10.0.0.... VMware.VimAutomation.License        Get-LicenseDataManager                                                                                                                           

                      Script     10.2.0.... VMware.VimAutomation.Nsxt           {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtService}                                                                                     

                      Script     10.0.0.... VMware.VimAutomation.PCloud         {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}                                                                 

                      Script     10.1.0.... VMware.VimAutomation.Sdk            {Get-PSVersion, Get-InstallPath}                                                                                                                 

                      Script     10.0.0.... VMware.VimAutomation.Srm            {Connect-SrmServer, Disconnect-SrmServer}                                                                                                        

                      Script     10.1.0.... VMware.VimAutomation.Storage        {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStoragePolicy, Get-KeyManagementServer...}                                                      

                      Script    VMware.VimAutomation.StorageUtility Update-VmfsDatastore                                                                                                                             

                      Script     10.1.0.... VMware.VimAutomation.Vds            {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}                                                 

                      Script     10.0.0.... VMware.VimAutomation.Vmc            {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-VmcServer...}                                                                              

                      Script     10.0.0.... VMware.VimAutomation.vROps          {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}                                                                   

                      Script VMware.VumAutomation                {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}                                                                                

                      • 8. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                        RobGoodworth Lurker

                        We are also having this issue, any response from VMware support?  Thinking i'll need to also raise a Service Request on this as it's causing us grief with our automation activities.

                        • 9. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                          LucD Guru
                          vExpertCommunity WarriorsUser Moderators

                          And is that with the latest PowerCLI release (currently 11.0.0)?

                          Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
                          • 10. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                            MarcelSwartjes Novice

                            Same issue here, with powercli 11.0.0
                            we had this on vSphere 6.5 and now after upgrading to 6.7u1 it's still there


                            The behaviour a bit unstable, repeating the same command will normaly work fine.

                            • 11. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                              LucD Guru
                              User ModeratorsvExpertCommunity Warriors

                              I would also suggest to open a SR (see above).
                              The more they receive on this issue, the sooner it might be fixed.


                              Out of curiosity, are you running with an integrated PSC or a separate PSC?

                              Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
                              • 12. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                                MarcelSwartjes Novice

                                We use integrated PSC on our VCSA servers
                                I will open a SR on this issue.

                                1 person found this helpful
                                • 13. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                                  NelsonCandela Enthusiast

                                  I upgraded the customer's environment from 6.0 to 6.7 two days ago, and unfortunately with that upgrade the daily reporting scripts stopped working.

                                  Took me a while to find out this was related to the filtering by tags.


                                  Here are my version details:


                                  PowerCLI Version
                                     VMware PowerCLI 11.3.0 build 13990089
                                  Component Versions
                                     VMware Cis Core PowerCLI Component PowerCLI Component 11.3 build 13964830
                                     VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 11.3 build 13964826


                                  The following commands related to tagging thus fail:




                                  Has anyone managed to solve this in the meantime?

                                  MarcelSwartjes did you file a SR and did you ever hear back from an engineering team at VMware?


                                  I only read about very complicated workarounds by using wrappers, vRA, vRO and other tinkering I don't manage to wrap my head around.





                                  • 14. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                                    MarcelSwartjes Novice

                                    The SR has been open for a long time and closed without a good solution.
                                    VMware has created a best-practice white paper https://communities.vmware.com/create-advanced-comment.jspa?id=2926953&draftID=578391

                                    This best-practice does not really solve the problem, the numbers are very low, for example 10K VMs each with 3 tag assignments is to much.


                                    What really did help was this command on de VCSA:

                                    cloudvm-ram-size -C 512 vmware-vapi-endpoint
                                    Followed by restart service vmware-vapi-endpoint

                                    service-control --stop vmware-vapi-endpoint

                                    service-control --start vmware-vapi-endpoint


                                    cloudvm-ram-size -l | grep vmware-vapi-endpoint shows a value of 608, the original value is 256.


                                    note: You need to repeat this command after upgrading to a new version of VCSA.


                                    Our TAM told me that VMware has build a paging mechanism for querying tag assignments that's part of 6.7u2 and will probably be implemented in a future version of PowerCli.