12 Replies Latest reply on Feb 8, 2019 1:47 PM by MarcelSwartjes

    Unstable Retrieval of Tag Assignments Introduced In 6.5?

    Paul.Knight Novice

      I am observing reproducible errors when querying the tag assignments of about 1500 VMs managed by a single 6.5U1g VCSA.  I have verified that the query "Get-VM | Get-TagAssignment" produces this error.  Over repeated runs, the errors are not generated for the same VMs, nor are the number or errors the same from run to run.  The error occurs in clusters, meaning that when it does occur, it will occur for multiple VMs, then stop, then occur for another cluster.  I typically see two clusters of errors for the 1500 VMs.  The error I am seeing is

       

      Get-TagAssignment : 8/12/2018 2:49:23 AM Get-TagAssignment Unexpected character encountered while parsing value: <.

      Path '', line 0, position 0.

      At line:1 char:22

      + Get-VM | Sort Name | Get-TagAssignment

      +                      ~~~~~~~~~~~~~~~~~

          + CategoryInfo          : NotSpecified: (:) [Get-TagAssignment], CisException

          + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssign

      ment.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTagAssignment

       

      I have submitted this problem to VMware's SDK Support team.

        • 1. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
          LucD Guru
          Community WarriorsUser ModeratorsvExpert

          You didn't mention which PowerCLI version you are using.

          In some older PowerCLI versions there were indeed issues with the Tag related cmdlets.


          These issues should have disappeared in more recent PowerCLI versions, where the Tag cmdlets use the REST API.

          • 2. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
            Paul.Knight Novice

            Name                                          Value

            ----                                               -----

            PSVersion                                   5.1.14393.2395

            PSEdition                                    Desktop

            PSCompatibleVersions              {1.0, 2.0, 3.0, 4.0...}

            BuildVersion                               10.0.14393.2395

            CLRVersion                                4.0.30319.42000

            WSManStackVersion                  3.0

            PSRemotingProtocolVersion      2.3

            SerializationVersion                    1.1.0.1

             

            ModuleType Version                 Name                                                        ExportedCommands

            ----------         -------                    ----                                                             ----------------

            Script            6.7.0.8250345     VMware.DeployAutomation                      {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Co...

            Script            6.7.0.8250345     VMware.ImageBuilder                              {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare...

            Manifest       10.1.1.8827524    VMware.PowerCLI

            Script            6.7.0.8343295     VMware.Vim

            Script            10.1.0.8377811    VMware.VimAutomation.Cis.Core           {Connect-CisServer, Disconnect-CisServer, Get-CisService}

            Script            10.0.0.7893901    VMware.VimAutomation.Cloud               {Add-CIDatastore, Connect-CIServer, Disconnect-CIServe...

            Script            10.1.0.8342134    VMware.VimAutomation.Common

            Script            10.1.0.8344055    VMware.VimAutomation.Core                {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetwo...

            Script            6.5.4.7567193      VMware.VimAutomation.HA                   Get-DrmInfo

            Script            7.5.0.8827468      VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}

            Script            10.0.0.7893904    VMware.VimAutomation.License            Get-LicenseDataManager

            Script            10.1.0.8346947    VMware.VimAutomation.Nsxt                 {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtSe...

            Script            10.0.0.7893924    VMware.VimAutomation.PCloud            {Connect-PIServer, Disconnect-PIServer, Get-PIComputeI...

            Script            10.1.0.8342078    VMware.VimAutomation.Sdk                  {Get-PSVersion, Get-InstallPath}

            Script            10.0.0.7893900    VMware.VimAutomation.Srm                 {Connect-SrmServer, Disconnect-SrmServer}

            Script            10.1.0.8313015    VMware.VimAutomation.Storage           {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStora...

            Script            1.2.0.0                  VMware.VimAutomation.StorageUtility   Update-VmfsDatastore

            Script            10.1.0.8344219    VMware.VimAutomation.Vds                  {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHos...

            Script            10.0.0.7893902    VMware.VimAutomation.Vmc                 {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-...

            Script            10.0.0.7893921    VMware.VimAutomation.vROps             {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, G...

            Script            6.5.1.7862888      VMware.VumAutomation                        {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Com...

            • 3. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
              Paul.Knight Novice

              I did a complete uninstall of PowerCLI and reinstall with, what I'm assuming is, the latest 10.1 version.

              • 4. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                Paul.Knight Novice

                I started noticing this odd behavior after our recent upgrade to 6.5U1g from 6.0.  The error is non-terminating, so it was not an immediate show-stopper.  I have implemented defensive coding where possible, forcing it to terminate using "-ErrorAction Stop" in a try/catch, then throwing a custom exception when necessary.  The problem is that this introduces uncertainty and we perform unrecoverable operations based on our metadata, like removing snapshots.

                • 5. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                  LucD Guru
                  vExpertUser ModeratorsCommunity Warriors

                  Yes, that's the latest version, PowerCLI 10.1.1.

                   

                  I don't know if there is a direct link with vSPhere 6.5U1g, but you're definitely not the only experiencing this issue.
                  See for example PowerCLI 10 - Get-tag and get-tagassignment are not working as expected.

                   

                  If this is indeed a bug, and it looks like it, then you did well creating a SR.

                  Note that you can open SR for PowerCLI with your regular Support Contract, no need to have a Developer Support Contract.

                  See PowerCLI Support Breakdown

                  • 6. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                    Paul.Knight Novice

                    I'm really glad you'd said that, Luc!  I tried to open an SR against PowerCLI, but Support insisted they wanted to send this to the SDK team, and they've both been dogging me for an SDK Entitlement.  I've repeatedly explained that I shouldn't need one if there is a bug in the product!  I've got our account exec looped in too.

                     

                    I know that 6.5 introduced noticeable behavioral changes in the metadata retrieval mechanism, like limiting the number of returned tag-associated items to 128.  I assume that was to correct issues related to UI queries timing out, or something similar.  As a long-time coder, changing an API's behavior in that way seems to break its contract with the caller.  If, instead, the cmdlets batched queries until exhaustion, the contract would be preserved.  I have been wondering if this bug is a side effect of those mods.

                     

                    Anyway, I will update as I get more info from Support.  Please let me know if there is anything else I can do to help.

                    • 7. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                      eric_silberberg Novice

                      Same issue, but with Powercli 10.2 and Vsphere 6.5U2

                       

                      I'll try opening a ticket next week.

                      PS C:\Users\r> $vms=Get-VM

                       

                       

                      PS C:\Users\> $vms.Count

                      1696

                       

                       

                      PS C:\Users\> $tags=get-tag

                      get-tag : 9/7/2018 4:19:48 PM Get-Tag vSphere single sign-on failed for connection '/VIServer=xxxyyy:443/'. Future operations which require single sign-on on this

                      connection will fail. The underlying cause was: There was no endpoint listening at https://xxxyyypsc:7444/sts/STSService/vsphere.local that could accept the message. This is often caused by

                      an incorrect address or SOAP action. See InnerException, if present, for more details.

                      At line:1 char:7

                      + $tags=get-tag

                      +       ~~~~~~~

                          + CategoryInfo          : NotSpecified: (:) [Get-Tag], CisException

                          + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTag.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTag

                       

                       

                      Manifest   10.2.0.... VMware.PowerCLI                                                                                                                                                                      

                      Script     6.7.0.9... VMware.Vim                                                                                                                                                                           

                      Script     10.1.0.... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}                                                                                        

                      Script     10.0.0.... VMware.VimAutomation.Cloud          {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}                                                                         

                      Script     10.1.0.... VMware.VimAutomation.Common                                                                                                                                                          

                      Script     10.1.0.... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}                                             

                      Script     6.5.4.7... VMware.VimAutomation.HA             Get-DrmInfo                                                                                                                                      

                      Script     7.5.0.8... VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}                                                                                                          

                      Script     10.0.0.... VMware.VimAutomation.License        Get-LicenseDataManager                                                                                                                           

                      Script     10.2.0.... VMware.VimAutomation.Nsxt           {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtService}                                                                                     

                      Script     10.0.0.... VMware.VimAutomation.PCloud         {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}                                                                 

                      Script     10.1.0.... VMware.VimAutomation.Sdk            {Get-PSVersion, Get-InstallPath}                                                                                                                 

                      Script     10.0.0.... VMware.VimAutomation.Srm            {Connect-SrmServer, Disconnect-SrmServer}                                                                                                        

                      Script     10.1.0.... VMware.VimAutomation.Storage        {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStoragePolicy, Get-KeyManagementServer...}                                                      

                      Script     1.2.0.0    VMware.VimAutomation.StorageUtility Update-VmfsDatastore                                                                                                                             

                      Script     10.1.0.... VMware.VimAutomation.Vds            {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}                                                 

                      Script     10.0.0.... VMware.VimAutomation.Vmc            {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-VmcServer...}                                                                              

                      Script     10.0.0.... VMware.VimAutomation.vROps          {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}                                                                   

                      Script     6.5.1.7... VMware.VumAutomation                {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}                                                                                

                      • 8. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                        RobGoodworth Lurker

                        We are also having this issue, any response from VMware support?  Thinking i'll need to also raise a Service Request on this as it's causing us grief with our automation activities.

                        • 9. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                          LucD Guru
                          Community WarriorsvExpertUser Moderators

                          And is that with the latest PowerCLI release (currently 11.0.0)?

                          • 10. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                            MarcelSwartjes Lurker

                            Same issue here, with powercli 11.0.0
                            we had this on vSphere 6.5 and now after upgrading to 6.7u1 it's still there

                             

                            The behaviour a bit unstable, repeating the same command will normaly work fine.

                            • 11. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                              LucD Guru
                              User ModeratorsvExpertCommunity Warriors

                              I would also suggest to open a SR (see above).
                              The more they receive on this issue, the sooner it might be fixed.

                               

                              Out of curiosity, are you running with an integrated PSC or a separate PSC?

                              • 12. Re: Unstable Retrieval of Tag Assignments Introduced In 6.5?
                                MarcelSwartjes Lurker

                                We use integrated PSC on our VCSA servers
                                I will open a SR on this issue.