I am observing reproducible errors when querying the tag assignments of about 1500 VMs managed by a single 6.5U1g VCSA. I have verified that the query "Get-VM | Get-TagAssignment" produces this error. Over repeated runs, the errors are not generated for the same VMs, nor are the number or errors the same from run to run. The error occurs in clusters, meaning that when it does occur, it will occur for multiple VMs, then stop, then occur for another cluster. I typically see two clusters of errors for the 1500 VMs. The error I am seeing is
Get-TagAssignment : 8/12/2018 2:49:23 AM Get-TagAssignment Unexpected character encountered while parsing value: <.
Path '', line 0, position 0.
At line:1 char:22
+ Get-VM | Sort Name | Get-TagAssignment
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-TagAssignment], CisException
+ FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssign
ment.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTagAssignment
I have submitted this problem to VMware's SDK Support team.
You didn't mention which PowerCLI version you are using.
In some older PowerCLI versions there were indeed issues with the Tag related cmdlets.
These issues should have disappeared in more recent PowerCLI versions, where the Tag cmdlets use the REST API.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Name Value
---- -----
PSVersion 5.1.14393.2395
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.14393.2395
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 6.7.0.8250345 VMware.DeployAutomation {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Co...
Script 6.7.0.8250345 VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare...
Manifest 10.1.1.8827524 VMware.PowerCLI
Script 6.7.0.8343295 VMware.Vim
Script 10.1.0.8377811 VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Script 10.0.0.7893901 VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServe...
Script 10.1.0.8342134 VMware.VimAutomation.Common
Script 10.1.0.8344055 VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetwo...
Script 6.5.4.7567193 VMware.VimAutomation.HA Get-DrmInfo
Script 7.5.0.8827468 VMware.VimAutomation.HorizonView {Connect-HVServer, Disconnect-HVServer}
Script 10.0.0.7893904 VMware.VimAutomation.License Get-LicenseDataManager
Script 10.1.0.8346947 VMware.VimAutomation.Nsxt {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtSe...
Script 10.0.0.7893924 VMware.VimAutomation.PCloud {Connect-PIServer, Disconnect-PIServer, Get-PIComputeI...
Script 10.1.0.8342078 VMware.VimAutomation.Sdk {Get-PSVersion, Get-InstallPath}
Script 10.0.0.7893900 VMware.VimAutomation.Srm {Connect-SrmServer, Disconnect-SrmServer}
Script 10.1.0.8313015 VMware.VimAutomation.Storage {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStora...
Script 1.2.0.0 VMware.VimAutomation.StorageUtility Update-VmfsDatastore
Script 10.1.0.8344219 VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHos...
Script 10.0.0.7893902 VMware.VimAutomation.Vmc {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-...
Script 10.0.0.7893921 VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, G...
Script 6.5.1.7862888 VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Com...
I did a complete uninstall of PowerCLI and reinstall with, what I'm assuming is, the latest 10.1 version.
I started noticing this odd behavior after our recent upgrade to 6.5U1g from 6.0. The error is non-terminating, so it was not an immediate show-stopper. I have implemented defensive coding where possible, forcing it to terminate using "-ErrorAction Stop" in a try/catch, then throwing a custom exception when necessary. The problem is that this introduces uncertainty and we perform unrecoverable operations based on our metadata, like removing snapshots.
Yes, that's the latest version, PowerCLI 10.1.1.
I don't know if there is a direct link with vSPhere 6.5U1g, but you're definitely not the only experiencing this issue.
See for example PowerCLI 10 - Get-tag and get-tagassignment are not working as expected.
If this is indeed a bug, and it looks like it, then you did well creating a SR.
Note that you can open SR for PowerCLI with your regular Support Contract, no need to have a Developer Support Contract.
See PowerCLI Support Breakdown
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I'm really glad you'd said that, Luc! I tried to open an SR against PowerCLI, but Support insisted they wanted to send this to the SDK team, and they've both been dogging me for an SDK Entitlement. I've repeatedly explained that I shouldn't need one if there is a bug in the product! I've got our account exec looped in too.
I know that 6.5 introduced noticeable behavioral changes in the metadata retrieval mechanism, like limiting the number of returned tag-associated items to 128. I assume that was to correct issues related to UI queries timing out, or something similar. As a long-time coder, changing an API's behavior in that way seems to break its contract with the caller. If, instead, the cmdlets batched queries until exhaustion, the contract would be preserved. I have been wondering if this bug is a side effect of those mods.
Anyway, I will update as I get more info from Support. Please let me know if there is anything else I can do to help.
Same issue, but with Powercli 10.2 and Vsphere 6.5U2
I'll try opening a ticket next week.
PS C:\Users\r> $vms=Get-VM
PS C:\Users\> $vms.Count
1696
PS C:\Users\> $tags=get-tag
get-tag : 9/7/2018 4:19:48 PM Get-Tag vSphere single sign-on failed for connection '/VIServer=xxxyyy:443/'. Future operations which require single sign-on on this
connection will fail. The underlying cause was: There was no endpoint listening at https://xxxyyypsc:7444/sts/STSService/vsphere.local that could accept the message. This is often caused by
an incorrect address or SOAP action. See InnerException, if present, for more details.
At line:1 char:7
+ $tags=get-tag
+ ~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-Tag], CisException
+ FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTag.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTag
Manifest 10.2.0.... VMware.PowerCLI
Script 6.7.0.9... VMware.Vim
Script 10.1.0.... VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Script 10.0.0.... VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}
Script 10.1.0.... VMware.VimAutomation.Common
Script 10.1.0.... VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}
Script 6.5.4.7... VMware.VimAutomation.HA Get-DrmInfo
Script 7.5.0.8... VMware.VimAutomation.HorizonView {Connect-HVServer, Disconnect-HVServer}
Script 10.0.0.... VMware.VimAutomation.License Get-LicenseDataManager
Script 10.2.0.... VMware.VimAutomation.Nsxt {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtService}
Script 10.0.0.... VMware.VimAutomation.PCloud {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}
Script 10.1.0.... VMware.VimAutomation.Sdk {Get-PSVersion, Get-InstallPath}
Script 10.0.0.... VMware.VimAutomation.Srm {Connect-SrmServer, Disconnect-SrmServer}
Script 10.1.0.... VMware.VimAutomation.Storage {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStoragePolicy, Get-KeyManagementServer...}
Script 1.2.0.0 VMware.VimAutomation.StorageUtility Update-VmfsDatastore
Script 10.1.0.... VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}
Script 10.0.0.... VMware.VimAutomation.Vmc {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-VmcServer...}
Script 10.0.0.... VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}
Script 6.5.1.7... VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}
We are also having this issue, any response from VMware support? Thinking i'll need to also raise a Service Request on this as it's causing us grief with our automation activities.
And is that with the latest PowerCLI release (currently 11.0.0)?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Same issue here, with powercli 11.0.0
we had this on vSphere 6.5 and now after upgrading to 6.7u1 it's still there
The behaviour a bit unstable, repeating the same command will normaly work fine.
I would also suggest to open a SR (see above).
The more they receive on this issue, the sooner it might be fixed.
Out of curiosity, are you running with an integrated PSC or a separate PSC?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
We use integrated PSC on our VCSA servers
I will open a SR on this issue.
I upgraded the customer's environment from 6.0 to 6.7 two days ago, and unfortunately with that upgrade the daily reporting scripts stopped working.
Took me a while to find out this was related to the filtering by tags.
Here are my version details:
PowerCLI Version
----------------
VMware PowerCLI 11.3.0 build 13990089
---------------
Component Versions
---------------
VMware Cis Core PowerCLI Component PowerCLI Component 11.3 build 13964830
VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 11.3 build 13964826
The following commands related to tagging thus fail:
Get-Tag
Get-TagAssignment
Get-TagCategory
Has anyone managed to solve this in the meantime?
MarcelSwartjes did you file a SR and did you ever hear back from an engineering team at VMware?
I only read about very complicated workarounds by using wrappers, vRA, vRO and other tinkering I don't manage to wrap my head around.
Thanks.
BR
NC
The SR has been open for a long time and closed without a good solution.
VMware has created a best-practice white paper https://communities.vmware.com/create-advanced-comment.jspa?id=2926953&draftID=578391
This best-practice does not really solve the problem, the numbers are very low, for example 10K VMs each with 3 tag assignments is to much.
What really did help was this command on de VCSA:
cloudvm-ram-size -C 512 vmware-vapi-endpoint
Followed by restart service vmware-vapi-endpoint
service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint
cloudvm-ram-size -l | grep vmware-vapi-endpoint shows a value of 608, the original value is 256.
note: You need to repeat this command after upgrading to a new version of VCSA.
Our TAM told me that VMware has build a paging mechanism for querying tag assignments that's part of 6.7u2 and will probably be implemented in a future version of PowerCli.