Issues with Global get-tagassigment

[JBartholomew]

We recently started running into an issue in one of our vCenters with a script that I had written to grab the inventory of VMs in our environment including the Tags associated with the VMs. We have 3 vCenters, all running 6.5, and 2 of the 3 vCenters run the script fine, but when we run the script against the 3rd vCenter, we get an error:

Get-TagAssignment : 5/14/2018 4:40:31 PM        Get-TagAssignment               com.vmware.vapi.std.errors.internal_server_error {'messages': [com.vmware.vapi.std.localizable_message {'id':
vapi.bindings.method.impl.unexpected, 'default_message': Provider method implementation threw unexpected exception: Read timed out, 'args': [Read timed out]}], 'data':}

At line:1 char:1
+ Get-TagAssignment
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-TagAssignment], CisException
    + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssignment.Error,VMware.VimAutomation.ViCore.Cmdlets.Comma
   nds.Tagging.GetTagAssignment

The vCenter with the issues is our largest site, and has the most tags, so I believe we have possibly hit a limit of the number of results get-tagassignment can return as this script worked fine until a week ago when we added some more tags to the VMs.

I have looked at a few other threads on here, and in one of them LucD recommended using his rCIS module to get around a very similar issue. I have attempted to run the rCIS module and am getting an error with it as well:

Get-View : 5/14/2018 4:39:19 PM Get-View                You are not currently connected to any servers. Please connect first using a Connect cmdlet.
At FileLocation\Projects\VMWare\rCisTag\rCisTag.psm1:408 char:35
+ ...                   Entity = (Get-View -id $sMoRef -Property Name).Name
+                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [Get-View], ViServerConnectionException
    + FullyQualifiedErrorId : Core_BaseCmdlet_NotConnectedError,VMware.VimAutomation.ViCore.Cmdlets.Commands.DotNetInterop.GetVIView

I have verified that both get-tag and get-rcistag works fine, and using either get-tagassigment or get-rcistagassignment using the -entity filter works, but using either without a filter fails. unfortunately due to the way the code was written, using a filter on the get-tagassignment isnt easy to re-write

Below is the section of code that is calling the get-tagassignment and failing

foreach($tag in (Get-TagAssignment)){
    $tagCat += $tag.Tag.Category.Name
    $key = $tag.Entity.Name
    if($tagTab.ContainsKey($key)){
    ` $val = $tagTab.Item($key)
    }
    else{
        $val = @{}
    }
    $val.Add($tag.Tag.Category.Name,$tag.Tag.Name)
    $tagTab[$key] = $val
}

any assistance would be greatly appreciated

[LucD]

Are you able to restart that vCenter?

Which PowerCLI version are you running?

[JBartholomew]

VCSA has been restarted, and now the Get-rCISTagassigment is returning results, but it is taking quite a long time to return results

> Measure-Command {Get-rCisTagAssignment}

Days              : 0

Hours             : 0

Minutes           : 7

Seconds           : 16

Milliseconds      : 170

Ticks             : 4361700549

TotalDays         : 0.00504826452430555

TotalHours        : 0.121158348583333

TotalMinutes      : 7.269500915

TotalSeconds      : 436.1700549

TotalMilliseconds : 436170.0549

Here is the version info

> $psversiontable

Name                           Value

----                           -----

PSVersion                      5.1.16299.251

PSEdition                      Desktop

PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}

BuildVersion                   10.0.16299.251

CLRVersion                     4.0.30319.42000

WSManStackVersion              3.0

PSRemotingProtocolVersion      2.3

SerializationVersion           1.1.0.1

get-powercliversion

WARNING: The cmdlet "Get-PowerCLIVersion" is deprecated. Please use the 'Get-Module' cmdlet instead.

PowerCLI Version

----------------

   VMware PowerCLI 10.1.0 build 8346946

---------------

Component Versions

---------------

   VMware Cis Core PowerCLI Component PowerCLI Component 10.1 build 8377811

   VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 10.1 build 8344055

PS > get-module -name VMWare*

ModuleType Version    Name                                ExportedCommands

---------- -------    ----                                ----------------

Script     6.7.0.8... VMware.Vim

Script     10.1.0.... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}

Script     10.1.0.... VMware.VimAutomation.Common

Script     10.1.0.... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAd...

Script     10.1.0.... VMware.VimAutomation.Sdk            {Get-InstallPath, Get-PSVersion}

Now that the restart has gotten the rCIS Module to run, I am going to try to implement that in the script

[LucD]

When you are on PowerCLI 10.1.0, you can use the Get-TagAssignment , no need to use the cmdlets from my rCisTag module.

Curious if you see the same times with the PowerCLI native cmdlet (which also uses the REST API).

[JBartholomew]

That is the issue, when I run Get-tagassignment against this particular VCSA it fails with the error in the original post. The other 2 VCSA's run get-tagassignment fine.

[LucD]

I didn't get that.

So my Get-rCisTagAssignment does work against that vCenter?

[JBartholomew]

yes the get-rCISTagAssignment works against that vCenter after rebooting the VCSA, but the Get-Tagassignment does not. I am trying to schedule time to reboot the other VCSA's to see if I can get the rCIS to work their so I can get a comparison between runtimes on Get-Tagassignment and Get-rCISTagassignment

[JBartholomew]

So I am ultimately not sure why the Global Get-tagassignment has stated failing in this vCenter, but I have worked around the problem by calling the Get-tagassignment on a per VM basis using:

Foreach ($vm in $vms) {

    $VMTags += Get-TagAssignment -Entity $vm.name

    }

I had implemented the get-rcistagassignment in the code and it was failing out on VMs without any tags, so I felt that this was a reasonable work around in the mean time.

I should probably go back through the entire script to see if it can be streamlined, but at this point I dont really have a lot of time to re-write something that works.

[27560]

Get-TagAssignment yields 'read timed out'

Get-rCisTagAssignment doesn't yield anything for me

I'm exploring PoshRSJob, and breaking up my list of VMs into groups, and running a Get-TagAssignment in parallel

[LucD]

Does Get-rCisTagAssignment return anything when not run in parallel?
You can also try Kyle's newer VMware.Community.CISTag module.

[27560]

I've installed the CISTag module, and I am running the command Get-CISTagAssignment, its been running for ~10 minutes. Is there a quicker way to obtain all the tag assignments for all VMs in a particular datacenter? I've got ~3000 vms each with ~5 tag assignments each. Would attempting a parallel call be worth the effort?

[LucD]

The Get-TagAssignment cmdlet is an 'expensive' cmdlet.
Introducing parallelism would help, provided the cmdlet supports running in parallel.

Would using Get-VM -Tag xyz be an option?

You could do this in a ForEach loop over all tags returned by Get-Tag.

That would definitely be faster.

[27560]

This is also taking forever:

foreach($tag in Get-Tag){

    Get-VM -Tag $tag | Select Name, @{N='Category';E={$tag.Category}}, @{N='Tag';E={$tag.Name}}

}

[LucD]

I'm afraid the Tag cmdlets are not the fastest of the bunch :smileycry:

[27560]

Looks like I will forge ahead with a parallel command effort, I saw your article about parallelism here:
http://www.lucd.info/2015/03/17/powercli-and-powershell-workflows/PowerCLI and PowerShell Workflows - LucD notes

but since my server is offline, I have no way of using the auto-loading modules, and thoughts?

[LucD]

How do you mean?
The auto-loading happens on the station where you run the script.

I most of the time use Start-Job to run tasks in parallel.

[27560]

Woud you mind sharing an example to get me started? I've not used Start-Job before

[LucD]

There are a few examples in this community (just search for Start-Job).
For example Re: Trying to get start-job to work

[27560]

Using the following I was able to obtain over 15000 tags in ~15 minutes which should work for me:

Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false

$BIGLIST = 0..$(((get-vm | Measure-Object -line).lines) - 1)

$COUNTER = [pscustomobject] @{ Value = 0 }

$GROUPSIZE = [int](($BIGLIST.length)/10)      #### Can change 10 to a larger number to increase parallelism

$GROUPS = $BIGLIST | Group-Object -Property { [math]::Floor($COUNTER.Value++ / $GROUPSIZE)}

$JOBCMD = {

    $DATA = $input.getEnumerator()

    Import-Module VMware.VimAutomation.Core

    $CONNECTION = Connect-VIServer -Server $DATA[0] -Protocol https -User $DATA[1] -Password $DATA[2]

    Get-VM |

    Select-Object -Index ($DATA[3].Split('-')[0]..$DATA[3].Split('-')[1]) |

    Get-TagAssignment |

    Select Entity, @{N='Category';E={$_.Tag.Category}}, @{N='Name';E={$_.Tag.Name}}, @{N='Manager';E={$DATA[0]}} |

    ConvertTo-Csv -NoTypeInformation |

    Select-Object -Skip 1 |

    Set-Content "C:\Temp\VMTAG-$($DATA[3]).csv"

    Disconnect-VIServer $CONNECTION -Confirm:$false -Force:$true

}

foreach ($GROUP in $GROUPS) {

   $START = ($GROUP.Group[0] | Out-String) -replace "`t|`n|`r",""

   $END = ($GROUP.Group[-1] | Out-String) -replace "`t|`n|`r",""

   Start-Job -InputObject @($DATACENTER,$VCUSERNAME,$VCPASSWORD,"$START-$END") -ScriptBlock $jobcmd

}

[LucD]

Great!
And thanks for sharing