Generating Tag list, maintaining Entity-Tag-relationship and compare against array instead of Inventory database

[NelsonCandela]

Dear all,

I am - once again - in need of some guidance with a programming issue.

In my daily routines I need to execute a script to find VMs with speficic tags and add together disk space consumed depending on the datastore they reside on and on the tags that are attached to these VMs. The script in general works fine but it runs now a little more than 2 1/2 hours and I would like to speed this up.

Currently I'm running several foreach-loops, in each loop comparing and looking for specific tags and putting together a new variable to sum up disk space consumed per Tag.

Script:

$VMsWithTags = Get-Cluster "Cluster-Win-2008" | Get-VM -Tag "Backup-Enabled" | Sort-Object

$VmsWithTags | ForEach-Object {
    if (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_BLA*'}) {
        $BCK_BLA += $_.Name
        $BCK_BLA_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }

I'm doing this, as mentioned, in several loops and it takes very long as the list contains several hundred VMs.

So my idea is to simply generate ONE list as an array and write it to a file instead (if even needed!) and all the operations to compare should run against that newly generated list. I hope to achieve way better speeds because after the list in generated once all comparisons are done "locally" instead of against the vCenter Inventory database.

So in my mind generating the list would be initiated with this:

$VmsWithTags | ForEach-Object {
    Get-VM $_.Name | Get-TagAssignment
}

Result:

Tag                                      Entity                         ---                                      ------                         Operating System/Win2008R2_x64           Test-VM001                       System Type/Development                  Test-VM001                       Backup/Backup-Enabled                    Test-VM001                       Backup/BCK_VGA_WIN_SA                    Test-VM001                       Temporary Tag/WIN2008EOL                 Test-VM001                       Operating System/Win2008R2_x64           Test-VM002                     Backup/Backup-Enabled                    Test-VM002                     Temporary Tag/WIN2008EOL                 Test-VM002 ...   

Of course I would need to output that to a file. So far, so good.

If I could write that to a file but maintain the structure of a VM and its related Tags that would be great.

Yet again it shows I don't know how to do this as I am not sure if I need to work with more-dimensional arrays or how to put together a construct where this Tag-Entity-relationship is kept intact so I was counting on you guys helping me overcome this and be more effective with the time it takes for the script to generate the list I can compare against.

Thank you guys in advance!

BR

NC

[NelsonCandela]

SORRY!

Auto-format destroys the output in the above code section.

It's supposed to look like this when outputting this (or moving it to a more complicated array/variable):

[LucD]

I'm not sure I understand why you do the Get-VM with the Tag parameter, and then you do a Get-TagAssignment for each of those VMs.
In principle, you're doing the same query twice.

Unless there is some logic behind this that I don't get.

Also, try to limit the calls to Get-TagAssignment, which is an execution time expensive cmdlet.

Rather do 1 Get-TagAssignment for all VMs and then use a Group-Object to split the result up per VM (Entity.Name).

Another point, comparing the Tag property is slower than comparing with Tag.Name.
At least if that is what you testing there.

[NelsonCandela]

Hi LucD​,

thanks -- please let me clarify.

First of all I'm generating a list of those VMs that fit my query: Only VMs that have a specific Tag attached to them.

Then, with that newly generated list, I'm running several foreach-loops and do comparisons tasks for my disk space calculations.

I was under the impression that if I put everything in one bigger loop the script failed so I divided it into three or four smaller loops.

One loop looks like that:

$VmsWithTags | ForEach-Object {
    if (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC1_FR*'}) {
        $BCK_DC1_FR += $_.Name
        $BCK_DC1_FR_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
    elseif (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC1_SA*'}) {
        $BCK_DC1_SA += $_.Name
        $BCK_DC1_SA_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
    elseif (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC1_SU*'}) {
        $BCK_DC1_SU += $_.Name
        $BCK_DC1_SU_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
    elseif (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC2_FR*'}) {
        $BCK_DC2_FR += $_.Name
        $BCK_DC2_FR_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
    elseif (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC2_SA*'}) {
        $BCK_DC2_SA += $_.Name
        $BCK_DC2_SA_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
    elseif (Get-VM $_.Name | Get-TagAssignment | where {$_.Tag -like '*BCK_DC2_SU*'}) {
        $BCK_DC2_SU += $_.Name
        $BCK_DC2_SU_Usage += (Get-HardDisk -VM $_.Name | Measure-Object -Sum CapacityGB).Sum
    }
}

I know it's far from perfect and it's performing badly, that's why I would like to do it using a different approach.

To be more clear about what I want to do:

I want to loop through that list of VMs with all belonging tags only once and write it to a file. This file probably would look like the screenshot in my 2nd post.

Then I would want to loop through that file instead of looping against the Inventory database, which as mentioned and as you are aware of takes ages. Then I could load that file and do all the comparisons file-based which should be way faster.

Sorry if this still sounds complicated, I don't really know how to put that in a better context 😞

BR

NC

[LucD]

To help me understand, so you have VMs with 2 tags?
One tag is Backup-Enabled and the other tag can be any of those others you tested on (like *BCK_DC1_FR*)

Is that correct?

A side question, are there VMs which have Backup-Enabled and not one of those other tags?

The reason for asking, if there aren't, I don't really see the point of having the Backup-Enabled tag.

[NelsonCandela]

Hi LucD​,

One tag is Backup-Enabled and the other tag can be any of those others you tested on (like *BCK_DC1_FR*)

Is that correct?

that assumption is correct, but shows not all.

The environment holds roughly 1100 VMs.

~700 of these VMs have the tag "BCK_Enabled" attached to them.

~100 of these VMs have the tag "BCK_Disabled" attached to them.

The rest (~ 300 VMs) doesn't have either tag but may have other tags attached!

(It's a customer's environment so I cannot show all or have to censor it.)

So the first step is to get all information only for those VMs that have the tag BCK_Enabled. And only on that list I'm filtering for other VMs that also have additional tags describing their backup schedule or datastore locations.

Now you may be right that a different approach would be better and the environment is not in my hands when it comes to maintaining tags or finding better way to put things; I'm now looking simply for a more elegant and mostly quicker way to filter for those VMs and tags.

Thanks!

BR

NC

[LucD]

Does this produce what you are after?

Note that you will have to update the Category name to which these tags belong (I used 'Backup' in the example)

Get-VM -Tag 'BCK_Enabled' | Get-TagAssignment -Category Backup |

Group-Object -Property {$_.Tag.Name} -PipelineVariable group |

where{$group.Name -ne 'BCK_Enabled' -and $group.Name -match "BCK_"} |

ForEach-Object -Process {

    New-Object -TypeName PSObject -Property ([ordered]@{

        Tag = $group.Name

        Entity = $group.Group.Entity.Name -join '|'

        SizeGB = $group.Group.Entity | Get-HardDisk | Measure-Object -Property CapacityGB -Sum | select -ExpandProperty Sum

    })

}

[NelsonCandela]

Dear LucD​,

thanks for your suggestion - unfortunately by the result it it's not exactly what I'm looking for.

Your assumption is right to browse within the Category "Backup", nevertheless is the output in Powershell ISE cut off and not exactly what I meant.

Please don't worry as of yet, I'll try to create an example.
I know you most likely find better ways to create the query and gather the info but I'm really looking more for the programming part of this.

I'll refine my previous posts and add something; now I'm in a hurry, sadly, but thanks again, I'll pick this up asap.

BR

NC

[LucD]

Save the output in a CSV, then you should see the complete content for each property.