PowerCLI: Exception calling "CreateCollectorForTasks" with "1" argument(s): "The operation is not allowed in the current state."

[KLHadson]

Hi,

I have the following PowerCLI script that I've hacked together from numerous examples. The problem is it works for a while then starts repeating the following error before for every line.

error:

Exception calling "CreateCollectorForTasks" with "1" argument(s): "The operation is not allowed in the current state."

At C:\myscript.ps1:37 char:72

+             $taskCollector = Get-View ($taskMgr.CreateCollectorForTasks <<<< ($filter))

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : DotNetMethodException

script (myscript.ps1):

$VISRV = "xxxxxxxxxxxx"

Connect-VIServer $VISRV

Write-Host "VM Name,Snapshot Name,Snapshot Size,Date Created,Creator"

$myVMs = Get-VM

$VMsWithSnaps = @()

foreach ($vm in $myVMs) {

$snapshots = Get-Snapshot -VM $vm

    foreach ($snapShot in $snapShots){

        $VM = $snapShot.VM

        $Name = $snapShot.Name

        $Created = $snapShot.Created

        $Description = $snapShot.Description

        $Size = $snapShot.SizeMB

        #ignore VEEAM restore points

        if ((!($VM)) -or ($Name -match "Restore Point")){

        continue

        }

  if ( $snap.Created -lt (Get-Date).AddDays( -$Age ) ) {

    #getting the snapshot creator is a bit more complex.

            $taskMgr = Get-View TaskManager

            #define the number of tasks to return

            $numberofTasks = 999         

            #create a filter based on the snapshot creation time (5 seconds before $created)

            $filter = New-Object VMware.Vim.TaskFilterSpec

            $filter.Time = New-Object VMware.Vim.TaskFilterSpecByTime

            $filter.Time.beginTime = (($Created).AddSeconds(-5))

            $filter.Time.timeType = "startedTime"

            #get all the tasks based on the filter

            $taskCollector = Get-View ($taskMgr.CreateCollectorForTasks($filter))

            #rewind to the begin time specified in the filter (setting equal to blah just keeps it's output off the screen)

if ($taskCollector -ne $null)

{

$blah = $taskCollector.RewindCollector

#read in the first 999 tasks from our starting point (which is 5 seconds before $created)

$allTasks= $taskCollector.ReadNextTasks($numberofTasks)

#crank through those first 999 tasks

if ($allTasks -ne $null)

{

foreach ($task in $alltasks){

#find the createSnapshot  task for $VM

if ($task -ne $null)

{

if (($task.DescriptionId -eq "VirtualMachine.createSnapshot") -and ($task.EntityName -eq $VM)){

#Get the reason username which is our creator                      

$creator = $task.reason.username 

}

}

} 

}

}

                Write-Host $VM","$Name","$Size","$Created","$creator }

}

}

$VMsWithSnaps | Sort CreationDate

Regards,

[LucD · Accepted solution]

There is by default a maximum of 32 collectors one can use in a script/application.

You should remove the collector after you're finished with it.

See for example my Events – Part 8 – vMotion history post.

[RvdNieuwendijk]

According to the documentation for Managed Object - TaskManager this error is "Thrown if there are more than the maximum number of task collectors".  So you should keep track of this number and stop creating new collectors when the maximum number of collectors is reached until some old collectors have finished.

[KLHadson]

Thanks people, that was the problem

There's an issue with my script though - it sometimes doesn't correctly identify the creator of a snapshot - it gets it wrong and selects someone else - the wrong person.

@LucD: do you have any thoughts on this? I know you've dealt with the general problem of finding the creator of a snapshot before e.g. http://communities.vmware.com/message/1290894.

Thanks in advance!

Regards,

[LucD]

I don't immediatly see a reason why the wrong account would be picked.

But I would start by changing this

...

foreach ($vm in $myVMs)

$snapshots = Get-Snapshot -VM $vm

    foreach ($snapShot in $snapShots){

        $VM = $snapShot.VM

        .....

You seem to overwriting the loop variable inside the loop.

Although this is technically allowed, and could be useful in some situations, I would suggest to change 1 of the 2 variables to another name.

Just to avoid potential problems with the $vm variable

[LucD]

Additional question, is the "wrong" person an account that created a snapshot just before or after the snapshot concerned ?

[KLHadson]

Thanks LucD, I will try that.

Regarding your question, I will have to go through the events in more detail. But before that, let me just highlight what I'm seeing:

Script looks at snapshots for VM #1

<User account X selected for all snapshots. Seems correct>

Script looks at snapshots for VM#2

<User account Y selected for all snapshots. Seems correct>

Script looks at snapshots for VM#3

<My account selected for all snapshots. Seems correct>

Script looks at snapshots for VM#4

<No snapshots>

Script looks at snapshots for VM#5

<My account selected for all snapshots. Definately INCORRECT>

Script looks at snapshots for VM#6

<User account A selected for all snapshots. Seems correct>

Script looks at snapshots for VM#7

<User account B selected for all snapshots. Seems correct>

Script looks at snapshots for VM#8

<User account C selected for all snapshots. Seems correct>

Script looks at snapshots for VM#8

<User account D selected for all snapshots. Seems correct>

Script looks at snapshots for VM#9

<User account C selected for all snapshots. Seems HIGHLY IMPLAUSIBLE>

And so forth. There seems to be a trend that a user account that had been correctly matched to snapshots, is subsequently incorrectly matched. If it's selecting the wrong account, I suspect that means it's failing to find the correct snapshot task, and my filter criteria are not adequate, IMHO!

[LucD]

Try restting the $creator variable at the start of each snapshot loop.

...

foreach ($vm in $myVMs) {

$snapshots = Get-Snapshot -VM $vm

    foreach ($snapShot in $snapShots){

        $creator = ""

        $VM = $snapShot.VM

        ...

[KLHadson]

Hi LucD,

Setting $creator to "" at the start of each snapshot loop mostly solved the issue of the wrong user account being selected. However, I'm positive that it's still making at least one mistake.

Could we walk through how the following code actually finds the user that created a snapshot? It seems like it finds the nearest createSnapshot task related to the VM, to the time the snapshot was created? Perhaps there is another way todo this?

#getting the snapshot creator is a bit more complex.

$taskMgr = Get-View TaskManager

#define the number of tasks to return

$numberofTasks = 999         

#create a filter based on the snapshot creation time (5 seconds before $created)

$filter = New-Object VMware.Vim.TaskFilterSpec

$filter.Time = New-Object VMware.Vim.TaskFilterSpecByTime

$filter.Time.beginTime = (($Created).AddHours(-24))

$filter.Time.timeType = "startedTime"

#get all the tasks based on the filter

$taskCollector = Get-View ($taskMgr.CreateCollectorForTasks($filter))

#rewind to the begin time specified in the filter (setting equal to blah just keeps it's output off the screen)

if ($taskCollector -ne $null)

{

$blah = $taskCollector.RewindCollector

#read in the first 999 tasks from our starting point (which is 5 seconds before $created)

$allTasks= $taskCollector.ReadNextTasks($numberofTasks)

#crank through those first 999 tasks

if ($allTasks -ne $null)

{

foreach ($task in $alltasks){

#find the createSnapshot  task for $VM

if ($task -ne $null)

{

if (($task.DescriptionId -eq "VirtualMachine.createSnapshot") -and ($task.EntityName -eq $VM))

{

#Get the reason username which is our creator                      

$creator = $task.reason.username 

}

}

} 

}

}

The second issue is that this code is unable to find the user account for a number of snapshots, which currently shows up on my report as "Unknown" i.e. when $creator is $null. Any thoughts?

Thanks for all your help so far!

Regards,

[KLHadson]

OK so found the problem: the code gets the first 999 tasks that started at least just before the snapshot was created, but looking at my VCenter server, there might not even be tasks in the same year as when the snapshot was created i.e. no guarantee that a task has more or less the same created time as the snapshot.

Is there another way of determining who created a snapshot, that does not make the above assumption?

Regards

[LucD]

I used that method, the nearest in time, because the timestamp on a snapshot and the event it generates are not exactly the same.

The difference can be more important in a busier vCenter.

To be honest, I don't see many alternatives of finding the "creator" of a snapshot with the available information in the snapshot and the event.

Perhaps the ESXi logs would be an alternative, but most of the time the retention time on these logs is quite limited.

These "unknown" snapshot creators are most of the time system initiated snapshots.

A backup tool that uses snapshot to take a backup is a likely candidate.

[KLHadson]

HI LucD,

I am able to retrieve create snapshot events and their related users for all snapshots, but only manually via the vSphere GUI. Seeing that this is possible, I tried to do the same in a script:

$creator = $null

$events = Get-VIEvent -Entity $vm -MaxSamples 999 -Type info -Start $Start

foreach ($event in $events)

{

if ($event.vm.name -eq $snapShot.VM -and  $event.FullFormattedMessage -eq "Task: Create virtual machine snapshot" -and $creator -eq $null)

{

$creator = $event.UserName

}

}

However the Get-VIEvent procedure seems to always return events in decending order of time, from the current date. And it never includes the required snapshot events because it never returns enough events. It does this even if I specify the -Finish parameter. Do you know the correct way to retrieve events between a specific time window?

Thanks.

[LucD]

The order of the events can easily be changed with the Sort-Object cmdlet.

By default the Get-VIEvent cmdlet only returns 100 events, use the MaxSamples parameter to get more.

Something like this

$events = Get-VIEvent -Entity $vm -MaxSamples ([int]::MaxValue) -Type info -Start $Start |
  Sort-Object -Property CreatedTime  

 

[KLHadson]

Ok, that got the sorting right, but, for a particular snapshot, the earliest event it's able to retrieve occurs on 2012/10/09 11:43:48 AM, while the snapshot created date is 10/06/2011 14:04:51.

This seems wierd because if I extract the events to a txt file the vCenter GUI, I can see all events on 10/06/2011

[LucD]

Which data did you specify for the Start parameter ?

Normally the Get-VIEvent cmdlet and the export from the vSphere client should have give access to the same events

[KLHadson]

OK I was wrong - the earliest event available from the vCenter GUI occurs on 1/18/2012 2:56:22 PM. It seems events are deleted after a while. So if we had the events, it would work.

Thanks very much for all your help.

Regards,

[LucD]

Normally events are only deleted when you explicitely specify a retention period on the vCenter.

Check from the vSphere client under <Administration><vCenter Server Settings><Database Retention Policy>