VMware Cloud Community
ddd2020
Contributor
Contributor

ESXi host QueryEvents alternative EventHistoryCollector questions

I am not in a vcenter environment, we only have one esxi host.

I am trying to "adjust" the VCheck script to run cleanly on a ESXi host (it is wrote for a vcenter environment.)

I am working on the below error...

The Vcheck plugin script is 14 VMs restarted due to Guest OS Error

Exception calling "QueryEvents" with "1" argument(s): "The requested operation is not implemented by the server."

Below is the VCheck code...

$EventFilterSpec = New-Object VMware.Vim.EventFilterSpec

$EventFilterSpec.Category = "info"

$EventFilterSpec.Time = New-Object VMware.Vim.EventFilterSpecByTime

$EventFilterSpec.Time.beginTime = (get-date).adddays(-$HAVMresetold)

$EventFilterSpec.eventTypeId = "TaskEvent"

$HAVMresetlist = @((get-view (get-view ServiceInstance -Property Content.EventManager).Content.EventManager).QueryEvents($EventFilterSpec) | ?{$_.FullFormattedMessage -match "reset due to a guest OS error"} |select CreatedTime,FullFormattedMessage |sort CreatedTime -Descending)

I have researched and found the reason its failing is because of QueryEvents

The QueryEvents API is not supported on the ESX server. It would only work for the VC. Therefore instead of the QueryEvents API you might want to use the EventHistoryCollector for tracking the events using the help of EventFilterSpec data object and CreateCollectorForEvents method. Using these you would be able to query both the VC and ESX server.

I think I am going to have to create a CreateCollectorForEvents that runs continuously, and the adjust the vcheck script to use EventHistoryCollector & EventFilterSpec to check against the CreateCollectorForEvents?

I new to powercli and wanted to ask for help on this one.

To sum it up, the above vcheck script works on vcenter, I am trying to modify it to work on a ESXi host.

Can anyone offer powercli examples of using EventHistoryCollector in place of QueryEvents?

Thanks

17 Replies
LucD
Leadership
Leadership

Sure, try like this

$serviceInstance = get-view ServiceInstance

$eventMgr = Get-View $serviceInstance.Content.EventManager

$efilter = New-Object VMware.Vim.EventFilterSpec

$efilter.time = New-Object VMware.Vim.EventFilterSpecByTime

$efilter.time.beginTime = (Get-Date).Adddays(-7)

$efilter.time.endtime = Get-Date

$events = @()

$ecollectionImpl = Get-View ($eventMgr.CreateCollectorForEvents($efilter))

$ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

while($ecollection -ne $null){

   $events += $ecollection

   $ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

}

$ecollectionImpl.DestroyCollector()

$events

But be warned that the EventManager on an ESXi node doesn't "keep" the events for a long time!


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ddd2020
Contributor
Contributor

Thanks LucD!

But, I copied pasted your example into a test ps1 file and ran it, but I got the below error...

Exception calling "ReadNextEvents" with "1" argument(s): "A specified parameter

was not correct.

"

zdastest3.ps1:23 char:47

+ $ecollection = $ecollectionImpl.ReadNextEvents <<<< ($eventnumber)

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : DotNetMethodException

Should you example brought back something?

Reply
0 Kudos
LucD
Leadership
Leadership

Yes, you need to initialise the variable $eventnumber in the beginning.

It defines the number of events to be read in 1 call.

I usually set it to 100.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ddd2020
Contributor
Contributor

Thanks again LucD!

I added $eventnumber and set it to 100

That worked, sort of...

It pulled back 10 events...

I then went to my vsphere client console, highlighted my host in the upper left corner, then clicked on the Events tab.

I saw alot more than 10 events...

I then shutdown & restarted one of my test Vms...

I saw the events in the Events tab.

I then reran your script, and again it only pulled back the last 10 events. (I did see my stop & restart events!)

Forgive me for being blind ( i have stared at your code for 30 minutes), but I cant find why I am only pulling back just 10 events?

Do you have any suggestions?

Thanks



Reply
0 Kudos
ddd2020
Contributor
Contributor

LucD, I do have another question also...

Alan's vhcheck script filters on {$_.FullFormattedMessage -match "reset due to a guest OS error"}

And, I would also like to add two additional filters of:

A. When VM was powered off & time powered off...

- FullFormattedMessage : is powered off

-

CreatedTime      : 3/3/2015 5:26:23 PM

B. When VM was powered on & time powered off...

- FullFormattedMessage : is powered on

-

CreatedTime 

3/3/2015 5:26:23 PM

For a quick reference, below is Alan's code...

$HAVMresetlist = @((get-view (get-view ServiceInstance -Property Content.EventManager).Content.EventManager).QueryEvents($EventFilterSpec) | ?{$_.FullFormattedMessage -match "reset due to a guest OS error"} |select CreatedTime,FullFormattedMessage |sort CreatedTime -Descending)

Can you give me a quick pointer on where to add these filters in your code?

Thanks again so very much!

Reply
0 Kudos
ddd2020
Contributor
Contributor

Hi LucD...

I did a little more troubleshooting...

My problem is that I am only pulling back 10 events?

I adjusted:

$efilter.time.endtime = (Get-Date).Adddays(-1)

I did get 10 events from yesterday, so that part is working...

I then added two "write-output" to troubleshoot...

$eventnumber = 100

$serviceInstance = get-view ServiceInstance

$eventMgr = Get-View $serviceInstance.Content.EventManager

$efilter = New-Object VMware.Vim.EventFilterSpec

$efilter.time = New-Object VMware.Vim.EventFilterSpecByTime

$efilter.time.beginTime = (Get-Date).Adddays(-7)

$efilter.time.endtime = Get-Date

# $efilter.time.endtime = (Get-Date).Adddays(-1)

$events = @()

$ecollectionImpl = Get-View ($eventMgr.CreateCollectorForEvents($efilter))

$ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

write-output "ecollection is $ecollection" >> c:\util\vlog.txt

while($ecollection -ne $null){

   $events += $ecollection

   $ecollection >> c:\util\vlog.txt

   $eventnumber >> c:\util\vlog.txt

   $ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

}

write-output "ecollection is $ecollection" >> c:\util\vlog.txt

$ecollectionImpl.DestroyCollector()

$events

Here is my vlog.txt output...

+++

ecollection is VMware.Vim.UserLogoutSessionEvent VMware.Vim.UserLoginSessionEvent VMware.Vim.UserLoginSessionEvent VMware.Vim.UserLogoutSessionEvent VMware.Vim.UserLogoutSessionEvent VMware.Vim.UserLoginSessionEvent VMware.Vim.UserLogoutSessionEvent VMware.Vim.UserLoginSessionEvent VMware.Vim.UserLogoutSessionEvent VMware.Vim.UserLoginSessionEvent

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

IpAddress            : 10.

     :

IpAddress            : 10.

IpAddress            : 10.

100

ecollection is

+++

Note- to make it easier to read I purposely only kept the IpAddress field...

Note2-

Notice the top "ecollection is VMware" - there are 10 entries...

Notice the bottom "ecollection is" is empty (null)

So the script appears to be stopping because it hits a Null, but why?

Any help that you could provide would be very appreciated.

Thanks

Reply
0 Kudos
LucD
Leadership
Leadership

To what type of vSphere server are you connected, a vCenter or an ESXi node ?

With an ESXi connection you might see that behaviour.

The following will only return the eventtypes you requested, it seems to be working without an issue for me.

For each read of events, the number of events will be shown

At the end the script shows how many of each type where returned.

$eventnumber = 1000

$serviceInstance = get-view ServiceInstance -Server $global:DefaultVIServer

$eventMgr = Get-View $serviceInstance.Content.EventManager

$efilter = New-Object VMware.Vim.EventFilterSpec

$efilter.time = New-Object VMware.Vim.EventFilterSpecByTime

$efilter.time.beginTime = (Get-Date).Adddays(-1)

$efilter.time.endtime = Get-Date

$efilter.EventTypeId = 'VmPoweredOffEvent','VmPoweredOnEvent','com.vmware.vc.ha.VmRestartedByHAEvent'

$events = @()

$ecollectionImpl = Get-View ($eventMgr.CreateCollectorForEvents($efilter))

$ecollectionImpl.RewindCollector()

$ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

while($ecollection -ne $null){

   $ecollection.Count

   $events += $ecollection

   $ecollection = $ecollectionImpl.ReadNextEvents($eventnumber)

}

$ecollectionImpl.DestroyCollector()

$events | Group-Object -Property {$_.GetType().Name}

You might also want to have a look at my Get-VIEventPlus function in Get the vMotion/svMotion history and HA VM failover tracking

where I explain some more on event collectors and eventtypes.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ddd2020
Contributor
Contributor

Hi LucD

Thanks for your code, I will try it and let you know how it works.

To answer your question, we have one ESXi host (version 5.5 SP2), which is currently running 4 test VMs.

I use vsphere client to access our ESXi host.

In regards to your initial code only bringing back 10 events...

You mention, with a ESXi connection you might see that behavior.

Can you further explain?

And, will your new code also have the same issue of only bringing back 10 events?

(I will test and let you know)

I am just trying to understand why your initial code was only bringing back 10 events.

Thanks

Reply
0 Kudos
LucD
Leadership
Leadership

I'm afraid you will see the same behavior with the new code.

An ESXi node only retains Tasks and Events for a limited time (+/- 1 day), a vCenter collects these (and adds its own Tasks and Events) and stores them in the vCenter database.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ddd2020
Contributor
Contributor

Hi LucD,

Yes, I did notice that our ESXi host Events tab only shows events from today's date, and I also noticed that there are 100 events (I counted them)

So, using your original code, why are only 10 events returned when there are 100 in the Event Tab?

Do you possibly think we are hitting a Null on the While check before all 100 events were processed?

I am fine with only having one days worth of events to review, so I would really like to troubleshoot why your original code is only bringing back 10 events.

I know there are 100 events, why are only 10 brought back?

If you can think of any other troubleshooting thinks I can try, I would really appreciate that.

Thanks

Reply
0 Kudos
ddd2020
Contributor
Contributor

Hi LucD...

Here is a quick update...

I purposely remarked out the below line:

$efilter.EventTypeId = 'VmPoweredOffEvent','VmPoweredOnEvent','com.vmware.vc.ha.VmRestartedByHAEvent'

I wanted to see how many events were brought back first...

I got 489 events!!!

(Thanks for the counter number!)

So I wanted to let you know that your second code does work, in that it brings back more than just 10 events.

I dont know what is different between first & second code, but second code does work!

I then, purposely stopped and restarted one of my VMs...

Then, I added back in...

$efilter.EventTypeId = 'VmPoweredOffEvent','VmPoweredOnEvent'

And I pulled back 4 events!

(A stop and start from today, and a stop and start from yesterday!)

I am going to continue working with your second code to try and integrate it into Alan's vcheck script so that your second code results are part of Alan's vcheck results.

I will keep you updated and possibly ask more questions if I get stuck.

Thanks again so very very very much!

Reply
0 Kudos
LucD
Leadership
Leadership

Great, glad you got it working.

I have no idea what could explain the difference between the two versions.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ddd2020
Contributor
Contributor

Hi LucD...

Following back up to let you know that I got your code "plugged into" Alan's Vcheck script and everything worked perfectly!

I added a histcoll (history collector) counter variable to see the number of hits returned from the history collector so that I could display the hit count on Alan's report...

Kind of a sanity check, if 6 hits are displayed then you should see 6 entries listed.

Your code works perfectly and, you have made it possible for those without a vcenter environment to now include a modified plugin "14 VMs restarted due to Guest OS Error" in their Vcheck scripts!

Thank you LucD!!!

I have attached a printscreen of your code ran as a single job in Vcheck

Also, I wanted to ask if you would like a final copy of the code?

Maybe you could review it and post it here for others?

Thanks again, I really appreciate your help!

Reply
0 Kudos
GabCava
Enthusiast
Enthusiast

Hello Luc, is it still the case today with ESXi 6 or 6.5? I mean is there no way at all to  retrieve events from a single ESXi host that go beyond what one can see in the Events tab on the vSphere client? For some cases we are asked to spot some events after a week they occurred...

Reply
0 Kudos
LucD
Leadership
Leadership

I'm afraid not.

From the Understanding Events in the 6.5 SDK Programming Guide:

"Persistence of Event objects depends on the system setup.

Standalone ESXi hostsEvent objects are not persistent. Events are retained only for as long as the host system’s local memory can contain them. Rebooting a standalone ESXi host or powering off a virtual machine removes Event objects from local memory.

A standalone ESXi host might keep about 15 minutes worth of Event data, but this can vary depending on the processing load of the host, the number of virtual machines, and other factors.

Managed ESX/ESXi systems. Event objects are persistent. Managed ESX/ESXi systems send Event data to the vCenter Server system that manages them, and the vCenter Server system stores the information its database."


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

GabCava
Enthusiast
Enthusiast

Luc, many thanks

Reply
0 Kudos
LucD
Leadership
Leadership

You could run a scheduled task to collect them on a regular interval, but that would require a scheduling station and storage.

In any case, not something out of the box.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference