VMware Cloud Community
JasonGillis
Enthusiast
Enthusiast
Jump to solution

HostCtl exception in stats collection

Hi all,

I'm seeing a large number of errors (every 20 seconds) in my logs with the following signature:

Sep 20 22:10:00 host.com Hostd: [38840B90 verbose 'Statssvc.vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information


I don't see symptoms of this failure in working with the environment:  Everything seems fine, no errors, etc.  I've done as it suggested and looked in the VMkernel log, but I don't see any log entries there that look related or promising.  I've also done plenty of searches, and while a lot of community posts include logs that include lines similar, none of the posts I've found are focused on that particular error.

I'm evaluating VMware Log Insight and the sheer number of these errors makes it hard to clearly see other errors that might be more critical.


Does anyone have any insight into what this error means, or how to resolve it?


Thanks!

Jason



Reply
0 Kudos
1 Solution

Accepted Solutions
wagnerse1
Contributor
Contributor
Jump to solution

I was able to get rid these messages. On Servers which reported that message were several VMs configured with ISOs on Datastores as CD Device (not connected).

I changed it to "Client Device" and those messages are gone.

Hope this will help you.

This idea came from here.

Sebastian

View solution in original post

Reply
0 Kudos
11 Replies
marcelo_soares
Champion
Champion
Jump to solution

Are the statistics data being collected for this ESX? CAn you check on Performance tab if Real-time and historical data are being stored?

Marcelo Soares
Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

Yes, that data is being collected and stored.  The only "gap" I can see is that the rollup values for the Disk Number chart (queue depths) isn't present, although, the data is there on the real-time chart.

I won't be surprised if there's a metric that it is trying to collect but can't.  It just isn't very clear in helping figure out which one...

Jason

Reply
0 Kudos
marcelo_soares
Champion
Champion
Jump to solution

Can you try to restart management agents on the host? vCenter may be trying to get information abount an object (maybe a datastore) that is not present anymore on the system.

Marcelo Soares
Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

I will give that a try, but I can say that this does persist across reboots of the hosts, so I expect it won't change.  I'll report back shortly with results.

Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

OK, restarted host agents, and no change in the error.  It's still hitting every 20 seconds.

Reply
0 Kudos
marcelo_soares
Champion
Champion
Jump to solution

Yes... if a reboot didn't resovlve, the agents would not either. Well, to try to narrow down who is doing this, can you remove the ESX from vCenter to check if the error persists?

Another question: is this ESX an OEM installation or it was installed from an ISO from VMware? Is this host an HP host?

Marcelo Soares
Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

It would take some doing to be able to pull one of the hosts from vCenter.  I might be able to tackle that tomorrow morning.  I'm not sure if I mentioned it earlier, but this error appears on all the hosts in our vCenter.

The hosts are all Cisco UCS systems.  9 of them are B200M3 blades and 3 are C220 rack servers.  I believe that they were installed from the Cisco OEM ESXi 5.1 ISO rather than the VMware ISO.

Reply
0 Kudos
wagnerse1
Contributor
Contributor
Jump to solution

Did you solve the issue?

I got the same issue on 4 of our 24 Servers. I can not determine what the root cause is. All Servers are installed with HP OEM iso-file working on ESXi 5.0.0 Build 1117897.

Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

No, not yet.  We had a large number of vApps get deployed in our vCloud environment, which has pushed our memory usage to 90+% on all the hosts and has prevented me from being able to comfortably put a host into maintenance mode to work on it.  We've started the memory upgrade process, and I should be able to get back to this in the next day or two.

Jason

Reply
0 Kudos
wagnerse1
Contributor
Contributor
Jump to solution

I was able to get rid these messages. On Servers which reported that message were several VMs configured with ISOs on Datastores as CD Device (not connected).

I changed it to "Client Device" and those messages are gone.

Hope this will help you.

This idea came from here.

Sebastian

Reply
0 Kudos
JasonGillis
Enthusiast
Enthusiast
Jump to solution

PowerCLI Man showed up and helped me get through this in my environment.  After resetting all the CD drives, the errors have subsided.  A single VM with a mounted CD causes them to pop back up, though, which makes Log Insight a bear to work with due to the noise.

But, thank you for helping track down a solution!

Reply
0 Kudos