Esxi 5: Hardware monitoring service on this host is not responding or not available

[Andy90]

Hello,

Sometimes we have the phenomenon that when we open the Hardware Status Tab we can not see any information.

In return I only get the following message:

Hardware monitoring service on this host is not responding or not available

(Also the "Update"-function doesn't effect anything)

Funnily enough, it could be that in 30 minutes it will work fine.

Has anyone an idea what I can check / proof to solve this problem?

[VTsukanov]

Problem looks as  the connection problem with tomcat service

You can check your firiwall settings ( test URL https://<your_vcenter>:8443/cim-ui/scriptConfig.xml or https://<your_vcenter>:8443/cim-ui/index.html ) and try to find additional information about problem in tomcat logs ( directory C:\Program Files\VMware\Infrastructure\tomcat\logs\ )

[Andy90]

I tried the mentioned URLs:

• https://<your_vcenter>:8443/cim-ui/scriptConfig.xml --> Successfully
• https://<your_vcenter>:8443/cim-ui/index.html --> Error messages appeared

for the "index.html" site, the following error message appeared:

Also here is a short snippet of the vctomcat-stderr.2012-01-05.log:

INFO: Got a matching thumbprint

04.04.2012 09:27:40 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 09:28:13 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 09:28:36 org.apache.tomcat.util.http.Parameters processParameters

WARNUNG: Parameters: Invalid chunk '' ignored.

04.04.2012 09:28:37 org.apache.tomcat.util.http.Parameters processParameters

WARNUNG: Parameters: Invalid chunk '' ignored.

04.04.2012 09:28:47 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 09:29:20 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 09:29:53 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

Apart from that, I couldn't find any further error messages in this log.

[VTsukanov]

• https://<your_vcenter>:8443/cim-ui/index.html --> Error messages appeared

for the "index.html" site, the following error message appeared...

This is normal reaction when you try to access via browser

It looks hardware monitoring is working now

I think you should wait until you get an error message in vCenter and then check and logs and access URL

[Andy90]

Right now, i checked another ESXi Server, by chance there was the same error:

and here is the vctomcat-stderr.2012-01-05.log file in this time-range

04.04.2012 10:01:23 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:01:56 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:02:29 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:03:03 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:03:36 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:04:09 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:04:42 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

04.04.2012 10:05:15 com.vmware.vim.common.ssl.AuthSSLX509TrustManager checkServerTrusted

INFO: Got a matching thumbprint

Please give me any further advices to fix this problem

[VTsukanov]

Ok

Take a look at ESX Hardware Status logs file ( Usually C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vws.log ) for  error

[Andy90]

I checked the file, but I couldn't find any error.

Enclosed you can find a snippet of the vsw.log

[VTsukanov]

Andy wrote:

I checked the file, but I couldn't find any error.

Message "Couldn't contact CIMOM on <DEVICE_NAME>/host-164539 "  indicates that there is a connection problem (between ESXi host and vCenter)

take a look at vmware kb The Hardware Status tab displays the error: Hardware monitoring service on this host is not respondi...

and another note :

Health state poll on <DEVICE_NAME> /host-249 took 28922 (ms)

shows that the receipt of monitoring data has been slow (30 sec) The reason may be the workload on your host

[Andy90]

I think this is the entry, as i pushed the button "Update" for the ESXi on the vCenter:

[2012-04-04 12:02:42,435 http-8443-8  INFO  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] Cancelling future health poll on <DEVICE_NAME>/host-162691

[2012-04-04 12:02:42,435 http-8443-8  INFO  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] Refresh request on <DEVICE_NAME>/host-162691 enqueued

[2012-04-04 12:02:42,466 pool-56-thread-10  WARN  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] Couldn't contact CIMOM on <DEVICE_NAME>/host-162691

[2012-04-04 12:02:42,466 pool-56-thread-10  INFO  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] scheduled host-162691 for health poll in 2700000 millis

Also I can see the following errors in the log, but I can't see from which server this error appears:

[2012-04-04 12:04:04,748 Thread-36  ERROR com.vmware.vim.health.impl.XmlUtil] Error validating health from stream

[2012-04-04 12:04:04,748 Thread-36  ERROR com.vmware.vim.health.impl.XmlUtil] Invalid health data: org.apache.commons.httpclient.AutoCloseInputStream@2dc13955

org.xml.sax.SAXParseException: Premature end of file.

     at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)

     at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)

     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)

     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)

     at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)

     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

     at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

     at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)

     at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)

     at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:124)

     at com.vmware.vim.health.impl.XmlUtil.getDocumentFromStream(XmlUtil.java:119)

     at com.vmware.vim.health.impl.ComponentSpec.retrieveHealthFromUrl(ComponentSpec.java:323)

     at com.vmware.vim.health.impl.ComponentSpec.retrieveHealth(ComponentSpec.java:266)

     at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealthFromUrl(HealthPollerImpl.java:116)

     at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:103)

     at com.vmware.vim.health.impl.HealthPollerImpl.computeHealth(HealthPollerImpl.java:185)

     at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:101)

     at com.vmware.vim.health.impl.HealthPollerImpl.pollHealth(HealthPollerImpl.java:84)

     at com.vmware.vim.health.impl.HealthPollerImpl.access$100(HealthPollerImpl.java:28)

     at com.vmware.vim.health.impl.HealthPollerImpl$PollerThread.run(HealthPollerImpl.java:54)

     at java.lang.Thread.run(Thread.java:662)

The curious fact is, that all ESX 4.1 Servers in our virtual environment haven't got this problem... Only the new ESXi 5.0 server (3 server) are having this problem.On one (I used for testing), there are no VMs running. So it could be a performance / workload problem.

Also I don't think about a firewall-problem, because just round about 20-40 minutes later, the hardware status updates by hisself und I can see all required information... Just randomly, I can't see these hardware states.

[VTsukanov]

1) You can try to find the problematic host, by analyzing the log files hosts (hostd.log & syslog.log). So you can execute command

grep "CIM error" /var/log/*.log 

or

grep "CIM" /var/log/*.log 

over ssh shell.

2) There may be many reasons for this error (certificates, permissions etc) ... Try first to find the problematic host

> Also I don't think about a firewall-problem ....

Yes, it is unlikely

[Andy90]

Right now I checked all the logs of /var/log/ but I couldn't find any error or something else.

Also I checked the logs for the time stamp I tried to update the Hardware Status, also without any error.

[VTsukanov]

Since the problem appears from time to time - no idea

You can contact vmware technical support or more detail investigate log files on hosts & vcenter

(+)

Also you can enable debug logging for vws

1. Using a text editor, open the file log4j.properties.
This file is located in C:\ProgramFiles\VMware\Infrastructure\tomcat\webapps\vws\WEB-INF\classes.
2. Change these parameters:

• log4j.logger.com.vmware.vim=DEBUG
• log4j.appender.LOGFILE.Threshold=DEBUG
• log4j.appender.LOGFILE.MaxBackupIndex=1000

3. Restart vcenter services

(++)

Also take a look at vmware kb vCenter Server Health status reports the error: Error retrieving health from url

[Andy90]

Valeriy Tsukanov wrote:

......

(++)

Also take a look at vmware kb vCenter Server Health status reports the error: Error retrieving health from url

On my vCenter Server there's running the Symantec Endpoint Protection.

I stopped all virsscanner processes / services, but Hardware Status is still not available for some ESXi 5.0 hosts.

[VTsukanov]

Do you can temporarily uninstall Symantec Endpoint Protection & enable debug logging?

[Andy90]

There is no different between deactivating the scanner and uninstall the scanner.

When I deactivate it, nothing will be blocked anymore.

Also it's not the point, because sometimes I can see the Hardware status and sometimes not.

(And no one is deactivating the scanner at this time).

Any further solutions?

[VTsukanov]

The difference between deactivation and uninstall of the scanner is that when you activate/deactivate the scanner, the kernel modules of scanner remain in the memory computer.

There no other suggestions yet.

[Andy90]

It's no use to uninstall the scanner.

I tried it, without a success!

Hardware Status still doesn't appear.

[VTsukanov]

try to enable debug logging (settings to log4j.properties on the previous page) and check logs

[Andy90]

We fixed this problem by updating the vCenter to the current Version and also updated the affected ESXi 5 Servers to the following build no. 702118.

Thanks

Edit 12-07-2012:

The problem is still available... Sometimes we can't access the hardware status.

[cuxport]

we had/have the same problem and vmware support told me that the hp-smx-provider seems to be the problem (due to the log files it kind of crashed). I updated it to the latest version, for now on no more problems (~2 weeks) but also havent had problems before for a period of a few months, so we will see....

But maybe that will help somebody else!