VMware Cloud Community
kenner
Contributor
Contributor

vCenter Service Status error

What does "Cannot access the health service!" mean? Where, precisely does that service run and how can it be restarted?

44 Replies
CH-Dave
Enthusiast
Enthusiast

Thanks for replying.

1) I am logged into the VCS as the domain administrator.

2) All my VMware Services are logging on as "Local System Account" except for the "VMware VCMSDS" service which was set to log on as "Network Service" or something like that. I changed it to "Local System Account", restarted the service and then exited, relaunched vSphere Client. No luck. Same problem. vCenter Service Status reports that it "Cannot access the health service".

3) I'm pretty sure the ports are not an issue as the health service was working fine. It stopped working when I had installed vCenter Servioer v4.0.0 Update 1.

4) Interesting.

5) No DNS issues that I know of. Again, was working fine. I already tried using IP addresses everywhere instead of "VCS02.RDLAB.COMPANYNAME.LOCAL" with no results.

6) Didn't mess with the SSL certs, and can't open a ticket at the moment as our support contract has lapsed. Pushing my director to renew it ASAP because I've been trying to troubleshoot this issue for over a week now and am pretty much out options. I needo work with someone who knows how this is all supposed to be properly set up.

Reply
0 Kudos
szymonunion
Contributor
Contributor

Hello,

we had the same problem now with vCenter 5.0. Plugins are enabled and in vCenter Service Status I got:

"Cannot access the health service"

I read all KB's about it, but still no success. We did not open SR yet.

Info about situation:

- w2k8R2, paths are fine (no need to change path in server.xml (tomcat) due to these two paths go to the same files.

- certs are fine, tomcat is listening on 8443, plugins are enabled

- localhost.log

SEVERE: Malformed moduleBaseURL:
java.net.MalformedURLException: no protocol:
at java.net.URL.<init>(URL.java:567)
at java.net.URL.<init>(URL.java:464)
at java.net.URL.<init>(URL.java:413)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.doGetSerializationPolicy(RemoteServiceServlet.java:196)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.getSerializationPolicy(RemoteServiceServlet.java:116)
at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.prepareToRead(ServerSerializationStreamReader.java:303)
at com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:234)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:162)
at com.vmware.vim.cimmonitor.gwt.server.GwtCimServiceImplementation.processCall(GwtCimServiceImplementation.java:232)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:85)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
2012-03-27 15:50:03 org.apache.catalina.core.ApplicationContext log
INFO: ERROR: The module path requested, null, is not in the same web application as this servlet, /vws.  Your module may not be properly configured or your client and server code maybe out of date.
2012-03-27 15:50:03 org.apache.catalina.core.ApplicationContext log
INFO: WARNING: Failed to get the SerializationPolicy 'FC7E86DEAEFD2AFEA8F62B491DE5C027' for module ''; a legacy, 1.3.3 compatible, serialization policy will be used.  You may experience SerializationExceptions as a result.

- Java is 1.6.0_29, but it was installed by VC installation program

- I downloaded newdata.zip and extracted to c:\Program Files\VMware\Infrastructure\tomcat\webapps\vws\ - there was no data\ catalog before. Files are 'archive' attribute and I am logged in as local administrator (rights)

- Services are running by LocalSystem

Any solutions?

Regards,

--

Simon

Reply
0 Kudos
Jellodyne
Contributor
Contributor

After upgrading to vCenter 5.1, we had this issue -- neither our vCenter Service Status plugin nor our vCenter Hardware Status plugin would load on our clients. Neither the Performance Overview pages nor our Storage View pages would not work. All services were running and working on the server. We're running a pretty stock setup with default security certificates.

I was able to resolve it on some clients by applying the Microsoft kb948963 (http://support.microsoft.com/kb/948963) patch per VMware article id 2031053 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203105...), The patch adds additional AES ciphers onto the client which allows the client to once again communicate to the tomcat on the vcenter server, which means any pages or plugins the vCenter Server delivers to the client via Tomcat will work again.

Unfortunately that kb fix is not available for Windows XP clients, but I was able to fix it for all clients by adjusting the available ciphers on the supply side.

The aggressivly secured tomcat server.xml file delivered with vCenter 5.1 (default path C:\Program Files\VMware\Infrastructure\tomcat\conf\server.xml) defines the SSL settings thusly:

<Connector SSLEnabled="true" acceptCount="100" ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_256_CBC_SHA, TLS_DH_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DH_RSA_WITH_AES_128_CBC_SHA, TLS_DH_DSS_WITH_AES_128_CBC_SHA" connectionTimeout="20000" executor="tomcatThreadPool" keystoreFile="${bio-vmssl.keyFile.name}" keystorePass="${bio-vmssl.SSL.password}" keystoreType="PKCS12" maxKeepAliveRequests="15" port="${bio-vmssl.https.port}" protocol="HTTP/1.1" redirectPort="${bio-vmssl.https.port}" scheme="https" secure="true"></Connector>

You'll note that all of the available ciphers are TLS + 128 or 256bit AES. Which is fine if your clients are okay with that, but as it turns out pretty much none of them are. I added some additional SSL cipers to the list on my server:

     <Connector SSLEnabled="true" acceptCount="100" ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_256_CBC_SHA, TLS_DH_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DH_RSA_WITH_AES_128_CBC_SHA, TLS_DH_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" connectionTimeout="20000" executor="tomcatThreadPool" keystoreFile="${bio-vmssl.keyFile.name}" keystorePass="${bio-vmssl.SSL.password}" keystoreType="PKCS12" maxKeepAliveRequests="15" port="${bio-vmssl.https.port}" protocol="HTTP/1.1" redirectPort="${bio-vmssl.https.port}" scheme="https" secure="true"></Connector>

Rebooted server, problem 100% fixed. Is it less secure? Well, maybe. A little. Not much. Do I have hackers trying to intercept our vm health status updates? Um, no.

EDIT - here's a nice link which describes the security of the ciphers in question http://www.techstacks.com/howto/j2se5_ssl_cipher_strength.html -- you'll note the ones I added are all 128 bit or better. I considered doing some experimentation to determine the minimum number of new ciphers I could add for my XP clients to work or if there was a TLS but-not-AES cipher (like maybe TLS_KRB5_WITH_RC4_128_MD5 or TLS_KRB5_WITH_RC4_128_SHA) which would work. But since my install is working and I have no objection to using 128 or 168 bit SSL to trasport trivial data inside my own network, I'll leave that as an exercise for the reader.

rceads
Contributor
Contributor

A tremendous "Thank You!" to Jellodyne for your post on the v5.1 vCenter Tomcat stuff!  I was in the same situation and have been unsuccessfully putzing with the same problems off and on for a couple of days and finally stumbled upon your post - - your links and SSL cipher additions fixed all of "it" for me. THANX!

Reply
0 Kudos
ermannoltt
Contributor
Contributor

Hello Jellodyne,

I've changed server.xml and plugins are working! Thank you

Now I've the same problem with inventory service (login to query service failed) that seems published with another tomcat istance on port 10443 but I don't see any server.xml into conf folder.

Massimo

Reply
0 Kudos