kenner
Contributor
Contributor

vCenter Service Status error

What does "Cannot access the health service!" mean? Where, precisely does that service run and how can it be restarted?

44 Replies
kenner
Contributor
Contributor

Using the URL directly in the browser is actually not meant to

work. This is because clients need to authenticate before accessing

the data. So simply using the browser will not work. The VI Client

does this authentication.

I didn't realize that because I was getting the same result in both.

So I guess I should get very methodical. Here's what I did:

(1) Stopped vCenter and Tomcat.

(2) Replaced the data directory with the one here.

(3) Started vCenter and Tomcat.

(4) Started the UI and selected status.

I got "An error occurred, please try again in another vSphere session."

I zipped up the data directory and will add that and the logs into

the SR once the support portal is back up.

0 Kudos
kenner
Contributor
Contributor

The portal came back up, so I uploaded both files to the SR.

0 Kudos
arindamiitk
Contributor
Contributor

What happens if after Step 4 above, you start a new UI client and select the VC service status? Does it still show the same error?

0 Kudos
kenner
Contributor
Contributor

What happens if after Step 4 above, you start a new UI client and

select the VC service status? Does it still show the same error?

It DID. I started a few clients. But I just did it from a client I started

a couple of hours ago and now it works! And the "Hardware Status" for

hosts work too, confirming they were the same problem.

Maybe I just needed to wait some number of hours after doing that.

So it seems that this workaround does work. I hope you can find what went

wrong from the logs and zip files I sent.

0 Kudos
arindamiitk
Contributor
Contributor

Cool. Yes, the later problem that you encountered ("An error occurred, please try again in another vSphere session", not the earlier unable to login/Xhive problem) is a known issue which we will fix in an update. The workaround for now is to simply start a new UI client.

We are looking at the original problem you had due to Xhive not loading up and this may be due to a bug in the Xhive version that we shipped with. We are still working on that.

cheers, Arindam

0 Kudos
kenner
Contributor
Contributor

Yes, the later problem that you encountered: "An error occurred,

please try again in another vSphere session", not the earlier unable

to login/Xhive problem) is a known issue which we will fix in an

update. The workaround for now is to simply start a new UI client.

I could have sworn I did, a few times, but I guess not. Thanks very

much for your help here. I think the two SRs should be closed to

avoid having the support organization waste more time on this.

0 Kudos
Atmos4
Enthusiast
Enthusiast

I have the same problem, with both vCenter Service Status and vCenter Hardware Status plugins not working. vCenter Service Status also displays "Cannot access the health service!", Hardware Status displays: "Error occured while trying to update the host data".

I tried exchanging the data directory and restarting vCenter + Tomcat, but that didn't help. FIles in the data directory do not get updated.

I noticed the following error in catalina.log every time I try to show one of the plugins:

Jul 1, 2009 1:37:42 PM org.apache.tomcat.util.http.Parameters processParameters

WARNING: Parameters: Invalid chunk ignored.

Though some googling shows, that this warning is probably harmless.

0 Kudos
kenner
Contributor
Contributor

I have the same problem, with both vCenter Service Status and vCenter

Hardware Status plugins not working. vCenter Service Status also displays

"Cannot access the health service!", Hardware Status displays: "Error

occured while trying to update the host data". I tried exchanging the

data directory and restarting vCenter + TOmcat, but that didn't

help. FIles in the data directory do not get updated.

If you read the thread carefully, you'll see that one of the versions

of those file was incorrectly set readonly. So either change to read/write

or use the correct version of those files.

0 Kudos
Atmos4
Enthusiast
Enthusiast

I found this additional error in Windows Application log:

Initialization error. Could not initialize certficate I/O error: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

I am using a a PKCS12 certificate created by Windows Certificate Signing Authority. I did specify keystorePass in server.xml of Tomcat to the correct password for the rui.pfx, which was required for VMware Management Server (Tomcat) to work. But it seems some other parrt of Tomcat still doesn't like the cert.

The keystorePass password specified in server.xml is correct, I could import the rui.pfx into windows keystore by using that password.

0 Kudos
Atmos4
Enthusiast
Enthusiast

I tried switching the paths in server.xml, web.xml and config.xml to a backup of the certs that came with vCenter, but this didn't fix the error. I also reset the keystorePass to testpassword, from the key required for my custom cert.

0 Kudos
Atmos4
Enthusiast
Enthusiast

The only way I found to fix the problem was to switch the whole vCenter to the default SSL certificates. After that I had to reconnect all hosts because of the cert mismatch.

Now vCenter Hardware Status, vCenter Service Status and Storage Views work.

This SSL issue really should be fixed, running with default certs is not a good ida from a security point of view.

0 Kudos
Atmos4
Enthusiast
Enthusiast

I have the same problem, with both vCenter Service Status and vCenter

Hardware Status plugins not working. vCenter Service Status also displays

"Cannot access the health service!", Hardware Status displays: "Error

occured while trying to update the host data". I tried exchanging the

data directory and restarting vCenter + TOmcat, but that didn't

help. FIles in the data directory do not get updated.

If you read the thread carefully, you'll see that one of the versions

of those file was incorrectly set readonly. So either change to read/write

or use the correct version of those files.

Of course I used the later posted data.zip and also made sure it wasn't read-only. But as my later posts indicate this wasn't my problem.

0 Kudos
BjarteGjerde
Contributor
Contributor

I had the exact same errors using custom a SSL certificate, but managed to solve the problem by recreating the rui.pfx file with 'testpassword' as the password (and resetting this in server.xml).

It seems to me that this password is written in plain text in the tomcat server.xml file used by most of the webservices but it is also hardcoded in vws\data\ xhiveDatabase.bootstrap file used by the healthservice.

Also putting SSL cert password in plain text in the server.xml file is also not i good idea from a security point of view ...

0 Kudos
ThorstenT
Contributor
Contributor

Hi,

having the SSL password in plain text on the server is no problem at all, since the unencrypted SSL key is placed in the same directory. There is no other way than having the unencrypted (or trivially crypted) key on the server. You would otherwise have to provide the key manually at each server restart.

Then again, using custom certs causes massive pain, last but not least due to the lack of documentation. It took me a whole day to figure out, how to create a PKCS12 file, that vCenter's Tomcat finally would accept. I thought of server.xml but not that Xhive would use the same key and certificate.

I would greatly appreciate an overview which component of vSphere uses which kind of certificate for which purpose. Another great improvement would be support for certificate chains. It is quite a mess to install custom certificates that are issued by intermediate CAs. You have to manually install the intermediate CAs certficiates on both the vCenter and on all clients to accomplish a secure SSL handshake.

Regards,

Thorsten

0 Kudos
zodal
Contributor
Contributor

Hello,

I would like to know what is the correct way how to create pfx file for the vcenter. Now I have working Hardware Status and Storage Views, but the Health Status is still not working. I followed this howto: http://www.vm-help.com/esx/esx3i/change_name_and_cert.php for my certs and how to get them into ESXi 4. After this I changed the data.zip (last one) and restarted VC. Still not working.

Thank you for help,

Tomas

0 Kudos
uma_kits
Contributor
Contributor

After I upgrade vCenter Patch 1, it shows an error same as you all. I have to uninstall vCenter and reinstall without patch 1. It's work for me.

0 Kudos
uma_kits
Contributor
Contributor

vCenter Server update 1 can help me solve this issue.

0 Kudos
skywalkr
Contributor
Contributor

Hi Guys,

There were several facets of the issue where the Performance Overview and Heath services were not functioning which we have been working on with VMware. The root issue was the password used on the SSL certificate MUST remain as "testpassword" at this time and changing it will not allow the tomcat web services to start listening on port 8443. However there were other related issues we created during debugging which I've outlined below, hopefully saving us and others grief in the future. The other root issue is the path for the SSL certs must also be updated if running on Windows 2008/64 bit etc.. The default install uses the the wrong path.

In our earlier attempts to repair, we incorrectly installed the correct version Java/JRE which VMware specifies into an incorrect location based on either a faulty KB article (1016049) or an error in setup. For tomcat, we had a JAVA path which contained "...jre\jre" included in it. To fix, VMware support removed the prior JAVA setup and then transferred the correct Java version to the correct location and updated the tomcat configuration to match.

The Java version we installed initially was not the 5.0.200.2 which is what VMware specifies must be available in a specific location. Using any other version is UNSUPPORTED and according to vmware support, we should not have to tinker with this version which installs with virtual center at all since it is buried in the setup.

We manually removed Java 6.17 from the machine.

In the end the problem with the hardware status tab and vCS Health Services not working was related tomcat not listening on port 8443 because it was not started to listen on port 8443. It was not conflict with any other service listening on 8443. The reason is linked to when we replaced the default SSL certificates on initial setup. In the VMware KB articles and PDF SSL examples, they show the rui.pfx file having a password of "testpassword" and in most instances, people want to change this password, using their own. However in this linked KB article it clearly states that you are NOT allowed to change the password --> http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101232... After we put the password back as specified and fixed the other items, the related services are now working in vCS. This needs to be clearly noted in the v4 VMware SSL PDF.

The default PFX path in the tomcat's server.xml (C:\Program Files (x86)\VMware\Infrastructure\tomcat\conf) is not specified correctly when running on Windows 2008/64 bit. You must update the path in this control file to the location of the actual SSL file as:

For Windows 2008/64 bit --> C:\Users\All Users\VMware\VMware VirtualCenter\SSL

Original file says...

lientAuth="false" sslProtocol="TLS"

keystoreFile="C:\ProgramData\VMware\VMware VirtualCenter\SSL\rui.pfx"

keystorePass="testpassword" keystoreType="PKCS12"

Once we reset the password using this KB article --> http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101232... ...so we did not have to regenerate the base CRT file, we updated what I thought was the correct tomcat server.xml file. However, being this is Windows 2008, we were not paying close enough attention to the layouts of the windows and instead of updating the server.xml file used by tomcat, we were updating the server.xml.xml backup file which was a backup for an older revision. Review the attached screen cap to see how easy it is to make this mistake. The Vmware support guys even made this mistake until we figured it out so I didn't feel quite as dense.

Lessons learned:

1) at this time, you must use "testpassword" as the password for the SSL certificates.

2) when working in Windows 2008, pay close attention to the window layouts as filenames are displayed as filename.extension.

3) http://www.vmware.com/files/pdf/vsp_4_vcserver_certificates.pdf needs updates and revs to handle Windows 2008 / 64 and vSphere 4.x installs, gotchas, etc..

Later,

GC Mobley, IBM Tivoli

Later, GC Mobley
0 Kudos
CH-Dave
Enthusiast
Enthusiast

CG,

Great post, and great information. Timing could not have been better as your post is less than 48 hours old when I came across it. I made the path corrections to the correct SERVER.XML file and rebooted my vCenter Server (currently v4.0.0. Update 1) but no luck. I still see Cannot access the health service! when I select vCenter Service Status and when I select an ESX host, the Hardware Status tab has a pop-up error message stating Do not have permission for this command.

I've been working on this problem for several days now and still cannot seem to fix it. Here's a quick recap to what brought me to this point:

  • Our original VCS (named "VCS01") was running Win 2003 and SQL 2005, and was solely used for Lab Manager 4.

  • Two weeks ago, I created a new and improved VCS (named "VCS02")using Win 2008 and SQL 2008.

  • I took the old VCS01 offline, copied the SQL database over to VCS02 and then brought VCS02 online.

  • There were a few minor kinks here and there, but VCS02 has been running now for 2 weeks just fine.

  • Using Update Manager, I updated and patched all 9 ESX and ESXi hosts in our Lab Manager cluster to be current.

  • We then updated VCS to v4.0.0 Update 1.

  • I then updated Update Manager to the latest as well. So now the entire infrastructure has been patched and updated.

  • Around this time I noticed that the vCenter Service Status was not working. I've tried everything I've found in multiple threads and KB articles including replacing the entire DATA folder with the one in the DATA.ZIP file and made sure it was all read-only. I even gave it ample time to rebuild the XHIVE database.Still no luck.

In my vCenter Server Settings, I currently have the following set:

I'm now stuck at the same point and have pretty much run out of options. I'm totally open to suggestions.

Instinct tells me to look inside log files, so I just checked out the "C:\Program Files (x86)\VMware\Infrastructure\tomcat\logs" folder and found the "localhost.2010-01-13.log" which has this error in it repeatedly:

Jan 13, 2010 4:57:42 PM org.apache.catalina.core.ApplicationContext log
SEVERE: Malformed moduleBaseURL:
java.net.MalformedURLException: no protocol:
at java.net.URL.<init>(URL.java:566)
at java.net.URL.<init>(URL.java:463)
at java.net.URL.<init>(URL.java:412)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.doGetSerializationPolicy(RemoteServiceServlet.java:196)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.getSerializationPolicy(RemoteServiceServlet.java:116)
at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.prepareToRead(ServerSerializationStreamReader.java:303)
at com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:234)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:162)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:85)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:595)
Jan 13, 2010 4:57:42 PM org.apache.catalina.core.ApplicationContext log
INFO: ERROR: The module path requested, null, is not in the same web application as this servlet, /vws. Your module may not be properly configured or your client and server code maybe out of date.
Jan 13, 2010 4:57:42 PM org.apache.catalina.core.ApplicationContext log
INFO: WARNING: Failed to get the SerializationPolicy 'A54E696C43E49725CD8446E4171EA2C4' for module ''; a legacy, 1.3.3 compatible, serialization policy will be used. You may experience SerializationExceptions as a result.

0 Kudos
skywalkr
Contributor
Contributor

Some of the steps we did to resolve..

1) Make sure you are logged in with an ADMIN id

2) Make sure your VMware services are using System.. I had used VCLOGIN to start my services and the support guy didn't like that..not sure it was ever related to the issue. I have not put it back to test yet..

3) Ensure NO other services are using those tomcat ports for heath services...

4) The Java install for this is buried down in the vc directory structrue and must be a very specific version 5.022.x which should have been installed when you installed VC 4.x. The engineer replaced my entire java structure there as part of PD. There was a KB article, which I mentioned, that had been pulled as of my original posting which explained this. I'm not sure it has been re-released.

5) Make sure the DNS resolution is working. There are some KB articles which discuss which files to change to IP only to verify.. but if you have a DNS issue, it will cause problems.

6) Open a support ticket. The engineer on this issue was very knowledgeable and said he had handled quite a few of these and most times it was related to the JAVA. unless you messed w/ the SSL certs..

That's all I can think of for now.

Later,

GC Mobley, IBM Tivoli

Later, GC Mobley
0 Kudos