SSL Error Messages after vSphere upgrade

[rbeu]

I upgraded to vCenter Server 4.0.0 today and the following plugins don't load when launching the VI client - vCenter Hardware Status and vCenter Service Status. The error message in the plug-in Manager window is "The following error occured while downloading the script plugin from https://servername:8443/cim-ui/scriptConfig.xml: Unable to connect to the remote server." The performance overview tab also doesn't work either. After some digging through the logs I have determined that when the Management Webservices service is starting it is generating SSL errors. In the vpxlicent log I see several "SSL Validation failed in WebRequestPatch" messages followed by "No connection could be made because the target machine actively refused it" when it tries to download the various scripts for the plugins. Furthermore, in the tomcat log, catalina.2009-07-23, I see this when the Webservices service is starting:

SEVERE: Error initializing endpoint

java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

Followed a few seconds later by this (after many more SSL error messages similar to the one above):

INFO: Stopping Coyote HTTP/1.1 on http-8443.

I am assuming that the last line is the reason why I can't connect on port 8443 and why a netstat doesn't show anything listening on that port on my VC box. I know my certificates work (to some extent) because if I connect over https on port 443 to my VC box it works fine. It's only when trying to connect to 8443 that this problem occurs. I have already tried the steps in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101064... even though I'm not getting any messages which point to name resolution problems and it didn't help.

Anyone have any ideas?

[mrohen]

At least, we are not alone - I have absolutely the same issue.

Went the same way up to the padding error,since have no idea what to do.

[rbeu]

Did you replace the default cert on the VCenter box with your own? If so, did you use internally signed or signed by a commercial CA? We have another environment here that was upgraded and it's working fine. One of the differences is that the default cert was never replaced after the orginal 2.0 install for the environment that's working.

[mrohen]

Yes, i've replaced default cert by commercially signed one (equifax) and it works just fine on 443 port.

catalina log filer contains such messages:

SEVERE: Coyote connector has not been started

Jul 27, 2009 5:01:22 PM org.apache.catalina.core.AprLifecycleListener init

INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\VMware\Infrastructure\tomcat\bin;.;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Microsoft SQL Server\90\Tools\binn\

Jul 27, 2009 5:01:23 PM org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080

Jul 27, 2009 5:01:23 PM org.apache.coyote.http11.Http11Protocol init

SEVERE: Error initializing endpoint

java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

at com.sun.net.ssl.internal.ssl.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1275)

at java.security.KeyStore.load(KeyStore.java:1150)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:319)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:259)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:410)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:378)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:125)

at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:496)

at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:177)

at org.apache.catalina.connector.Connector.initialize(Connector.java:1059)

at org.apache.catalina.core.StandardService.initialize(StandardService.java:677)

at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:792)

at org.apache.catalina.startup.Catalina.load(Catalina.java:518)

at org.apache.catalina.startup.Catalina.load(Catalina.java:538)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:260)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:412)

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded

at com.sun.crypto.provider.SunJCE_h.b(DashoA12275)

at com.sun.crypto.provider.SunJCE_h.b(DashoA12275)

at com.sun.crypto.provider.SunJCE_ac.b(DashoA12275)

at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndRC2_40.engineDoFinal(DashoA12275)

at javax.crypto.Cipher.doFinal(DashoA12275)

at com.sun.net.ssl.internal.ssl.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1272)

... 19 more

Jul 27, 2009 5:01:23 PM org.apache.catalina.startup.Catalina load

SEVERE: Catalina.start

LifecycleException: Protocol handler initialization failed: java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

at org.apache.catalina.connector.Connector.initialize(Connector.java:1061)

at org.apache.catalina.core.StandardService.initialize(StandardService.java:677)

at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:792)

at org.apache.catalina.startup.Catalina.load(Catalina.java:518)

at org.apache.catalina.startup.Catalina.load(Catalina.java:538)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:260)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:412)

Jul 27, 2009 5:01:23 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 1083 ms

Jul 27, 2009 5:01:23 PM org.apache.catalina.core.StandardService start

INFO: Starting service Catalina

Jul 27, 2009 5:01:23 PM org.apache.catalina.core.StandardEngine start

INFO: Starting Servlet Engine: Apache Tomcat/6.0.14

Jul 27, 2009 5:01:24 PM org.apache.catalina.loader.WebappClassLoader validateJarFile

INFO: validateJarFile(C:\Program Files\VMware\Infrastructure\tomcat\webapps\sms\WEB-INF\lib\servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class

Jul 27, 2009 5:01:33 PM org.apache.coyote.http11.Http11Protocol start

INFO: Starting Coyote HTTP/1.1 on http-127.0.0.1-8080

Jul 27, 2009 5:01:33 PM org.apache.coyote.http11.Http11Protocol start

SEVERE: Error starting endpoint

java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

at com.sun.net.ssl.internal.ssl.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1275)

at java.security.KeyStore.load(KeyStore.java:1150)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:319)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:259)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:410)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:378)

at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:125)

at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:496)

at org.apache.tomcat.util.net.JIoEndpoint.start(JIoEndpoint.java:515)

at org.apache.coyote.http11.Http11Protocol.start(Http11Protocol.java:204)

at org.apache.catalina.connector.Connector.start(Connector.java:1132)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:566)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded

at com.sun.crypto.provider.SunJCE_h.b(DashoA12275)

at com.sun.crypto.provider.SunJCE_h.b(DashoA12275)

at com.sun.crypto.provider.SunJCE_ac.b(DashoA12275)

at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndRC2_40.engineDoFinal(DashoA12275)

at javax.crypto.Cipher.doFinal(DashoA12275)

at com.sun.net.ssl.internal.ssl.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1272)

... 19 more

Jul 27, 2009 5:01:33 PM org.apache.catalina.startup.Catalina start

SEVERE: Catalina.start:

LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

at org.apache.catalina.connector.Connector.start(Connector.java:1139)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:566)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Jul 27, 2009 5:01:33 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 9746 ms

[rbeu]

That's pretty much exactly what I'm getting in my logs. And it also works on port 443 which is the part that's really confusing me. I've opened a call with VMware so I'll post the results when it's resolved.

[mrohen]

Well, is there any news or things to try?

[rbeu]

Nothing yet. It's been escalated internally at VMware but that was the last update I got (Thursday last week)

[sporkyUK · Accepted solution]

I had an identical issue as those described above. In my case the reason the 8443 service was not starting is that tomcat was attempting to use the incorrect password to open the rui.pfx file. I did the following to correct this problem.

1.) Stop the vCenter windows services

2.) Open %PROGRAMFILES%\VMware\Infrastructure\tomcat\conf\server.xml in Notepad

3.) Search for "<Connector port="8443"

4.) You'll see this line refers to the rui.pfx certificate file that was change when you updated your certificate. Update the "keystorPass=" to the rui.pfx certificate password.

5.) Start the vCenter services

NOTE: The password can't be blank for the rui.pfx as it was in my case as this will cause it to fail. I had to import then export the rui.pfx with a password to get the above to work for me.

Hope this helps guys.

[rbeu]

Thanks, that definitely looks like the solution as I didn't use "testpassword" when I created the pfx file and server.xml is expecting that. When you copied in the new pfx file with the password did you have to disconnect/reconnect your hosts to VC as well? I'm just trying to get an idea of the scope of recreating the pfx file. I could change the password in the server.xml file to what I used but it seems like the safer option is to use "testpassword" since this isn't the first time VMware has hardcoded that password and they'll probably do it again in the future.

[mrohen]

You are absolutely right, it is the same issue.

Just the minor hint: in order to minimize VC downtime, it's better to edit server.xml at first and then just restart service "VMware VirtualCenter Management Webservices"

Thank you very much!

[rbeu]

This fixed my problem as well, thanks! VMware is apparently aware of this issue and are working on an KB article and updated upgrade instructions. I'll post the links when and if I get them.

[mrohen]

And now as i have noticed, i have (still?) not working:

- Administration->vCenter Service Status ("Cannot access the health service!"),

- ESX host tab "Storage Views" ("Connection with vpxd service failed. User credentials could not be authenticated" when try to Update)

- ESX host tab "Hardware Status" ("The hardware monitoring service is not available")

I almost believe it's anyhow connected with certificates, but still not sure what to do.

vpxd logs are overloaded with diagnostic messages, but it seem, at least some of these problems correspond to:

-- BEGIN task-internal-13023 -- -- vim.SessionManager.loginExtensionByCertificate -- 11B9C5C4-44B9-4906-AF08-D6CBBBEB6E74

-- FINISH task-internal-13023 -- -- vim.SessionManager.loginExtensionByCertificate -- 11B9C5C4-44B9-4906-AF08-D6CBBBEB6E74

-- ERROR task-internal-13023 -- -- vim.SessionManager.loginExtensionByCertificate: vim.fault.NoClientCertificate:

(vim.fault.NoClientCertificate) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

am i alone?

[shayhyams]

לא אהיה במשרד עד יום שלישי ה 18 לחודש

אנא בכל פנייה יש לפנות לאריק גולן \ רונן פנחס

תודה, שי היימס

[rbeu]

I was just going to start investigating today all the things you mentioned because they aren't working for me either. So I think it's safe to say you're not alone and they are connected I also can't get the Converter plug-in to load and I'm seeing "The connection can't be made because the taget machine actively refused it" messages which is what I was seeing in the catalina logs as well. I'll let VMware know about this with the open case that I have.

[IRIX201110141]

@,

welcome to the club. Let me tell you that you arent alone in the dark. I open SR #1337121071 a couple of weeks ago because i stumpled into the same problems. We always replace default Certs agains custom selfsigned or officical Certs when ever its possible. With vc 2.5 it was very easy and works out of the box. With vSphere this step becomes the total nightmare.

My vCenter installations always use the short computer name as default but a SSL Cert uses the FQDN. To fix this you have to find and replace the computername in a couple of *.xml files. Otherwise you always receive the SSL Warnings. It also looks like that VUM uses this own CERT which isnt covered by VMware documenation on how to replace the SSL Certs on vCenter and ESX Hosts.

When ever replacing the default Certs the following happends:

• vCenter Server status doenst work any more

• HardwareHealth not working

• StorageViews reporting an error

• Performance Overview doesnt work, only the Advanced tab displaying results

If using a VI Client from a Workstation rather than the Client on the vCenter tends in the total lost of the StorageView.

My SR goes so slowly ahead that i think we see "Next Generation Software 5.0" first befor the problem is solved. The support recorded more than one sessions, i have send screenshots and up to 1GB of logs from several sessions. A fresh installation of vCenter show the same problems like the current we used (also a fresh installation and not upgraded from 2.5).

The used SSL Certs works fine on vCenter 2.5 and VMware reportet they a fine and correct. They also used the "testpassword" and have the right keylenght and encrytion method.

If you open your own SR please refer to #1337121071, maybe it speeds up the process to find the solution.

Regards

Joerg

'Remember if you found this or others answers helpful do not forget to award points by marking an answer as helpful or correct'

[rbeu]

Are you saying that even if you use "testpassword" when you create the pfx file all of this is still broken? That was going to be my fallback plan if I wasn't making any progress - recreate the certs with "testpassword" and just be done with it.

[IRIX201110141]

Are you saying that even if you use "testpassword" when you create the pfx file all of this is still broken? That was going to be my fallback plan if I wasn't making any progress - recreate the certs with "testpassword" and just be done with it.

Yes its broken. My PFX already contains the right password and keep in mind that they work perfectly with VC 2.5.

About the hardcoded places with the short computername VMware Support says that this w'll be changed in the upcomming U1 for vCenter 4.

Regards

Joerg

'Remember if you found this or others answers helpful do not forget to award points by marking an answer as helpful or correct'

[Iuridae]

Hi,

I have the exact same problem, but the problem I have is that I'm no expert with SSL and certificates. Can't recall stating any password for pfx.file either? Everything is just default installation with a local SQLexpress DB and the server is a member of a domain.

So what should I do? Performance Overview and vCenter Service Status, both displayes Connection refused.

[IRIX201110141]

VMware Support found the culprit which breaks half of my vCenter installation. The "rui.crt" contains some bogus information at the beginning of the file. We removed all unwanted text above "---BEGIN CERTIFICATE---" and restart the complete vCenter. This brings back StorageView, Hardware Status|Health and half of the vCenter server status. The last one have only the minor problem that the short computername is used instead the FQHN.

Regards

Joerg

'Remember if you found this or others answers helpful do not forget to award points by marking an answer as helpful or correct'

[Iuridae]

Hi again,

Noticed that its on my vCenter machine (ie. localhost) that those views doesn't work. Using another client and it works.....