vCO Service Won't Start

[millesonz]

After upgrading vCenter to 5.1 U2, the vCO Server was of course upgraded as well. The Server service for vCO failed to start after the upgrade. I logged into the vCO Configuration to check on its status. The database needed to be upgraded so I went and initiated that. It reported back saying it successfully upgraded. Then everything within the config went green.

I attempted to start the Server service within the configuration page. It said it started successfully but the "Start Service" link was still the only available. Since this is running on a windows server, I checked the service within the Windows Services. It was still stopped. I attempted to start it but received an error. I pulled the following from the application log:

A device or program has requested attention. Device or application: C:\Program Files\VMware\Orchestrator\app-server\bin\wrapper.exe. Message title: C:\Program Files\VMware\Orchestrator\app-server\bin\wrapper.exe.

I then opened up vCO's wrapper.log and found the following:

STATUS | wrapper  | 2014/02/18 15:21:40 | --> Wrapper Started as Service

STATUS | wrapper  | 2014/02/18 15:21:40 | Java Service Wrapper Professional Edition 64-bit 3.3.9

STATUS | wrapper  | 2014/02/18 15:21:40 |   Copyright (C) 1999-2009 Tanuki Software, Ltd.  All Rights Reserved.

STATUS | wrapper  | 2014/02/18 15:21:40 |     http://wrapper.tanukisoftware.org

STATUS | wrapper  | 2014/02/18 15:21:40 |   Licensed to VMware Global, Inc. for VMware vCenter Orchestrator

STATUS | wrapper  | 2014/02/18 15:21:40 |

STATUS | wrapper  | 2014/02/18 15:21:40 | Launching a JVM...

INFO   | jvm 1    | 2014/02/18 15:21:40 | WrapperManager: Initializing...

INFO   | jvm 1    | 2014/02/18 15:21:41 | Can not generate new SSL certificate. C:\Program Files\VMware\Orchestrator\app-server/server/vmo/lib/ext/bcprov-jdk15-1\.46\.jar: no match for bcprov-jdk15-1\.46\.jar at C:\Program Files\VMware\Orchestrator\app-server\server\vmo\lib\ext

INFO   | jvm 1    | 2014/02/18 15:21:41 | ch.dunes.o11n.init.SecurityActionExecutorException: C:\Program Files\VMware\Orchestrator\app-server/server/vmo/lib/ext/bcprov-jdk15-1\.46\.jar: no match for bcprov-jdk15-1\.46\.jar at C:\Program Files\VMware\Orchestrator\app-server\server\vmo\lib\ext

INFO   | jvm 1    | 2014/02/18 15:21:41 |     at ch.dunes.o11n.init.SecurityActionExecutor.getLibraries(SecurityActionExecutor.java:74)

INFO   | jvm 1    | 2014/02/18 15:21:41 |     at ch.dunes.o11n.init.SecurityActionExecutor.execute(SecurityActionExecutor.java:20)

INFO   | jvm 1    | 2014/02/18 15:21:41 |     at ch.dunes.jboss.Main.boot(Main.java:162)

INFO   | jvm 1    | 2014/02/18 15:21:41 |     at ch.dunes.jboss.Main$1.run(Main.java:396)

INFO   | jvm 1    | 2014/02/18 15:21:41 |     at java.lang.Thread.run(Unknown Source)

STATUS | wrapper  | 2014/02/18 15:21:43 | <-- Wrapper Stopped

Does anyone know why the JVM is unable to generate a new SSL cert?

Thanks in advance,

-Zach

[abhilashhb]

Do you see a Red alert on the configuration window next to the Server Certificate?

[millesonz]

Nope. All is green in the config.

[millesonz]

After restarting the configuration server, now there is an SSL cert issue. The Authentication section is red. I went to it, attempted to register vCO with our primary vCenter using SSO Authentication. It said that the SSL certificate is unknown. So I went to the SSL Trust Manager within the Network section. Both of our vCenters had their certs imported previously. To make sure the certs were up to date, I deleted the certs that were imported and readded them. I still get the SSL certificate is unknown error when trying to register vCO with vCenter in the authentication after I just imported the SSL certs.

[iiliev]

On Authentication page, you are configuring SSO, not vCenter connections, so vCenter certificates are not important there.

Could you try to delete and re-add SSO certificate (from [sso-addr-or-ip]:7444) ?

[FSvcoe]

I ran into the same issue. Had to delete the cert, and then pull it back in from SSO box using :7444. Also had to update my license and plugin credentials, but not the actual vCenter connection, that stayed green using the ldap settings that it had been using....

[millesonz · Accepted solution]

When I deleted the SSL certs for the vCenter, I failed to add the :7444 port to the end of the FQDN. That then fixed the issue for vCO not being registered.

After it was registered with vCO, the original problem then needed to be addressed. After a bit of troubleshooting with VMware support, it was suggested that the .jar file that the wrapper.log referenced was possibly missing. Sure enough it was, as well as all of the .jar files in the following folders:

C:\Program Files\VMware\Orchestrator\app-server\server\vmo\lib

C:\Program Files\VMware\Orchestrator\app-server\server\vmo\lib\ext

C:\Program Files\VMware\Orchestrator\app-server\server\vmo\lib\vmware

and the entire o11n folder within lib.

It seemed that during the upgrade from 5.1 u1 to u2, vCO deleted those files but never replaced them. Luckily our DR vCenter, which doesn't run vCO actively had vCO installed and had all of the files so it was a quick copy over.