vCenter 5.1 Services won't start after P2V

[mmohrhra]

Hello,

I've tried to P2V my vCenter 5.1 server twice now and have been unsuccessful in both tries. The following are the steps I took to complete this operation.

vCenter is in Simple Mode and the vCenter database is an external MS SQL database, services are set to start with various service accounts. It is all working as expected prior to the P2V. There is an SQL Express instance installed on the vCenter server but I'm unsure what its being used for.

1. Shut down SQL Express Services.

2. Shut down all vCenter Services.

3. Launch P2V process from the Windows server that is running the destination 6.0 vCenter. The P2V takes several hours, but completes successfully.

4. Shut down source Windows server.

5. Remove E1000 network card from the VM and replace with VMXNET3 network card.

6. Power on the VM, install VMware tools, reboot.

7. Configure the IP address of the machine on the network card.

8. Restart all vCenter services. ODBC connection tests are successful and all services start except the "VMware VirtualCenter Server" and "VMware VirtualCenter Management Webservices" services.

Symptoms:

Windows event log reports that "The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified.."

vpxd log reports the below in the logs:

2017-01-24T16:45:45.279-05:00 [06908 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Groupcheck URI set to: https://FQDN redacted:7444/sso-adminserver/sdk

2017-01-24T16:45:45.310-05:00 [06908 error 'Default'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1)

2017-01-24T16:45:45.326-05:00 [06908 error 'Default'] Found dangling SSL error: [1] error:00000001:lib(0):func(0):reason(1)

2017-01-24T16:45:45.326-05:00 [06908 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: vmodl.fault.SystemError.

2017-01-24T16:45:45.326-05:00 [06908 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

--> backtrace[00] rip 000000018018b86a

--> backtrace[01] rip 0000000180102ac8

--> backtrace[02] rip 0000000180103f9e

--> backtrace[03] rip 000000018008d22b

--> backtrace[04] rip 0000000000475bdc

--> backtrace[05] rip 0000000000496652

--> backtrace[06] rip 000000013fcff001

--> backtrace[07] rip 000000013fcf8e1c

--> backtrace[08] rip 000000013ff1d8db

--> backtrace[09] rip 000007fefe00a82d

--> backtrace[10] rip 00000000776c59cd

--> backtrace[11] rip 00000000777fa2e1

-->

2017-01-24T16:45:45.326-05:00 [06908 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 3323 ms

2017-01-24T16:45:45.326-05:00 [06908 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

Action:

I've followed the steps in this KB link but it did not correct the issue. I followed the recommendation of rebooting the server after making the change. VMware VirtualCenter Server service fails to start with the error: Unable to create SSO facade: vmod...

If anyone has any ideas, I'm all ears, I've read through several other KB articles but none of them directly apply to the situation I'm seeing.

[rcporto]

Can you confirm if your SSO database is running on the external SQL server or at the local SQL Express instance? Can you attach the following logs:

vCenter Server logs:

Windows XP/2000/2003 – % ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\

Windows Vista/7/2008 –  ProgramData\VMware\VMware VirtualCenter\Logs\

Single Sign-On logs:

In the %Program Files%\VMware\Infrastructure\SSOServer\logs\LookupServer.log

[mmohrhra]

Here are the logs you requested. I powered on the VM around 4:30 p.m. eastern time yesterday. I believe the SSO database is on the same SQL server as the vCenter database.

[rcporto]

Based on the following, extracted from lookupServer.log, the SSO service is unable to connect to SQL server, see:

[2013-04-10 22:36:30,063 DEBUG opID= pool-3-thread-1  com.vmware.vim.lookup.impl.DbStorage] Successfully loaded JDBC driver for DB type:Mssql

[2013-04-10 22:36:30,095 DEBUG opID= pool-3-thread-1  com.vmware.vim.lookup.impl.DbStorage] DB connection pool created

[2013-04-10 22:36:51,699 DEBUG opID= pool-3-thread-1  com.vmware.vim.lookup.impl.DbStorage] Error executing statement

java.sql.SQLException: Network error IOException: Connection timed out: connect

  at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:410)

  at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)

  at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)

  at java.sql.DriverManager.getConnection(DriverManager.java:582)

  at java.sql.DriverManager.getConnection(DriverManager.java:154)

  at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)

  at com.vmware.vim.lookup.impl.DbStorage$ConnectionFactoryWithSetup.createConnection(DbStorage.java:811)

  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)

  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)

  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

  at com.vmware.vim.lookup.impl.DbStorage$SqlExecutor.execute(DbStorage.java:698)

  at com.vmware.vim.lookup.impl.DbStorage.getConfigProperty(DbStorage.java:171)

  at com.vmware.vim.lookup.impl.DbStorage.<init>(DbStorage.java:102)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

  at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:96)

  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)

  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:237)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:923)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:833)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

  at java.security.AccessController.doPrivileged(Native Method)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)

  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)

  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)

  at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)

  at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)

  at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)

  at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4779)

  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)

  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:897)

  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)

  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)

  at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)

  at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)

  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

  at java.util.concurrent.FutureTask.run(FutureTask.java:138)

  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

  at java.lang.Thread.run(Thread.java:662)

So, you need to check where the SSO database is stored. I believe it is running on the SQL Express instance that is running on the vCenter server.

The following VMware KB article shows how to fix that error: VMware vCenter Server 5.1.x Single Sign On fails after rebooting the Windows host machine (2041818) ...

[mmohrhra]

According to the config.txt file its external.

{com.rsa.appserver.hostname=<vCenter FQDN>

com.rsa.appserver.port=7444

com.rsa.appserver.protocol=https

com.rsa.db.domain=

com.rsa.db.hostname=<External MSSQL IP Address>

com.rsa.db.instance=RSA

com.rsa.db.port=1433

com.rsa.db.type=MSSQL

com.rsa.instanceName=<vCenter Hostname>

com.rsa.ssl.ca.alias=root-ca

com.rsa.ssl.ca.store.path=D:\Program Files\VMware\Infrastructure\SSOServer\root-ca.jks

[mmohrhra · Accepted solution]

Thank you for getting me on the right path. The DB is external but I also used the information you gave to point myself to what appeared to be a missing line in the jndi.properties file.

com.rsa.db.msserverinstance=RSA

Once I reset the admin password with ssocli.cmd and added this line to jndi.properties and restarted the SSO service, I was able to start the vCenter services.

I used these two KB articles: After making a change or restarting vCenter Single Sign-On server system, vCenter Server 5.1.x fails... & Resetting the SSO admin password after upgrading to vCenter Server 5.1.0b fails with the error: Key ... to fix it.

[rcporto]

From the vCenter virtual machine, you're able to telnet the IP address of your SQL server on port 1433 ?