VMware Cloud Community
thakala
Hot Shot
Hot Shot

vCenter Server 5.5 fails to start after server reboot

I am testing vSphere 5.5 in our lab and I am experiencing some issues with vCenter Server or SSO. After a fresh install all works about fine, I can add AD authentication to SSO and AD login works to vSphere Client. I was having issues with vCenter Orchestator login so I rebooted vCenter Server VM. After a Windows reboot vCenter Server is no longer starting, it tries to start but SSO fails and vCenter Server process shuts down. Below is a snippet from vpxd.log with relevant output.

Domain Controller is Windows Server 2012.

vCenter Server and SSO is on Windows Server 2008 R2.

2013-09-28T12:49:09.679+03:00 [04988 info 'dbdbPortgroup'] [VpxdInvtDVPortGroup::PreLoadDvpgConfig] loaded [0] dvpg config objects

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Solution user set to: vCenterServer_2013.09.26_125138

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] VC's ServiceId in LookupService: Default-First-Site:b0f86720-9fa6-42b5-aaf4-26d700d6152a

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] STS URI set to: https://vCenter.vclass.local:7444/sts/STSService/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Admin URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Groupcheck URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] VC SSL certificate location: C:\ProgramData\VMware\VMware VirtualCenter\ssl\rui.crt

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] STS URI set to: https://vCenter.vclass.local:7444/sts/STSService/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Admin URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Groupcheck URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.737+03:00 [04988 warning 'Default'] Closing Response processing in unexpected state: 3

2013-09-28T12:49:09.740+03:00 [04988 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: Invalid response code: 404 Not Found.

2013-09-28T12:49:09.741+03:00 [04988 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

--> backtrace[00] rip 000000018018cd7a

--> backtrace[01] rip 0000000180106c48

--> backtrace[02] rip 000000018010803e

--> backtrace[03] rip 00000001800907f8

--> backtrace[04] rip 00000000003f5bac

--> backtrace[05] rip 0000000000416722

--> backtrace[06] rip 000000014075dd5a

--> backtrace[07] rip 00000001407578bc

--> backtrace[08] rip 000000014098800b

--> backtrace[09] rip 000007feff17a82d

--> backtrace[10] rip 0000000076ef652d

--> backtrace[11] rip 000000007712c541

-->

2013-09-28T12:49:09.742+03:00 [04988 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 3520 ms

2013-09-28T12:49:09.742+03:00 [04988 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

Tomi http://v-reality.info
63 Replies
a_p_
Leadership
Leadership

What's the status of the different VMware services in services.msc? Can you confirm the SSO services which are supposed to start automatically are started? Do you see any related entries in the Windows event logs?

André

Reply
0 Kudos
thakala
Hot Shot
Hot Shot

All identity provider services are running.

vCenter_SSO_55.png

I can't see any errors related to SSO in Windows event logs.

Tomi http://v-reality.info
Reply
0 Kudos
raog
Expert
Expert

Whats the result when you try accessing the lookupservice url via the browser?

Regards

Girish

To Virtualization and beyond! PS::If you felt the answer as helpful, please mark it as helpful/answered so that it helps other users as well! Blog:: www.virtualtipsntricks.com
Reply
0 Kudos
BobD01
Contributor
Contributor

We have exactly the same issue, same errors in log file.  We are logging a call with vmware, will keep you posted.

Same on Server 2012 and 2008, sounds like a bug to me.

Reply
0 Kudos
thakala
Hot Shot
Hot Shot

Btw, have you set vCenter Server %TMP% and %TEMP% to non-standard location, or are you installing vCenter Server components to non-default location?

I think %TMP% and %TEMP% folder location is a key in this bug, if they are on same drive as SSO install location everything is OK. If they are on another drive than SSO installation location, SSO fails.

Tomi http://v-reality.info
Reply
0 Kudos
BobD01
Contributor
Contributor

I've installed everything in the default locations, my variables are:

TEMP=C:\Users\DAB087~1.BDE\AppData\Local\Temp\2

TMP=C:\Users\DAB087~1.BDE\AppData\Local\Temp\2

My install has all components (SSO, VC etc) on the same server, using SQL Express.

Reply
0 Kudos
SJVMW
VMware Employee
VMware Employee

Hi,

   Please upload the vCenter Sever logs via "Generate vCenter Server log bundle", SSO logs via "Generate vCenter Single Sign-On log bundle",  and logs under %Temp%.

Thanks.

Reply
0 Kudos
Pompon
Contributor
Contributor

I've got exactly the same problem on Vmware Vcenter 5.5.

After a reboot, the service vmware virtualcenter server doesn't start, with the same error.

Thanks

Reply
0 Kudos
Pompon
Contributor
Contributor

I have successfully restart the service with this procedure :

- wait all the service to start or fail at boot

- stop Vmware VirtualCenter Server

- restart Vlware Secure token service

- start Vmware virtualcenter server

Bosmanke
Contributor
Contributor

Also I have the exact same issue and errors in the vpxd.log file.

I did a fresh install of vCenter 5.5 on a Windows 2008 R2 64-bit Enterprise SP1 server.

Everything was working fine, but after a reboot of the server, the VMware VirtualCenter Server service doesn't start anymore.

I'm using a remote SQL DB, and still able to connect to it from the vCenter server.

vCenter services.png

In the system eventlog there is an 'EventID: 7024' logged with the following information:

'The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified..'

And the application eventlog logged an 'EventID: 1000':

'The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Failed to intialize VMware VirtualCenter. Shutting down...

the message resource is present but the message is not found in the string/message table'

Regards,

Reply
0 Kudos
Bosmanke
Contributor
Contributor

That will do also the trick for me!!

Reply
0 Kudos
BobD01
Contributor
Contributor

That works fine!  I bet there is a dependency missing from that service.

Thank you

Reply
0 Kudos
Pompon
Contributor
Contributor

Yes I think there is a dependancy missing. I don't find which one :s     

Reply
0 Kudos
thakala
Hot Shot
Hot Shot

This fix does not apply to our environment, SSO is still failing.

But I think Pompon is on right tracks, it seems that SSO admin server is failing to start, here is snippet of ssoAdminServer.log.

[2013-10-01 15:25:56,997 localhost-startStop-1  ERROR org.springframework.web.context.ContextLoader] Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfo' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Cannot create inner bean 'ssoServerAboutInfoFactory$created#48c08ea5' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#48c08ea5' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public com.vmware.vim.sso.AboutInfo com.vmware.identity.admin.server.impl.AboutInfoFactory.createAboutInfo()] threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to query cluster id

  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:230)

  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:117)

  at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)

  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

  at java.security.AccessController.doPrivileged(Native Method)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)

  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)

  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)

  at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)

  at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)

  at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)

  at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)

  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)

  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)

  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)

  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)

  at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:968)

  at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1646)

  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

  at java.util.concurrent.FutureTask.run(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

  at java.lang.Thread.run(Unknown Source)

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#48c08ea5' defined in ServletContext resource

I will upload full log dumps.

Tomi http://v-reality.info
Reply
0 Kudos
thakala
Hot Shot
Hot Shot

Since vCenter Server does not start on this host I am unable to use vSphere Client for exporting logs, however I compressed manually every logs and cfg directory under a C:\ProgramData\VMware in the zip I uploaded to ftpsite.vmware.com/communities2293473/ folder. Please grab that file before it is expired.

edit. I just uploaded zip with contents of %TMP% and %TEMP%.

Tomi http://v-reality.info
Reply
0 Kudos
ShamR
Contributor
Contributor

We were having the same issue, the vCenter service would fail to start after a server reboot - this only happened when the Domain was added as a identity source.

Using PomPom's steps we have overcome this.

We have a call raised with Vmware to look into this.

Reply
0 Kudos
Pompon
Contributor
Contributor

I'm checking the services, and when I pass the service "Vmware Identity Management service" at "Automatic delayed", in place of "Automatic", all services are starting correctly at boot.


I'm validated it again restarting the server (very long to start :smileysilly:)

Reply
0 Kudos
Pompon
Contributor
Contributor

It works Smiley Happy

(I've got the domain attached to the vcenter too)    

Reply
0 Kudos
BobD01
Contributor
Contributor

That didn't work for me but the restarting the Secure Token Service does still work ok.

Reply
0 Kudos