vCenter Server 5.5 fails to start after server reboot

[thakala]

I am testing vSphere 5.5 in our lab and I am experiencing some issues with vCenter Server or SSO. After a fresh install all works about fine, I can add AD authentication to SSO and AD login works to vSphere Client. I was having issues with vCenter Orchestator login so I rebooted vCenter Server VM. After a Windows reboot vCenter Server is no longer starting, it tries to start but SSO fails and vCenter Server process shuts down. Below is a snippet from vpxd.log with relevant output.

Domain Controller is Windows Server 2012.

vCenter Server and SSO is on Windows Server 2008 R2.

2013-09-28T12:49:09.679+03:00 [04988 info 'dbdbPortgroup'] [VpxdInvtDVPortGroup::PreLoadDvpgConfig] loaded [0] dvpg config objects

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Solution user set to: vCenterServer_2013.09.26_125138

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] VC's ServiceId in LookupService: Default-First-Site:b0f86720-9fa6-42b5-aaf4-26d700d6152a

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] STS URI set to: https://vCenter.vclass.local:7444/sts/STSService/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Admin URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] Groupcheck URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][SsoFactory_CreateFacade]'] VC SSL certificate location: C:\ProgramData\VMware\VMware VirtualCenter\ssl\rui.crt

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] STS URI set to: https://vCenter.vclass.local:7444/sts/STSService/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Admin URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.685+03:00 [04988 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Groupcheck URI set to: https://vCenter.vclass.local:7444/sso-adminserver/sdk/vsphere.local

2013-09-28T12:49:09.737+03:00 [04988 warning 'Default'] Closing Response processing in unexpected state: 3

2013-09-28T12:49:09.740+03:00 [04988 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: Invalid response code: 404 Not Found.

2013-09-28T12:49:09.741+03:00 [04988 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

--> backtrace[00] rip 000000018018cd7a

--> backtrace[01] rip 0000000180106c48

--> backtrace[02] rip 000000018010803e

--> backtrace[03] rip 00000001800907f8

--> backtrace[04] rip 00000000003f5bac

--> backtrace[05] rip 0000000000416722

--> backtrace[06] rip 000000014075dd5a

--> backtrace[07] rip 00000001407578bc

--> backtrace[08] rip 000000014098800b

--> backtrace[09] rip 000007feff17a82d

--> backtrace[10] rip 0000000076ef652d

--> backtrace[11] rip 000000007712c541

-->

2013-09-28T12:49:09.742+03:00 [04988 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 3520 ms

2013-09-28T12:49:09.742+03:00 [04988 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

[zetaz]

@str1k3r, have you found a fix for your symptom? I have encountered exact same problem on 2008 R2 and only have that KB article. Did VMware support have a fix for it? Thanks.

[str1k3r]

@zetaz, I found solution: it was problem with fantom shapshots on Windows 2003 VM. When I discovered this, it have more than 400 snaps. So, I've removed information about this VM from database. After my vCenter started without any issues.

[BostonTechGuy]

In our case, we ended up rebuilding the VCENTER and migrated everything over to the new system.  The P2V that was giving issue however in our case was on Windows 2008 R2.

[vmvn01]

@str1k3er: Can you show me about detail checking steps for VM snapshots? how did you find out the snapshots when VC service wasn't running? Or did you connect to ESXi host directly to checked? I'm having the same your problem and my vcenter service can not start perfectly.

Thank you for your instructions ! waiting for you.

[Carllennon]

Thanks CHK_VMware,

Stopping and disabling IIS Admin and WWW publishing service and starting my VMware services resolved the issue for me.

[ehsdala]

I just want to give our experience on this problem and say thanks to Pompon for the solution.

We have vCenter 5.5 installed on Server 2008 R2, both system and user %TEMP% & %TMP% points to a temp drive T:, databases are located on a remote database cluster and problems arose at first reboot.

The delayed start of "Vmware Secure Token Service", "Vmware Identity Management service" and « Vmware VirtualCenter Server solved our problem! Our dependencies was correct by the way.

[viet360i]

Thanks CHK_VMware! Days of struggling and activating all sorts of unnecessary services, following guides to reset and modify things, only to find it was just needing to disale IIS Admin and WWW publishing service. Argh, wish I stumbled upon your post earlier. Thanks heaps!!!

[manickam2011101]

This is reply to initial issue that is described in comment#0 ( Sep 28, 2013 2:57 AM). Reason for this failures is that

Under few circumstances when too many services starting at the same time on system reboot, will cause SSO IDM service to take a long time to initialize. Due to this,

- SSOAdminServerice (and STS) will then start and try to connect to IDM. However, since the IDM client retry count/interval only covers 1.5 minutes, SSOAdminServer will hardfail after the retries are over.

- VC cannot start since its init stage connects to SSOAdminServer.

In above condition, you will see the error messages in log files as stated above. Workaround for this issue is that to  VC/SSO will be unusuable until restart of STS service happens manually. Steps provided in comment#9 also will work here and resolve the issue.

[ZeCavalheri]

Hi colleagues,

After sometime taking a look in all logs and so one, and as a good lab should not be 🙂 (All features installed in a single vcenter), I finally found the issue.

The problem happen when the vcenter has IIS installed on it, letting crazy the apache, stilling the port 80/443 and so one..

This probably happen in your lab if you are testing proxy authentication for example or any other thing..

So just stop the services iis admin and world wide web.. and go ahead and restart your vcenter services

I hope to solve the problem for you guys.

Regards,

Jose Cavalheri

zecavalheri@gmail.com

[srivard]

TKS CHK_VMware)

It is the only thing working for me:)

Do i restart those 2 services after vcenter is started?

[Tinker13]

My issue was related to having IIS running on the server (Virtual Center runs Apache web server, both use port 80) and the IIS interferes with the Virtual Center service initializing.

To resolve the issue go to Services

Click Start > In the search/run box type > Services.msc {enter} or via Control Panel --> Administrative Tools --> Services

Locate the IIS Admin Service and/or World Wide Publishing Service --> Stop and then Disable it.

[MikeStoica]

Hi,

I also encountered the following issue.

When i'm connecting my vCenter through vSphere web client i receive on the bottom the following message: Could not connect to one or more vCenter Server Systems: https://vcenter:443/sdk

I've checked the vmware vcenter service and it's starting, but it won't start indeed. Current version of vCenter is 5.5 u2.

Haven't made any changes to the environment, everything happened suddenly.

Any suggestion?

[Tinker13]

I had the same issue; did you check the IIS (including  world wide publishing) services? If they are running; stop then disable them. Then restart the VM VirtualCenter Management web services

[MikeStoica]

I don't have IIS service started.

[AlexanderKurnit]

For me the following fix works perfectly on two different servers, that had a problem with starting before the change:

sc.exe config vpxd depend= ProtectedStorage/lanmanworkstation/MSSQL"$"VIM_SQLEXP/VMwareSTS

I've just added to vCenter dependencies VmWare Security Token Service

[vmharik]

Very Helpful Answer.

This procedure helped me in fixing vCenter login problem.

Tasks did was

- Stopped Secure Token Service

- Stopped Identity Management Service

- Started Identity Management Service, waited it to start

- Started Secure Token Service, waited it to start

- Started vCenter Server, and it did start

on services.msc

Thanks

Message was edited by: vmharik

[thitisue]

Thank to Thakla this step is work for me.  vCenter 5.5 on Windows Server 2008R2 x64

[mcogan1966]

This is actually not working for me right now.  I do not have IIS or WWW running as a service either.

For me, the VMware VirtualCenter Server is still stuck in "starting" after the latest Windows patch.

[topetg71]

All,

My solution worked below...

1) Stopped both the "VirtualCenter Management Webservices" and "VirutalCenter Server"

2) Started the VirtualCenter Management Webservices "first" and my VirtualCenter Server started automatically

[AshSmith]

For those that haven't used the SQL Express install, the dependencies of VMware VirtualCenter Server shouldn't include it ... because it won't have been installed.

If it's set as a dependency but not present on the server then the vCenter service won't start.

Hence the command for the dependencies should be: 

sc.exe config vpxd depend= ProtectedStorage/lanmanworkstation/VMwareSTS

Cheers.