VMware Cloud Community
Asnivor
Contributor
Contributor

VMware VirtualCenter Server Service not starting

I've been fighting with this issue for a while now and haven't been able to find a solution - even with checking the posts on this forum.

I am running VMware vCenter Server 5.1.0.32743 on a Windows 2008R2 physical server (not on a domain). Whenever I make a hardware change (ie, adding a SCSI card) the VirtualCenter Server service (and also the VirtualCenter Management Webservices service) refuses to start.

When I remove the SCSI card the issue still persists and the only way for me to resolve so far is to restore the server from backup - obviously not ideal and I need to connect tape drives to this server.

servicenotstarting1.png

I get the standard event 1000 in event viewer also.

I have tried changing the service startup types so that SSO service comes up first (as suggested in one of the KB articles) but this has no effect.

The back end of the vpxd-xx.log file shows this:

2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] Invoking callbacks for key log.level, pre commit
2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] Invoking callbacks for key log.level, pre commit
2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] No change to log.level
2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMain'] [VpxdMain] Setting OpenSSL verify locations CAFile=C:\ProgramData\VMware\SSL\ca_certificates.crt CAPath=C:\ProgramData\VMware\SSL
2012-11-22T12:35:51.780Z [00564 info 'Default'] Creating SSL Contexts
2012-11-22T12:35:51.858Z [00564 info 'utilvpxdDbParallelLoader'] Num rows: 6, num CPUs: 4, threads: 1, step: 6
2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-8 in mirror for host vim.HostSystem:host-10
2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-8 in mirror for host vim.HostSystem:host-13
2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-53 in mirror for host vim.HostSystem:host-54
2012-11-22T12:35:53.028Z [00564 warning 'VpxProfiler'] Vpxd::ServerApp::Init [VpxdInvtHost::Init2(gDB)] took 1108 ms
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-241 'unprotected' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-241 'n/a' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-61 'unprotected' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-61 'n/a' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-59 'unprotected' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-59 'n/a' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-60 'unprotected' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-60 'n/a' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-57 'unprotected' -> 'protected'
2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-57 'n/a' -> 'protected'
2012-11-22T12:35:53.106Z [00564 info 'dbdbPortgroup'] [VpxdInvtDVPortGroup::PreLoadDvpgConfig] loaded [0] dvpg config objects
2012-11-22T12:35:53.106Z [00564 info 'authvpxdMoSessionManager'] [SSO][SessionManagerMo::Init] Admin URI set to: https://10.105.110.110:7444/sso-adminserver/sdk
2012-11-22T12:35:53.106Z [00564 info 'authvpxdMoSessionManager'] [SSO][SessionManagerMo::Init] Downloading STS Root certificates ...
2012-11-22T12:35:53.122Z [00496 info 'Default'] Thread attached
2012-11-22T12:35:53.153Z [00564 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Unexpected exception
--> Backtrace:
--> backtrace[00] rip 000000018018977a
--> backtrace[01] rip 0000000180100c98
--> backtrace[02] rip 0000000180101fae
--> backtrace[03] rip 000000018008aeab
--> backtrace[04] rip 0000000000514971
--> backtrace[05] rip 00000000004b1298
--> backtrace[06] rip 00000000004b16c9
--> backtrace[07] rip 0000000000420fae
--> backtrace[08] rip 000000014040bfb8
--> backtrace[09] rip 000000013f300078
--> backtrace[10] rip 000000013f30016a
--> backtrace[11] rip 000000013f300279
--> backtrace[12] rip 000000013f300609
--> backtrace[13] rip 000000013f642903
--> backtrace[14] rip 000000013fdee4b9
--> backtrace[15] rip 000000013fde835c
--> backtrace[16] rip 0000000140008a3b
--> backtrace[17] rip 000007fefecda82d
--> backtrace[18] rip 000000007764652d
--> backtrace[19] rip 00000000779dc521
-->
2012-11-22T12:35:53.153Z [00564 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 4353 ms
2012-11-22T12:35:53.153Z [00564 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
2012-11-22T12:35:53.153Z [00564 info 'vpxdvpxdSupportManager'] Wrote uptime information
2012-11-22T12:36:00.813Z [03108 warning 'VpxProfiler' opID=SWI-eac14804] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2012-11-22T12:36:12.825Z [03108 warning 'VpxProfiler' opID=SWI-63f9e37] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2012-11-22T12:36:24.837Z [03108 warning 'VpxProfiler' opID=SWI-b155aed1] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2012-11-22T12:36:36.849Z [03108 warning 'VpxProfiler' opID=SWI-7a9c1975] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2012-11-22T12:36:48.861Z [03108 warning 'VpxProfiler' opID=SWI-a2d6fb25] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2012-11-22T12:36:49.126Z [00564 info 'Default'] Forcing shutdown of VMware VirtualCenter now

Interestingly (and this is where I cannot find anything online relating to this) discover-is from C:\Program Files\VMware\Infrastructure\SSOServer\utils\logs shows:

2012-11-21 16:15:07,110 - DDNBGF5J,,,,Executing action: 'discover-is'

2012-11-21 16:15:07,111 - DDNBGF5J,,,,Discovering identity sources

2012-11-21 16:15:08,022 - DDNBGF5J,,,,ERROR: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)

     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

     at $Proxy0.executeCommand(Unknown Source)

     at com.rsa.command.CacheableHttpInvokerTarget.executeCommand(HttpInvokerCommandTarget.java:261)

     at com.rsa.command.DelegatingCommandTarget.executeCommand(DelegatingCommandTarget.java:66)

     at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:270)

     at com.rsa.authn.LoginCommand.execute(LoginCommand.java:612)

     at com.rsa.authn.AuthenticatedTargetImpl.login(AuthenticatedTargetImpl.java:161)

     at com.rsa.command.ConnectionFactory$ConnectionImpl.connect(ConnectionFactory.java:754)

     at com.rsa.command.ConnectionFactory.connect(ConnectionFactory.java:542)

     at com.rsa.riat.tools.internal.Utils.login(Utils.java:1050)

     at com.rsa.riat.tools.is.DiscoverIdentitySources.execute(DiscoverIdentitySources.java:81)

     at com.rsa.riat.tools.ConfigureRIATCmd.execute(ConfigureRIATCmd.java:192)

     at com.rsa.riat.tools.ConfigureRIATCmd.main(ConfigureRIATCmd.java:517)

Caused by: org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.validateResponse(CommonsHttpInvokerRequestExecutor.java:214)

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.doExecuteRequest(CommonsHttpInvokerRequestExecutor.java:131)

     at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)

     ... 14 more

2012-11-21 17:20:55,979 - DDNBGF5J,,,,Executing action: 'discover-is'

2012-11-21 17:20:55,979 - DDNBGF5J,,,,Discovering identity sources

2012-11-21 17:20:56,884 - DDNBGF5J,,,,ERROR: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)

     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

     at $Proxy0.executeCommand(Unknown Source)

     at com.rsa.command.CacheableHttpInvokerTarget.executeCommand(HttpInvokerCommandTarget.java:261)

     at com.rsa.command.DelegatingCommandTarget.executeCommand(DelegatingCommandTarget.java:66)

     at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:270)

     at com.rsa.authn.LoginCommand.execute(LoginCommand.java:612)

     at com.rsa.authn.AuthenticatedTargetImpl.login(AuthenticatedTargetImpl.java:161)

     at com.rsa.command.ConnectionFactory$ConnectionImpl.connect(ConnectionFactory.java:754)

     at com.rsa.command.ConnectionFactory.connect(ConnectionFactory.java:542)

     at com.rsa.riat.tools.internal.Utils.login(Utils.java:1050)

     at com.rsa.riat.tools.is.DiscoverIdentitySources.execute(DiscoverIdentitySources.java:81)

     at com.rsa.riat.tools.ConfigureRIATCmd.execute(ConfigureRIATCmd.java:192)

     at com.rsa.riat.tools.ConfigureRIATCmd.main(ConfigureRIATCmd.java:517)

Caused by: org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.validateResponse(CommonsHttpInvokerRequestExecutor.java:214)

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.doExecuteRequest(CommonsHttpInvokerRequestExecutor.java:131)

     at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)

     ... 14 more

If I try to go to that commandserver url in the log above in a browser, it also get a 404:

404.png

The only thing similar I can find online is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203617...

Very similar exception error but nothing similar in my discover-is.log file.

From that kb link above I have run the following command:

C:\Program  Files\VMware\Infrastructure\SSOServer\utils>ssocli.cmd configure-riat  -a discover-is -u admin -p <masterPassword>

And I get the following output which confirms the discover-is log:


cli.png

Netstat shows that port 7444 is opened by Java and nothing else. The other potential problem port (443) is not in use by anything else that it shouldnt be.

I suspect this may be the cause of the problem but I have no idea where to go next. Any suggestions would be greatly appreiciated.

Many thanks,

Matt

Reply
0 Kudos
19 Replies
Asnivor
Contributor
Contributor

Also, if I follow the https://10.105.110.110:7444/sso-adminserver/sdk link (from the vpxd log) in a web browser I get the following:

<?xml version="1.0" encoding="UTF-8"?>
-<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">-<soapenv:Body>-<soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring>Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0]</faultstring>-<detail><RuntimeFaultFault xmlns:vim25="urn:vim25" xmlns="urn:vim25" xsi:type="vim25:InvalidRequest"/></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>

Not sure if this is normal behavior or not though....

Matt

Reply
0 Kudos
iw123
Commander
Commander

Hi,

Have you seen this KB? http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203617...

Thanks,

*Please, don't forget the awarding points for "helpful" and/or "correct" answers
Reply
0 Kudos
Asnivor
Contributor
Contributor

Hi,

Yes, this is the one I linked to in my post. The errors I am seeing are slightly different but I have tried the rsautil resolution mentioned and still the same problem.

Thanks,

Matt

Reply
0 Kudos
spravtek
Expert
Expert

What do you see when you try to browse to:

http://<ip-address-sso-server>:7080/lookupservice/sdk

or

https://<ip-address-sso-server>:7444/lookupservice/sdk

Reply
0 Kudos
Asnivor
Contributor
Contributor

Exactly the same response for both links:

404.png

Reply
0 Kudos
Asnivor
Contributor
Contributor

Sorry, wrong screenshot above

404_2.png

Reply
0 Kudos
spravtek
Expert
Expert

Are you sure you configured the correct ports during setup? (7444 vs eg 8888) ... Just making sure Smiley Happy

Did you try this KB? http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203617...

Maybe is a long shot, but you never know ...

I remember a KB about VMware View having an issue like this, there they renamed the folders and restarted the service to let it be recreated ... But maybe not something to experiment with ... I'll try to find this KB.

Reply
0 Kudos
Asnivor
Contributor
Contributor

Yes I tried that KB article resolution.

I did not set this machine up, but it has been running for a few months now without issues (and has had many reboots). The only thing that has changed is the installation of a PCI-E SCSI card (and potentially an unexpected poweroff the day before).

Reply
0 Kudos
spravtek
Expert
Expert

Seems we need a more robust SSO system doesn't it Smiley Wink

I found that KB from View: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200127...

But as I said... Might be a long shot ... Might not even work at all ...

Maybe you can check out the web.xml file for any obvious errors ...

Reply
0 Kudos
spravtek
Expert
Expert

Another thing came to mind, maybe rebuilding the SSO database might help (maybe it got corrupt for example).

http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-EBB03FB7-...

Reply
0 Kudos
Asnivor
Contributor
Contributor

I havent gone that far yet, but from that link I can see the following:

Run the register-is.bat command to update the stored configuration information of the Inventory Service.

register-is.bat vCenter_Server_URL Inventory_Service_URL Lookup_Service_URL

Use the following example as a model.

register-is.bat https://machinename.corp.com:443/sdk https://machinename.corp.com:10443 https://machinename.corp.com:7444/lookupservice/sdk

In this example, 443, 10443, and 7444 are the default HTTPS port numbers for vCenter Server, Inventory Service, and vCenter Single Sign On respectively. If you use custom ports, replace the port numbers in the example with the port numbers you use.

If i manually go to the 3 links above in a browser I get the following:

https://machinename.corp.com:443/sdk (Browser times out and displays no page - this is because vcenter service is not running i think)
https://machinename.corp.com:10443 (400- bad request - this is working but cannot serve to a web browser)
https://machinename.corp.com:7444/lookupservice/sdk (404 - web server responds that it cannot find the file)

It seems to me that the web server itself cannot find and serve up the single sign on files from the given link (port 7444)?

By rebuilding the SSO database, will this potentially fix webserver? I want to be fairly sure before I wipe out the database...

Thanks,

Matt

Reply
0 Kudos
spravtek
Expert
Expert

Only rebuild the database if you have a backup, or you're otherwise stuck without any other options ...

But, if your webservice isn't starting we need to find out why, the errors don't tell us much ... Therefore it can be that the webservice doesn't start because of a corrupt SSO database... But it can also be that the files from the webservice are corrupt of course...

Again, these are just assumptions, without knowing all the finer details and other errors in the logs (if there are any) ...

The only other option is to do a complete re-install of sso and set the backup of the sso DB back, if you have it... And even then.

Wish I could be of more help.

Reply
0 Kudos
spravtek
Expert
Expert

Reply
0 Kudos
Asnivor
Contributor
Contributor

Ok, finally got to the bottom of it with VMware support.

Because we are using SQL express, apparently it by default uses dynamic ports that can when the server is rebooted.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203351...

So basically the issue was that the SQL port had changed and SSO needed to be reconfigured with the new port.

Thanks for all your help.

Matt

Reply
0 Kudos
spravtek
Expert
Expert

Ah yes, that will do it as well ... Currently a lot of that going on lately Smiley Wink

Reply
0 Kudos
dejanrodiger
Contributor
Contributor

Problem similar to mine:

http://communities.vmware.com/thread/426751?tstart=0

vCenter doesn't start, but I am using normal SQL2008 server on port 1433 on different VM :smileycry:

-- Dejan
Reply
0 Kudos
glanzmann
Enthusiast
Enthusiast

Hello Matt,
just had the very same problem. Thank you for reporting and posting the solution of it.

Cheers,
      Thomas
Reply
0 Kudos
Griffes
Contributor
Contributor

Hello Matt,
just had the very same problem. Thank you for reporting and posting the solution of it.

Cheers,

Greg
Reply
0 Kudos
amitsaxena96
Contributor
Contributor

Hi,

I had the same issue with SQL 2008 R2.

I had created a new, dedicated instance for vCenter. However, the dynamic port was enabled. Thus, my vcetner service would crash every second day. I rebuilt the entire enviroment again but since dynamic ports was enabled, perhaps thats why it would break again.

Now, I have disbaled the dymic port and configured for it to work only on static port. I ran the commands to rebuild the database connection in SSO. I rebuilt the inventory but I just cannot get this to work.

My vCenter service is failing, with same symptoms as of this thread. I do see connections from vCetner server to SQL when I start the SSO service.

Any other suggestion, please?

Regards,

Amit.

Reply
0 Kudos