VMware Cloud Community
PavelBolshakov
Contributor
Contributor

Can`t start VirtualCenter service after restart the server

Hi All,

my problem is with VirtualCenter service wich can`t start after OS restart and we couldn`t connect to vCenter server.

This is the third server with this problem. In the first two cases it had to completely reinstall the vCenter server with it components and this solved the problem, but it is too long and can not be a permanent solution.


After some troubleshooting i saw that VMware Secure Token Service didn`t start.

I think that the problem is related to an error when loading tomcat server of STS service, but I could be wrong.

Could you please help me to resolve this problem! Thanks in advance!

vCenter server OS: Windows Server 2008 R2

vCenter version: 5.5.0

vCenter build: 264648


Here is latest content of log files:


c:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-2788.log

2016-07-29T03:51:04.013+03:00 [03500 warning 'Default'] Failed to connect socket; <io_obj p:0x000000000ac8bee8, h:988, <TCP '0.0.0.0:0'>, <TCP '146.55.32.155:7444'>>, e: system:10061(No connection could be made because the target machine actively refused it)

2016-07-29T03:51:05.042+03:00 [03500 warning 'Default'] Failed to connect socket; <io_obj p:0x000000000ac88258, h:988, <TCP '0.0.0.0:0'>, <TCP '[::1]:7444'>>, e: system:10061(No connection could be made because the target machine actively refused it)

2016-07-29T03:51:05.042+03:00 [03500 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:000000000b14b530, TCP:i2k8ru0055vg001.mycompany.com:7444>; cnx: (null), error: class Vmacore::SystemException(No connection could be made because the target machine actively refused it)

2016-07-29T03:51:05.042+03:00 [03504 error '[SSO][SsoCertificateManagerImpl]'] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: class Vmacore::SystemException(No connection could be made because the target machine actively refused it)

2016-07-29T03:51:05.042+03:00 [03504 warning '[SSO][SsoCertificateManagerImpl]'] [RetryOnConnectionFailure] Max handshake attempts (10) reached. Giving up ...

2016-07-29T03:51:05.042+03:00 [03504 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Probably connection exception: No connection could be made because the target machine actively refused it

2016-07-29T03:51:05.058+03:00 [03504 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: No connection could be made because the target machine actively refused it.

2016-07-29T03:51:05.058+03:00 [03504 warning 'VpxProfiler'] Vpxd::ServerApp::Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 110636 ms

2016-07-29T03:51:05.058+03:00 [03504 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

--> backtrace[00] rip 000000018018b7fa

--> backtrace[01] rip 0000000180104c78

--> backtrace[02] rip 0000000180105f6e

--> backtrace[03] rip 000000018008ff08

--> backtrace[04] rip 0000000000565bac

--> backtrace[05] rip 0000000000586722

--> backtrace[06] rip 00000001404770ca

--> backtrace[07] rip 0000000140470c2c

--> backtrace[08] rip 00000001406a98eb

--> backtrace[09] rip 000007fefe44a82d

--> backtrace[10] rip 00000000774c59bd

--> backtrace[11] rip 00000000775fa2e1

-->


c:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs\wrapper.log

STATUS | wrapper  | 2016/07/29 03:39:28 | --> Wrapper Started as Service

STATUS | wrapper  | 2016/07/29 03:39:30 | Launching a JVM...

INFO  | jvm 1    | 2016/07/29 03:39:30 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org

INFO  | jvm 1    | 2016/07/29 03:39:30 |  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

INFO  | jvm 1    | 2016/07/29 03:39:30 |

INFO  | jvm 1    | 2016/07/29 03:39:37 | Jul 29, 2016 3:39:37 AM org.apache.catalina.startup.Catalina load

INFO  | jvm 1    | 2016/07/29 03:39:37 | WARNING: Catalina.start using conf/server.xml: Error at (7, 91) : com.springsource.tcserver.licensing.LicensingLifecycleListener

INFO  | jvm 1    | 2016/07/29 03:39:37 | Jul 29, 2016 3:39:37 AM org.apache.catalina.startup.Catalina load

INFO  | jvm 1    | 2016/07/29 03:39:37 | WARNING: Catalina.start using conf/server.xml: Error at (7, 91) : com.springsource.tcserver.licensing.LicensingLifecycleListener

INFO  | jvm 1    | 2016/07/29 03:39:37 | Jul 29, 2016 3:39:37 AM org.apache.catalina.startup.Catalina start

INFO  | jvm 1    | 2016/07/29 03:39:37 | SEVERE: Cannot start server. Server instance is not configured.

STATUS | wrapper  | 2016/07/29 03:39:39 | <-- Wrapper Stopped

c:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs\catalina.2016-07-29.log

29-Jul-2016 03:39:34.583 SEVERE [WrapperSimpleAppMain] org.apache.tomcat.util.digester.Digester.<clinit> Unable to load property source[com.springsource.tcserver.security.PropertyDecoder].

java.lang.ClassNotFoundException: com.springsource.tcserver.security.PropertyDecoder

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.security.AccessController.doPrivileged(Native Method)

  at java.net.URLClassLoader.findClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at java.lang.Class.forName0(Native Method)

  at java.lang.Class.forName(Unknown Source)

  at org.apache.tomcat.util.digester.Digester.<clinit>(Digester.java:100)

  at org.apache.catalina.startup.Catalina.createStartDigester(Catalina.java:303)

  at org.apache.catalina.startup.Catalina.load(Catalina.java:552)

  at org.apache.catalina.startup.Catalina.load(Catalina.java:664)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:281)

  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:455)

  at net.covalent.tomcat.wrapper.ThreadDumpWrapper.main(ThreadDumpWrapper.java:74)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)

  at java.lang.Thread.run(Unknown Source)

29-Jul-2016 03:39:37.157 SEVERE [WrapperSimpleAppMain] org.apache.tomcat.util.digester.Digester.startElement Begin event threw exception

java.lang.ClassNotFoundException: com.springsource.tcserver.licensing.LicensingLifecycleListener

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.security.AccessController.doPrivileged(Native Method)

  at java.net.URLClassLoader.findClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:144)

  at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1288)

  at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)

  at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1561)

  at org.apache.catalina.startup.Catalina.load(Catalina.java:616)

  at org.apache.catalina.startup.Catalina.load(Catalina.java:664)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:281)

  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:455)

  at net.covalent.tomcat.wrapper.ThreadDumpWrapper.main(ThreadDumpWrapper.java:74)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)

  at java.lang.Thread.run(Unknown Source)

29-Jul-2016 03:39:37.172 SEVERE [WrapperSimpleAppMain] org.apache.tomcat.util.digester.Digester.startElement Begin event threw exception

java.lang.ClassNotFoundException: com.springsource.tcserver.licensing.LicensingLifecycleListener

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.net.URLClassLoader$1.run(Unknown Source)

  at java.security.AccessController.doPrivileged(Native Method)

  at java.net.URLClassLoader.findClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at java.lang.ClassLoader.loadClass(Unknown Source)

  at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:144)

  at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1288)

  at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)

  at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)

  at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)

  at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1561)

  at org.apache.catalina.startup.Catalina.load(Catalina.java:616)

  at org.apache.catalina.startup.Catalina.start(Catalina.java:678)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)

  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:456)

  at net.covalent.tomcat.wrapper.ThreadDumpWrapper.main(ThreadDumpWrapper.java:74)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)

  at java.lang.Thread.run(Unknown Source)

Tags (3)
0 Kudos
2 Replies
PavelBolshakov
Contributor
Contributor

After much reading log files, I decided to reinstall the tomcat server (tc server) and this is resolved the problem.

What id did:

1) Stop all SSO services

VMware Secure Token Service

VMware Identity Management Service

VMware Certificate Service

VMware Kdc Service

VMware Directory Service

2) Open "Program and Features", uninstall "VMware vCenter Server - tc Server". When the installer warn then click OK.: "Caution: after you uninstall vCenter Server - tc Server, system will not be functional. To avoid this loss of functionality, before uninstalling vCenter Server - tc Server, uninstall all components that depend on this vCenter Server - tc Server."

3) Run the installer "\Single Sign-On\prerequisites\VMware-tcserver.exe" of "VMware vCenter Server - tc Server" from mounted vCenter installation ISO. At the end of the installation, the installer can not be able to perform start service "VMware Virtual Center" and displays an error message.

4) Start all SSO services

VMware Secure Token Service

VMware Identity Management Service

VMware Certificate Service

VMware Kdc Service

VMware Directory Service

5) Start sevice "VMware Virtual Center"

PavelBolshakov
Contributor
Contributor

Today I have found the root of issue.

We have an anti-virus software deployed on our environment (Kaspersky Security for Virtualization). Anti-virus software detected a Trojan (HEUR:Backdoor.Java.Generic) in file "C:\​Program Files\​Common Files\​VMware\​VMware vCenter Server - tc Server\​lib\​tcServer.jar" on vCenter server and removed it.

So after the restart the vCenter server it could not start services "Secure Token Service" and "VirtualCenter Server".

Now I have created an exception to the anti-virus program for this file.