Hi, I am not vmware administrator so don't have expert skill to troubleshoot vCenter problem.
In my lab, there Version:6.0.0.20000 running on Linux built by another person who left. Recently the / and /storage/log are 100% full. I have cleared the older logs to reclaim disk space. Reboot from vcsa console but multiple services cannot come up.
From service-control --status
Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vsphere-client ()
I checked the logs of different services (vpxd, invsc) but cannot find which services I should focus on first.
From /var/local/vmware/eam/wrapper.log
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition
with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processP
roperties(PropertyPlaceholderConfigurer.java:268)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcess
BeanFactory(PropertyResourceConfigurer.java:75)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactor
yPostProcessors(AbstractApplicationContext.java:554)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactor
yPostProcessors(AbstractApplicationContext.java:528)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(Abstract
ApplicationContext.java:363)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(Class
PathXmlApplicationContext.java:139)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(Class
PathXmlApplicationContext.java:83)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO | jvm 1 | 2018/08/20 21:47:42 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
STATUS | wrapper | 2018/08/20 21:47:44 | <-- Wrapper Stopped
From inv-svc.log
2018-08-20T21:51:40.042Z [WrapperListener_start_runner FATAL com.vmware.vim.dataservices.WrapperListenerImpl opId=] Error starting service:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vlsi-server' defined in class path resource [server/config/server-
config.xml]: Cannot create inner bean 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#1b3a959a' of type [com.vmware.vim.vmomi.server.http.impl.FilterImp
l] while setting bean property 'filters' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean wi
th name 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#1b3a959a' defined in class path resource [server/config/server-config.xml]: Cannot resolve refer
ence to bean 'authFilter' while setting bean property 'filter'; nested exception is rg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authFilter' defined in class path resource [server/config/server-config.xml]: Cannot resolve reference to bean 'authChecker' while setting bean property 'authChecker'; nested exception is rg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authChecker' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'userSessionManager' while setting bean property 'user
SessionManager'; nested exception is rg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSessionManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authorizationManager' while setting bean property 'authorizat
ionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizationManager' defined in
class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authProvider' while setting bean property 'dataProvider'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authProvider' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'memCache' while setting bean property 'parentChainCache'; nested exception is org.springf
ramework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]:
Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.f
actory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]:
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.
query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.intero
p.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
From vpxd.log
2018-08-18T10:04:44.164Z info vpxd[7F0064CE67A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI
endpoint.
2018-08-18T10:04:44.169Z warning vpxd[7F0064CE67A0] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2018-08-18T10:04:44.169Z warning vpxd[7F0064CE67A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::System
Error while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)
2018-08-18T10:04:44.169Z error vpxd[7F0064CE67A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reac
hed. Giving up ...
2018-08-18T10:04:44.169Z error vpxd[7F0064CE67A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore4Soap24Invali
dResponseExceptionE(Invalid response code: 404 Not Found).
2018-08-18T10:04:44.169Z warning vpxd[7F0064CE67A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacad
eConstPtr)] took 90064 ms
2018-08-18T10:04:44.169Z error vpxd[7F0064CE67A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, sso
FacadeConstPtr)
--> Backtrace:
-->
--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-3634794, tag: vpxd
--> backtrace[00] libvmacore.so[0x003C6234]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001F01C3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x0019A11D]: Vmacore::Throwable::Throwable(std::string const&)
--> backtrace[03] vpxd[0x00BCC0DE]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)
--> backtrace[04] vpxd[0x00BC9B5A]
--> backtrace[05] vpxd[0x00BB5E20]
--> backtrace[06] vpxd[0x00AF4189]
--> backtrace[07] libc.so.6[0x0001EC36]
--> backtrace[08] vpxd[0x00AF3BDD]
--> [backtrace end]
-->
2018-08-18T10:04:44.171Z warning vpxd[7F0064CE67A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 182849 ms
2018-08-18T10:04:44.171Z error vpxd[7F0064CE67A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...
2018-08-18T10:04:44.171Z info vpxd[7F0064CE67A0] [Originator@6876 sub=SupportMgr] Wrote uptime information
2018-08-18T10:06:44.172Z info vpxd[7F0064CE67A0] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now
From vsphere-client/wrapper.log
INFO | jvm 1 | 2018/08/20 22:02:10 | [2018-08-20T22:02:10.321Z] [ERROR] start-signalling-1 <DE0006E> Start failed for bundle 'com.vmware.vimservices' version '6.0.0'. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cloudVmServiceManagerImpl' defined in
URL [bundleentry://176.fwk1333501652/META-INF/spring/bundle-context.xml]: Cannot resolve reference to bean 'ssoServiceImpl' while setting constructor arg
ument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServiceImpl' defined in URL [bundlee
ntry://176.fwk1333501652/META-INF/spring/bundle-context.xml]: Cannot resolve reference to bean 'ssoLocatorImpl' while setting constructor argument; nested
exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoLocatorImpl' defined in URL [bundleentry://176.fw
k1333501652/META-INF/spring/bundle-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: C
ould not instantiate bean class [com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl]: Constructor threw exception; nested exception is com.vmware.af.V
mAfClientNativeException
I can access <vcenter>:5480 and see SSO domain status is running and overhealth status is good. Which service is the root cause of the problem that I should focus first?
If the only resolution is to rebuid a new VCSA, how do I get all the managed hosts information from this existing VCSA. Without access to the web client, I do not have the complete list of managed ESXi hosts.
Thank you very much for your help.
