Hi !
I have a problem with my VCSA server (v 6.0.0.2)
The server storage was full and I had to delete some *.log files to start the server.
But when I restarted the VCSA, I try to connect to the web page of the vSphere Web Client, this message is displayed :
I checked the running services with the command service-control --status and the result is :
INFO:root:Running:
vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)
Running:
vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)
INFO:root:Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()
Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()
I don't know what to do next. Could you help me to recover my VCSA server ?
Power off/On does not help here. Please follow this KB VMware Knowledge Base
Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state
Post restarting inventory, you can restart the vpxd service and others.
Thanks,
MS
Are you using external PSC or Embedded ?
Provide df -h output
Thanks for your reply :
The primary service controller is embedded.
The result of the command
/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
=>
(******* is the FQDN of the VCSA)
and the result of df -h :
Hello...
Can you check the /var/log/vmware/invsvc/invsvc.log output and share the last ending part of the email.
This service is down and is required for vcenter as it is dependency.
Thanks,
MS
Hello msripada and thank you for your reply
The "inv-svc.log" file is very long, it contains several errors like this :
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 75 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)
... 96 more
Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)
at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
... 98 more
Is the "Invalid credentials LDAP error" message the origin of the problem ?
How to resolve this ?
When I open the URL https://myhostname.mydomain.loc:7444/lookupservice/sdk , I have this message :
type status report
message /lookupservice/sdk
description The requested resource is not available
Do the power off and power on , not restart
once done wait for 10 minutes then try to logon
Power off/On does not help here. Please follow this KB VMware Knowledge Base
Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state
Post restarting inventory, you can restart the vpxd service and others.
Thanks,
MS
RajeevVCP4 : if only it was so simple... I rebooted / shutdown many times, but the problem is still there...
msripada : thanks for the link.
I followed all the steps on the KB and... IT WORKS !!!!
Congratulations msripada, you are the best.