DataSovince
Contributor
Contributor

Vcenter error 503

Jump to solution

Hi !

I have a problem with my VCSA server (v 6.0.0.2)

The server storage was full and I had to delete some *.log files to start the server.

But when I restarted the VCSA, I try to connect to the web page of the vSphere Web Client, this message is displayed :

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f92c30900f0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

I checked the running services with the command service-control --status and the result is :

INFO:root:Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

INFO:root:Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

I don't know what to do next. Could you help me to recover my VCSA server ?

0 Kudos
1 Solution

Accepted Solutions
msripada
Virtuoso
Virtuoso

Power off/On does not help here. Please follow this KB VMware Knowledge Base

Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state

Post restarting inventory, you can restart the vpxd service and others.

Thanks,

MS

View solution in original post

0 Kudos
7 Replies
RajeevVCP4
Expert
Expert

Are you using external PSC or Embedded ?

Provide df -h output

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
0 Kudos
DataSovince
Contributor
Contributor

Thanks for your reply :

The primary service controller is embedded.

The result of the command

/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost

=>

https://*********/lookupservice/sdk

(******* is the FQDN of the VCSA)

and the result of df -h :

df-h.png

0 Kudos
msripada
Virtuoso
Virtuoso

Hello...

Can you check the /var/log/vmware/invsvc/invsvc.log output and share the last ending part of the email.

This service is down and is required for vcenter as it is dependency.

Thanks,

MS

0 Kudos
DataSovince
Contributor
Contributor

Hello msripada and thank you for your reply

The "inv-svc.log" file is very long, it contains several errors like this :

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

        ... 75 more

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

        ... 85 more

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)

        ... 96 more

Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)

        at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

        at java.lang.reflect.Constructor.newInstance(Unknown Source)

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)

        ... 98 more

Is the "Invalid credentials LDAP error" message the origin of the problem ?

How to resolve this ?

When I open the URL https://myhostname.mydomain.loc:7444/lookupservice/sdk , I have this message :

Etat HTTP 404 - /lookupservice/sdk

type status report

message /lookupservice/sdk

description The requested resource is not available


Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE

0 Kudos
RajeevVCP4
Expert
Expert

Do the power off and power on , not restart

once done wait for 10 minutes then try to logon

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
0 Kudos
msripada
Virtuoso
Virtuoso

Power off/On does not help here. Please follow this KB VMware Knowledge Base

Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state

Post restarting inventory, you can restart the vpxd service and others.

Thanks,

MS

0 Kudos
DataSovince
Contributor
Contributor

RajeevVCP4​ : if only it was so simple... I rebooted / shutdown many times, but the problem is still there...

msripada​ : thanks for the link.

I followed all the steps on the KB and... IT WORKS !!!! Heart

Congratulations msripada, you are the best.

0 Kudos