503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f92c30900f0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

DataSovince · ‎12-26-2018

Hi !

I have a problem with my VCSA server (v 6.0.0.2)

The server storage was full and I had to delete some *.log files to start the server.

But when I restarted the VCSA, I try to connect to the web page of the vSphere Web Client, this message is displayed :

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f92c30900f0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

I checked the running services with the command service-control --status and the result is :

INFO:root:Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

INFO:root:Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

I don't know what to do next. Could you help me to recover my VCSA server ?

msripada · ‎12-26-2018

Power off/On does not help here. Please follow this KB VMware Knowledge Base

Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state

Post restarting inventory, you can restart the vpxd service and others.

Thanks,

MS

View solution in original post

RajeevVCP4 · ‎12-26-2018

Are you using external PSC or Embedded ?

Provide df -h output

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you

DataSovince · ‎12-26-2018

Thanks for your reply :

The primary service controller is embedded.

The result of the command

/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost

=>

https://*********/lookupservice/sdk

(******* is the FQDN of the VCSA)

and the result of df -h :

msripada · ‎12-26-2018

Hello...

Can you check the /var/log/vmware/invsvc/invsvc.log output and share the last ending part of the email.

This service is down and is required for vcenter as it is dependency.

Thanks,

MS

DataSovince · ‎12-26-2018

Hello msripada and thank you for your reply

The "inv-svc.log" file is very long, it contains several errors like this :

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 75 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)
        ... 96 more
Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
        at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)
        at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
        ... 98 more

Is the "Invalid credentials LDAP error" message the origin of the problem ?

How to resolve this ?

When I open the URL https://myhostname.mydomain.loc:7444/lookupservice/sdk , I have this message :

Etat HTTP 404 - /lookupservice/sdk

type status report
message /lookupservice/sdk
description The requested resource is not available

Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE

RajeevVCP4 · ‎12-26-2018

Do the power off and power on , not restart

once done wait for 10 minutes then try to logon

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you

msripada · ‎12-26-2018

Power off/On does not help here. Please follow this KB VMware Knowledge Base

Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state

Post restarting inventory, you can restart the vpxd service and others.

Thanks,

MS

DataSovince · ‎12-27-2018

RajeevVCP4 : if only it was so simple... I rebooted / shutdown many times, but the problem is still there...

msripada : thanks for the link.

I followed all the steps on the KB and... IT WORKS !!!!

Congratulations msripada, you are the best.

All

Vcenter error 503

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f92c30900f0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Etat HTTP 404 - /lookupservice/sdk

Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE