vCenter server 6.5 throws HTTP 503 errors

[srebitnami]

Hi all!     I'm obtaining HTTP 503 errors in vCenter.

If I try to access vcenter webserver at  / location I get:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f5150003020] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

If I try to access vcenter/vsphere-client/, the login screen shows up, but after a successful login, I got a message that says: "Could not connect to one or more vCenter systems at vcenter.vrealize.local/sdk"

If I try to access vcenter's /sdk, I obtain

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007f5150001ea0] _serverNamespace = /sdk action = Allow _port = 8085)

I'm also obtaining errors on other several files under /var/log/vmware/

eam.log:

2018-01-18T09:30:49.011Z |  WARN | vim-monitor | VcListener.java | 110 | Trying to recover from error
com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
    at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
    at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
    at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
    at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
    at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
    at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:580)
    at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:561)
    at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:347)
    at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:307)
    at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:181)
    at com.sun.proxy.$Proxy32.retrieveContent(Unknown Source)
    at com.vmware.eam.vc.VcConnection.retrieveServiceContent(VcConnection.java:645)
    at com.vmware.eam.vc.VcConnection.retrieveContentEam(VcConnection.java:547)
    at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:199)
    at com.vmware.eam.vc.VcListener.main(VcListener.java:125)
    at com.vmware.eam.vc.VcListener.call(VcListener.java:108)
    at com.vmware.eam.vc.VcListener.call(VcListener.java:54)
    at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
    at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2018-01-18T09:30:49.011Z |  INFO | vim-monitor | VcListener.java | 117 | Retrying in 10 sec.
2018-01-18T09:30:49.011Z |  WARN | VLSI-client-connection-monitor-5693 | HttpConfigurationCompilerBase.java | 81 | Interrupted, no more connection pool cleanups will be performed.

vapi/endpoint/endpoint.log:

2018-01-18T09:32:07.642Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:9090/ds/vapi
2018-01-18T09:32:07.642Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8900/vmonapi
2018-01-18T09:32:07.642Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vim-adapter-settings-builder
2018-01-18T09:32:07.700Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vapi-vcenter-servlet-builder
2018-01-18T09:32:07.700Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild api-interfaces-builder
2018-01-18T09:32:07.722Z | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service 57298db7-629f-40a4-9e31-966b0b9dc122\com.vmware.cis.cls.vapi has failed.
com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): <HTML><BODY><H1>503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007f5e64002740] _serverNamespace = /cls/ action = Allow _port = 16666)</H1></BODY></HTML>
    at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpUtil.validateHttpResponse(ApacheHttpUtil.java:71)
    at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$SingleResponseConsumer.responseCompleted(ApacheHttpAsyncClientTransport.java:467)
    at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$DecoratorConsumer.responseCompleted(ApacheHttpAsyncClientTransport.java:333)
    at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:418)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:164)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:355)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:242)
    at org.apache.http.impl.nio.client.LoggingAsyncRequestExecutor.inputReady(LoggingAsyncRequestExecutor.java:87)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:264)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:73)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:37)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)

In vpxd.log :

2018-01-17T10:20:44.460Z error vpxd[7F86FD555800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)
2018-01-17T10:20:44.463Z error vpxd[7F86FD555800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument
2018-01-17T10:20:46.615Z error vpxd[7F86EEEB3700] [Originator@6876 sub=vmomi.soapStub[5]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
2018-01-17T10:20:46.670Z error vpxd[7F86EDE93700] [Originator@6876 sub=vmomi.soapStub[7]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
2018-01-17T10:20:46.895Z error vpxd[7F86FD555800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for embeddedEsx version e.x.p: vpx-upgrade-eesx-1-linux-*
2018-01-17T10:20:46.895Z error vpxd[7F86FD555800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for esx version e.x.p: vpx-upgrade-esx-7-linux-*
2018-01-17T10:20:47.156Z error vpxd[7F86B7870700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2018-01-17T10:20:47.183Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\bitnami1"
2018-01-17T10:20:47.183Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\bitnami1"
2018-01-17T10:20:47.183Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\bitnami1"
2018-01-17T10:20:47.183Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2018-01-17T10:20:47.183Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\bitnami1"
2018-01-17T10:20:47.319Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] Failed to add default permission: permission already exists
2018-01-17T10:20:47.319Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] Cannot start authorize - system has no access rules
2018-01-17T10:20:47.319Z error vpxd[7F86FD555800] [Originator@6876 sub=AuthorizeManager] [Auth] Failed to initialize: <N7Vmacore9Authorize13AuthExceptionE(Authorize Exception)
2018-01-17T10:20:47.320Z error vpxd[7F86FD555800] [Originator@6876 sub=Default] Failed to initialize authorizeManager
2018-01-17T10:20:47.320Z error vpxd[7F86FD555800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager
2018-01-17T10:20:47.320Z error vpxd[7F86FD555800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down

As you see I'm obtaining a lot of 400 and 50X HTTP errors, but service-control --status --all shows that the only failed service is `vpxd`

I think it's important to note that we needed to rotate the `administrator@vrealize.local` password on our vRA's Iaas  AD.

[msripada]

vpxd failed to start for multiple reasons and we can only identify from the latest vpxd.log

Can you post that logs please.

Thanks,

MS

[admin]

Can you attach the sceenshot of the error? Are you able to login the webclient? and if possible attach the vcsupport bundle.

Regards,

Randhir