7 Replies Latest reply on May 11, 2020 1:16 PM by Vijay2027

    503 Service Unavailable

    LukaszPr Lurker

      Hello,

      I have 2 vcenter 6.5 servers in one sso domain.

      Some time ago I have seen alarm about certificate. So i renewed all from gui, but alarm persisted.

      Today I could not log in to vspere web ui, had error "User name and password are required" on the first server, and blank screen on the second one. So i restarted both. After reboot

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005567895eb3d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

      error is displayed.

      In /var/log/vmware/messages there was error:

      faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 06:42:07 UTC 2020, endTime=Fri May 08 06:52:07 UTC 2020] :: Signing certificate is not valid at Fri May 08 06:42:07 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

      So I have decided to generate new certificates from cli with /usr/lib/vmware-vmca/bin/certificate-manager. Tried regenerating and resetting, but both operations failed when manager tried to start services. Anyway, using cmd /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text showed newly generated certificates.

       

      # service-control --status

      Running:

      applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

      Stopped:

      pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

      Don't know where to look at, but below are some logs. Seems like something with SSO.

      grep -i error /var/log/vmware/vpxd/vpxd.log

      2020-05-08T12:24:43.997Z error vpxd[7F6D42D58800] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

      2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

      2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down

      vapi/endpoint/endpoint.log

      2020-05-08T09:35:46.938Z | ERROR | state-manager1            | ComponentManagerClientWrapper  | SSO lookup failed.

      java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

              at com.vmware.cis.cm.client.ComponentManagerClient.lookup(ComponentManagerClient.java:876)

              at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:939)

              at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:933)

              at com.vmware.cis.cm.client.ComponentManagerClient.retry(ComponentManagerClient.java:548)

              at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:929)

              at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:908)

              at com.vmware.cis.cm.client.ComponentManagerClient.lookupSso(ComponentManagerClient.java:993)

              at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:171)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

      Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

              ... 3 more

      Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

              at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

              at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

              at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

              at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

              at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

              at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

              at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

              ... 3 more

      Caused by: java.net.ConnectException: Connection refused (Connection refused)

              at java.net.PlainSocketImpl.socketConnect(Native Method)

              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

              at java.net.Socket.connect(Socket.java:589)

              at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

              ... 13 more

      2020-05-08T09:35:46.941Z | ERROR | state-manager1            | SsoSettingsBuilder             | Failded to retrieve SSO settings.

      com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

              at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

      Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

              ... 3 more

      Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

              at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

              at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

              at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

              at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

              at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

              at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

              at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

              ... 3 more

      Caused by: java.net.ConnectException: Connection refused (Connection refused)

              at java.net.PlainSocketImpl.socketConnect(Native Method)

              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

              at java.net.Socket.connect(Socket.java:589)

              at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

              ... 13 more

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from component manager.

      2020-05-08T09:35:46.941Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.

      com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:63)

              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

      Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

              at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

              ... 10 more

      Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

              ... 3 more

      Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

              at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

              at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

              at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

              at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

              at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

              at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

              at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

              ... 3 more

      Caused by: java.net.ConnectException: Connection refused (Connection refused)

              at java.net.PlainSocketImpl.socketConnect(Native Method)

              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

              at java.net.Socket.connect(Socket.java:589)

              at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

              at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

              ... 13 more

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from component manager.

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

      2020-05-08T09:35:46.941Z | INFO  | state-manager1            | DefaultStateManager            | lock

      2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.

      2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | unlock

      2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | ApiEndpointServer              | Start shutting down...

      2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | DefaultStateManager            | shutdown

      2020-05-08T09:35:50.614Z | INFO  | shutdown-hook             | ApiEndpointServer              | Shutdown.

      But couldn't find nothing intresting in sso

       

      sso/ssoAdminServer.log

      [2020-05-08T12:24:43.988Z pool-6-thread-5 opId=21190014-abf6-4825-8234-ea672d2cdbb0 ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask] Failed to serialize response

      com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response

              at com.vmware.vim.vmomi.server.exception.ExceptionUtil.buildFaultForInternalException(ExceptionUtil.java:22) ~[vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:95) [vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:63) [vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.run(CompletionContinuerTask.java:53) [vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]

              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]

      Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Missing non-optional return value

              at com.vmware.vim.vmomi.server.impl.SoapBindingImpl.serializeResponse(SoapBindingImpl.java:168) ~[vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:84) ~[vlsi-server.jar:?]

              ... 6 more

      sso/vmware-identity-sts.log //still some problem with certificate?

      [2020-05-08T12:58:34.733Z tomcat-http--39 vsphere.local        97b81203-5e98-4499-b11f-1f5b6ade0789 INFO  com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 12:58:34 UTC 2020, endTime=Fri May 08 13:08:34 UTC 2020] :: Signing certificate is not valid at Fri May 08 12:58:34 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

      sso/lookupServer.log

      [2020-05-08T08:09:40.314Z ERROR] [OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=4cea3f17-670c-4ee6-938c-c7e1aaec7cfe,cn=ServiceRegistrations,cn=LookupService,cn=silp,cn=sites,cn=configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0

      com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object

      Any help would be appreciated