VMware Cloud Community
LukaszPr
Contributor
Contributor
Jump to solution

503 Service Unavailable

Hello,

I have 2 vcenter 6.5 servers in one sso domain.

Some time ago I have seen alarm about certificate. So i renewed all from gui, but alarm persisted.

Today I could not log in to vspere web ui, had error "User name and password are required" on the first server, and blank screen on the second one. So i restarted both. After reboot

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005567895eb3d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

error is displayed.

In /var/log/vmware/messages there was error:

faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 06:42:07 UTC 2020, endTime=Fri May 08 06:52:07 UTC 2020] :: Signing certificate is not valid at Fri May 08 06:42:07 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

So I have decided to generate new certificates from cli with /usr/lib/vmware-vmca/bin/certificate-manager. Tried regenerating and resetting, but both operations failed when manager tried to start services. Anyway, using cmd /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text showed newly generated certificates.

# service-control --status

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

Don't know where to look at, but below are some logs. Seems like something with SSO.

grep -i error /var/log/vmware/vpxd/vpxd.log

2020-05-08T12:24:43.997Z error vpxd[7F6D42D58800] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down

vapi/endpoint/endpoint.log

2020-05-08T09:35:46.938Z | ERROR | state-manager1            | ComponentManagerClientWrapper  | SSO lookup failed.

java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

        at com.vmware.cis.cm.client.ComponentManagerClient.lookup(ComponentManagerClient.java:876)

        at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:939)

        at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:933)

        at com.vmware.cis.cm.client.ComponentManagerClient.retry(ComponentManagerClient.java:548)

        at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:929)

        at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:908)

        at com.vmware.cis.cm.client.ComponentManagerClient.lookupSso(ComponentManagerClient.java:993)

        at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:171)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

        ... 3 more

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

        ... 3 more

Caused by: java.net.ConnectException: Connection refused (Connection refused)

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

        ... 13 more

2020-05-08T09:35:46.941Z | ERROR | state-manager1            | SsoSettingsBuilder             | Failded to retrieve SSO settings.

com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

        at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

        ... 3 more

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

        ... 3 more

Caused by: java.net.ConnectException: Connection refused (Connection refused)

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

        ... 13 more

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from component manager.

2020-05-08T09:35:46.941Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.

com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:63)

        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

        at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

        ... 10 more

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

        ... 3 more

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

        ... 3 more

Caused by: java.net.ConnectException: Connection refused (Connection refused)

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

        ... 13 more

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from component manager.

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

2020-05-08T09:35:46.941Z | INFO  | state-manager1            | DefaultStateManager            | lock

2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.

2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | unlock

2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | ApiEndpointServer              | Start shutting down...

2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | DefaultStateManager            | shutdown

2020-05-08T09:35:50.614Z | INFO  | shutdown-hook             | ApiEndpointServer              | Shutdown.

But couldn't find nothing intresting in sso

sso/ssoAdminServer.log

[2020-05-08T12:24:43.988Z pool-6-thread-5 opId=21190014-abf6-4825-8234-ea672d2cdbb0 ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask] Failed to serialize response

com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response

        at com.vmware.vim.vmomi.server.exception.ExceptionUtil.buildFaultForInternalException(ExceptionUtil.java:22) ~[vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:95) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:63) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.run(CompletionContinuerTask.java:53) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]

Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Missing non-optional return value

        at com.vmware.vim.vmomi.server.impl.SoapBindingImpl.serializeResponse(SoapBindingImpl.java:168) ~[vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:84) ~[vlsi-server.jar:?]

        ... 6 more

sso/vmware-identity-sts.log //still some problem with certificate?

[2020-05-08T12:58:34.733Z tomcat-http--39 vsphere.local        97b81203-5e98-4499-b11f-1f5b6ade0789 INFO  com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 12:58:34 UTC 2020, endTime=Fri May 08 13:08:34 UTC 2020] :: Signing certificate is not valid at Fri May 08 12:58:34 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

sso/lookupServer.log

[2020-05-08T08:09:40.314Z ERROR] [OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=4cea3f17-670c-4ee6-938c-c7e1aaec7cfe,cn=ServiceRegistrations,cn=LookupService,cn=silp,cn=sites,cn=configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0

com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object

Any help would be appreciated

Tags (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
Vijay2027
Expert
Expert
Jump to solution

one option is to set wrong date on vcsa, start services and follow the below process:

Generate a New STS Signing Certificate on the Appliance

Refresh the Security Token Service Certificate

GSS has an script to automate this process. You may open a SR.

View solution in original post

Reply
0 Kudos
7 Replies
daphnissov
Immortal
Immortal
Jump to solution

Have you opened a support request with VMware GSS?

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee
Jump to solution

Moderator: Thread moved to the vCenter Server area.


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
LukaszPr
Contributor
Contributor
Jump to solution

not yet, as I don't have required permissions. Will have to wait for that until Monday.

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

Looks like STS certificate is expired on this.

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

one option is to set wrong date on vcsa, start services and follow the below process:

Generate a New STS Signing Certificate on the Appliance

Refresh the Security Token Service Certificate

GSS has an script to automate this process. You may open a SR.

Reply
0 Kudos
LukaszPr
Contributor
Contributor
Jump to solution

You were right, the cause of the issue was expired STS certificate. Problem resolved by vmware support.

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

Good to know Smiley Happy

Reply
0 Kudos