Hello,
I have 2 vcenter 6.5 servers in one sso domain.
Some time ago I have seen alarm about certificate. So i renewed all from gui, but alarm persisted.
Today I could not log in to vspere web ui, had error "User name and password are required" on the first server, and blank screen on the second one. So i restarted both. After reboot
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005567895eb3d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
error is displayed.
In /var/log/vmware/messages there was error:
faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 06:42:07 UTC 2020, endTime=Fri May 08 06:52:07 UTC 2020] :: Signing certificate is not valid at Fri May 08 06:42:07 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]
So I have decided to generate new certificates from cli with /usr/lib/vmware-vmca/bin/certificate-manager. Tried regenerating and resetting, but both operations failed when manager tried to start services. Anyway, using cmd /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text showed newly generated certificates.
# service-control --status
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
Stopped:
pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm
Don't know where to look at, but below are some logs. Seems like something with SSO.
grep -i error /var/log/vmware/vpxd/vpxd.log
2020-05-08T12:24:43.997Z error vpxd[7F6D42D58800] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
vapi/endpoint/endpoint.log
2020-05-08T09:35:46.938Z | ERROR | state-manager1 | ComponentManagerClientWrapper | SSO lookup failed.
java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.cis.cm.client.ComponentManagerClient.lookup(ComponentManagerClient.java:876)
at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:939)
at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:933)
at com.vmware.cis.cm.client.ComponentManagerClient.retry(ComponentManagerClient.java:548)
at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:929)
at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:908)
at com.vmware.cis.cm.client.ComponentManagerClient.lookupSso(ComponentManagerClient.java:993)
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | ERROR | state-manager1 | SsoSettingsBuilder | Failded to retrieve SSO settings.
com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH ORANGE Failed to retrieve SSO settings from component manager.
2020-05-08T09:35:46.941Z | ERROR | state-manager1 | DefaultStateManager | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:63)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.
at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)
... 10 more
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 13 more
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthStatusCollectorImpl | HEALTH GREEN Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | Computed health status is = ORANGE
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from component manager.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.
2020-05-08T09:35:46.941Z | INFO | state-manager1 | DefaultStateManager | lock
2020-05-08T09:35:46.942Z | INFO | state-manager1 | DefaultStateManager | Initial state build failed. Will retry after 5 seconds.
2020-05-08T09:35:46.942Z | INFO | state-manager1 | DefaultStateManager | unlock
2020-05-08T09:35:50.607Z | INFO | shutdown-hook | ApiEndpointServer | Start shutting down...
2020-05-08T09:35:50.607Z | INFO | shutdown-hook | DefaultStateManager | shutdown
2020-05-08T09:35:50.614Z | INFO | shutdown-hook | ApiEndpointServer | Shutdown.
But couldn't find nothing intresting in sso
sso/ssoAdminServer.log
[2020-05-08T12:24:43.988Z pool-6-thread-5 opId=21190014-abf6-4825-8234-ea672d2cdbb0 ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask] Failed to serialize response
com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
at com.vmware.vim.vmomi.server.exception.ExceptionUtil.buildFaultForInternalException(ExceptionUtil.java:22) ~[vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:95) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:63) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.run(CompletionContinuerTask.java:53) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Missing non-optional return value
at com.vmware.vim.vmomi.server.impl.SoapBindingImpl.serializeResponse(SoapBindingImpl.java:168) ~[vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:84) ~[vlsi-server.jar:?]
... 6 more
sso/vmware-identity-sts.log //still some problem with certificate?
[2020-05-08T12:58:34.733Z tomcat-http--39 vsphere.local 97b81203-5e98-4499-b11f-1f5b6ade0789 INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 12:58:34 UTC 2020, endTime=Fri May 08 13:08:34 UTC 2020] :: Signing certificate is not valid at Fri May 08 12:58:34 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]
sso/lookupServer.log
[2020-05-08T08:09:40.314Z ERROR] [OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=4cea3f17-670c-4ee6-938c-c7e1aaec7cfe,cn=ServiceRegistrations,cn=LookupService,cn=silp,cn=sites,cn=configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0
com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object
Any help would be appreciated
one option is to set wrong date on vcsa, start services and follow the below process:
Generate a New STS Signing Certificate on the Appliance
Refresh the Security Token Service Certificate
GSS has an script to automate this process. You may open a SR.
Have you opened a support request with VMware GSS?
Moderator: Thread moved to the vCenter Server area.
not yet, as I don't have required permissions. Will have to wait for that until Monday.
Looks like STS certificate is expired on this.
one option is to set wrong date on vcsa, start services and follow the below process:
Generate a New STS Signing Certificate on the Appliance
Refresh the Security Token Service Certificate
GSS has an script to automate this process. You may open a SR.
You were right, the cause of the issue was expired STS certificate. Problem resolved by vmware support.
Good to know