Hi,
After running vCenter 6.5 for around 2 months, I suddenly stopped being able to connect via the web client to it. I tried restarting services, even the entire appliance with no luck. Any help pointing out the issue would be appreciated. I already found another discussion on these forums with the exact same issue that nobody answered so the solution was to move back to 6.0. https://communities.vmware.com/message/2635193
The web interface shows this error:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb0a0004530] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
In the vsphere_client_virgo.log I found the following errors:
[2017-02-03T09:52:32.471Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl Fetching SSO endpoints from CM
[2017-02-03T09:52:32.705Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl Retrieved locations of services from CM at https://PRG-LAB-VC01.praguelab.internal/cm/sdk?hostid=f68a7a75-4694-4ec9-bd57-0f2af1cd9746 in 249 milliseconds:
adminAppUrl: https://PRG-LAB-VC01.praguelab.internal/sso-adminserver/sdk/praguelabsso.internal, stsUrl: https://PRG-LAB-VC01.praguelab.internal/sts/STSService/praguelabsso.internal, metadataUrl: https://PRG-LAB-VC01.praguelab.internal/websso/SAML2/Metadata/praguelabsso.internal, vmodlVersion: interface com.vmware.vim.binding.sso.version.version3_5
[2017-02-03T09:52:32.917Z] [INFO ] CEIPSynchronizerTimer com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery Site affinity is disabled
[2017-02-03T09:52:33.118Z] [INFO ] CEIPSynchronizerTimer com.vmware.identity.token.impl.SamlTokenImpl SAML token for SubjectNameId [value=vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746@praguelabsso.internal, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
[2017-02-03T09:52:33.124Z] [INFO ] CEIPSynchronizerTimer com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl Successfully acquired token for user: {Name: vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746, Domain: praguelabsso.internal}
[2017-02-03T09:52:33.224Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl Initializing vmomi for vc - https://PRG-LAB-VC01.praguelab.internal:443/sdk at VMODL version com.vmware.vim.binding.vim.version.internal.version11
[2017-02-03T09:52:33.288Z] [ERROR] CEIPSynchronizerTimer com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503 com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)
at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)
at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-03T09:52:33.290Z] [ERROR] CEIPSynchronizerTimer com.vmware.ph.ceip.impl.Synchronizer Global consent status synchronizer has stopped unexpectedly. As a result the local consent state will not be synchronized with other consent states. To recover please verify/fix the exact reason in the cause exception and restart the service com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$ServiceInitializationException: Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:378)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)
at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)
at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)
... 4 common frames omitted
A couple questions and things to check.
Can you login the PSC or VCSA using https://FQDN:5480?
Is this an embedded PSC or external?
Can you login using administrator@vsphere.local?
Are you running in enhanced link mode?
Is time synced with NTP and set the same as AD, vCenter, PSC and host
Is the root password set to never expire on the PSC
I have a same problem.
Can you login the PSC or VCSA using https://FQDN:5480? Yes
Is this an embedded PSC or external? embedded
Can you login using administrator@vsphere.local? How? If the Web client isn't available
Are you running in enhanced link mode? No
Is time synced with NTP and set the same as AD, vCenter, PSC and host Yes
Is the root password set to never expire on the PSC Yes