Zarathustra82
Contributor
Contributor

vCenter 6.5 connection stopped working: Unexpected status code: 503

Hi,

After running vCenter 6.5 for around 2 months, I suddenly stopped being able to connect via the web client to it. I tried restarting services, even the entire appliance with no luck. Any help pointing out the issue would be appreciated. I already found another discussion on these forums with the exact same issue that nobody answered so the solution was to move back to 6.0. https://communities.vmware.com/message/2635193

The web interface shows this error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb0a0004530] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

In the vsphere_client_virgo.log I found the following errors:

[2017-02-03T09:52:32.471Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl            Fetching SSO endpoints from CM

[2017-02-03T09:52:32.705Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl            Retrieved locations of services from CM at https://PRG-LAB-VC01.praguelab.internal/cm/sdk?hostid=f68a7a75-4694-4ec9-bd57-0f2af1cd9746 in 249 milliseconds:

adminAppUrl: https://PRG-LAB-VC01.praguelab.internal/sso-adminserver/sdk/praguelabsso.internal, stsUrl: https://PRG-LAB-VC01.praguelab.internal/sts/STSService/praguelabsso.internal, metadataUrl: https://PRG-LAB-VC01.praguelab.internal/websso/SAML2/Metadata/praguelabsso.internal, vmodlVersion: interface com.vmware.vim.binding.sso.version.version3_5

[2017-02-03T09:52:32.917Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery       Site affinity is disabled

[2017-02-03T09:52:33.118Z] [INFO ] CEIPSynchronizerTimer         com.vmware.identity.token.impl.SamlTokenImpl                      SAML token for SubjectNameId [value=vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746@praguelabsso.internal, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element

[2017-02-03T09:52:33.124Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl           Successfully acquired token for user: {Name: vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746, Domain: praguelabsso.internal}

[2017-02-03T09:52:33.224Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl          Initializing vmomi for vc - https://PRG-LAB-VC01.praguelab.internal:443/sdk at VMODL version com.vmware.vim.binding.vim.version.internal.version11

[2017-02-03T09:52:33.288Z] [ERROR] CEIPSynchronizerTimer         com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl          Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503 com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)

        at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)

        at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)

        at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)

        at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)

        at java.util.TimerThread.mainLoop(Timer.java:555)

        at java.util.TimerThread.run(Timer.java:505)

[2017-02-03T09:52:33.290Z] [ERROR] CEIPSynchronizerTimer         com.vmware.ph.ceip.impl.Synchronizer                              Global consent status synchronizer has stopped unexpectedly. As a result the local consent state will not be synchronized with other consent states. To recover please verify/fix the exact reason in the cause exception and restart the service com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$ServiceInitializationException: Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:378)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)

        at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)

        at java.util.TimerThread.mainLoop(Timer.java:555)

        at java.util.TimerThread.run(Timer.java:505)

Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)

        at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)

        at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)

        at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)

        at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)

        at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)

        ... 4 common frames omitted

0 Kudos
2 Replies
mikejtucker
Contributor
Contributor

A couple questions and things to check.

Can you login the PSC or VCSA using https://FQDN:5480?

Is this an embedded PSC or external?

Can you login using administrator@vsphere.local?

Are you running in enhanced link mode?

Is time synced with NTP and set the same as AD, vCenter, PSC and host

Is the root password set to never expire on the PSC

0 Kudos
Dimon_SPB
Contributor
Contributor

I have a same problem.

Can you login the PSC or VCSA using https://FQDN:5480?     Yes

Is this an embedded PSC or external?         embedded

Can you login using administrator@vsphere.local?           How? If the Web client isn't available

Are you running in enhanced link mode?   No

Is time synced with NTP and set the same as AD, vCenter, PSC and host   Yes

Is the root password set to never expire on the PSC     Yes

0 Kudos