2 Replies Latest reply on Apr 10, 2017 1:38 PM by Dimon_SPB

    vCenter 6.5 connection stopped working: Unexpected status code: 503

    Zarathustra82 Novice

      Hi,

       

      After running vCenter 6.5 for around 2 months, I suddenly stopped being able to connect via the web client to it. I tried restarting services, even the entire appliance with no luck. Any help pointing out the issue would be appreciated. I already found another discussion on these forums with the exact same issue that nobody answered so the solution was to move back to 6.0. https://communities.vmware.com/message/2635193

       

      The web interface shows this error:

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb0a0004530] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

       

      In the vsphere_client_virgo.log I found the following errors:

       

      [2017-02-03T09:52:32.471Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl            Fetching SSO endpoints from CM

      [2017-02-03T09:52:32.705Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl            Retrieved locations of services from CM at https://PRG-LAB-VC01.praguelab.internal/cm/sdk?hostid=f68a7a75-4694-4ec9-bd57-0f2af1cd9746 in 249 milliseconds:

      adminAppUrl: https://PRG-LAB-VC01.praguelab.internal/sso-adminserver/sdk/praguelabsso.internal, stsUrl: https://PRG-LAB-VC01.praguelab.internal/sts/STSService/praguelabsso.internal, metadataUrl: https://PRG-LAB-VC01.praguelab.internal/websso/SAML2/Metadata/praguelabsso.internal, vmodlVersion: interface com.vmware.vim.binding.sso.version.version3_5

      [2017-02-03T09:52:32.917Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery       Site affinity is disabled

      [2017-02-03T09:52:33.118Z] [INFO ] CEIPSynchronizerTimer         com.vmware.identity.token.impl.SamlTokenImpl                      SAML token for SubjectNameId [value=vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746@praguelabsso.internal, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element

      [2017-02-03T09:52:33.124Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl           Successfully acquired token for user: {Name: vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746, Domain: praguelabsso.internal}

      [2017-02-03T09:52:33.224Z] [INFO ] CEIPSynchronizerTimer         com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl          Initializing vmomi for vc - https://PRG-LAB-VC01.praguelab.internal:443/sdk at VMODL version com.vmware.vim.binding.vim.version.internal.version11

      [2017-02-03T09:52:33.288Z] [ERROR] CEIPSynchronizerTimer         com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl          Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503 com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

              at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)

              at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)

              at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)

              at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)

              at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)

              at java.util.TimerThread.mainLoop(Timer.java:555)

              at java.util.TimerThread.run(Timer.java:505)

       

       

      [2017-02-03T09:52:33.290Z] [ERROR] CEIPSynchronizerTimer         com.vmware.ph.ceip.impl.Synchronizer                              Global consent status synchronizer has stopped unexpectedly. As a result the local consent state will not be synchronized with other consent states. To recover please verify/fix the exact reason in the cause exception and restart the service com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$ServiceInitializationException: Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:378)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)

              at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)

              at java.util.TimerThread.mainLoop(Timer.java:555)

              at java.util.TimerThread.run(Timer.java:505)

      Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

              at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)

              at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)

              at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)

              at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)

              at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)

              at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)

              ... 4 common frames omitted