503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000d54dcd5ef0] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

fhefter · ‎12-01-2016

Hallo,

we just installed a latest 6.5 vCenter on Windows 2012R2 with the latest patches.

Memory 16 GB / 4 Cores in a ESXi 6.0 VM, plenty of HD memory free.

The first few days the server worked very well. Without any changes it begun to get more and more unresponsive.

Finally we can't work anymore with it. After restart it is now not responding anymore. See below ...

After startup is finished the Web Client shows the following message.

URL: https://192.168.2.10/‌

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000d54dcd5ef0] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

The URL: https://192.168.2.10/‌vsphere-client/?csp

Show the login screen:

After login, the mask (sectioned screen, On the left "Hosts and Clusters" is selected by default.) without content in the white areas, appears. It never fills (waited 30 minutes).

On top a message says:

"Es konnte keine Verbindung zu einem oder mehreren vCenter Server-Systemen hergestellt werden:

https://vCenter.test.com:443/sdk"

When I go to this url I get this message:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x000000c10f8bf5d0] _serverNamespace = /sdk action = Allow _port = 8085)

Going to Home (Button on top) shows the expected screen.

But "Hosts and Clusters" does not respond.

All other Menues on the left do work.

Memory consumption is 7,5GB of 16GB, no load on CPU (4%).

-------

I triple checked all hosts files and IP addresses.

I checked vsphere_client_virgo.log and found many exceptions.

.....

[2016-11-30T23:56:57.349-08:00] [WARN ] cm-catalog-manager-pool-17

com.vmware.vise.vim.cm.CmCatalogManager

downloadResourceBundle: Download C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\cmCatalog\com.vmware.imagebuilder.imagebuilder.zip

from https://vCenter.test.com:443/sca/resourcebundle/imagebuilderResourceBundle.zip failed, but previous local cache exists and will be used.

Error message is: org.apache.http.conn.HttpHostConnectException: Connect to vCenter.test.com:443 [vCenter.test.com/192.168.2.10, vCenter.test.com/10.1.1.10, vCenter.test.com/0:0:0:0:0:0:0:1]

failed: Connection refused: connect

....

[2016-12-01T06:08:04.549-08:00] [ERROR] CEIPSynchronizerTimer com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503 com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)

at com.sun.proxy.$Proxy617.retrieveContent(Unknown Source)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)

at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)

at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)

at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)

at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

....

[2016-12-01T15:55:03.498+01:00] [WARN ] data-service-pool-136 70000186 100003 200002 c.v.l.client.cis.ls.impl.CacheAuthorizationManagerWrapperImpl Cannot cache licensing admin privilege granted for scope: licensing_service_version1;

AuthorizationException occurred: com.vmware.license.client.cis.ls.AuthorizationException: com.vmware.vim.binding.vmodl.RuntimeFault: Berechtigungen k n nicht berechnet werden. Weitere Details finden Sie im Protokoll.

at com.vmware.license.client.cis.ls.impl.VpxAuthorizationManagerImpl.isLicensingAdminPrivilegeGranted(VpxAuthorizationManagerImpl.java:69)

at com.vmware.license.client.cis.ls.impl.CacheAuthorizationManagerWrapperImpl.isLicensingAdminPrivilegeGranted(CacheAuthorizationManagerWrapperImpl.java:63)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy360.isLicensingAdminPrivilegeGranted(Unknown Source)

at com.vmware.license.client.cis.impl.LicensingGlobalDataEditorImpl.isLicensingAdminPrivilegeGranted(LicensingGlobalDataEditorImpl.java:90)

at com.vmware.license.client.cis.impl.LicensingGlobalDataEditorImpl.getLicensingGlobalData(LicensingGlobalDataEditorImpl.java:69)

at com.vmware.license.client.cis.adapter.LicensingGlobalDataProviderAdapterImpl.getObjectsByUris(LicensingGlobalDataProviderAdapterImpl.java:67)

at com.vmware.license.client.cis.adapter.LicensingGlobalDataProviderAdapterImpl.getObjectsByUris(LicensingGlobalDataProviderAdapterImpl.java:18)

at com.vmware.license.client.cis.adapter.BaseLicensingObjectsDataProviderAdapterImpl.getObjectsByRefs(BaseLicensingObjectsDataProviderAdapterImpl.java:230)

at com.vmware.license.client.cis.adapter.BaseLicensingObjectsDataProviderAdapterImpl.getObjectsByQuerySpec(BaseLicensingObjectsDataProviderAdapterImpl.java:196)

at com.vmware.license.client.cis.adapter.BaseLicensingObjectsDataProviderAdapterImpl.getResultSet(BaseLicensingObjectsDataProviderAdapterImpl.java:42)

at com.vmware.license.client.cis.adapter.BaseDataProviderAdapterImpl.processQuery(BaseDataProviderAdapterImpl.java:66)

at com.vmware.license.client.cis.adapter.BaseDataProviderAdapterImpl.getData(BaseDataProviderAdapterImpl.java:39)

at com.vmware.vise.data.query.impl.DataAdapterUtil.getDataFromAdapter(DataAdapterUtil.java:218)

at com.vmware.vise.data.query.impl.DataAdapterUtil.getResultFromAdapter(DataAdapterUtil.java:175)

at com.vmware.vise.data.query.impl.DataAdapterUtil.access$000(DataAdapterUtil.java:42)

at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:81)

at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:78)

at com.vmware.vise.util.concurrent.ExecutorUtil$2.call(ExecutorUtil.java:654)

at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingTask.call(ExecutorUtil.java:957)

at com.vmware.vise.data.query.internal.profiling.MdcAwareExecutorService$1.call(MdcAwareExecutorService.java:148)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.vmware.vim.binding.vmodl.RuntimeFault: Berechtigungen k n nicht berechnet werden. Weitere Details finden Sie im Protokoll.

at com.vmware.vise.vim.security.impl.LegacyAuthorizationServiceImpl.throwRuntimeFault(LegacyAuthorizationServiceImpl.java:259)

at com.vmware.vise.vim.security.impl.LegacyAuthorizationServiceImpl.getPrivilegesInternal(LegacyAuthorizationServiceImpl.java:161)

at com.vmware.vise.vim.security.impl.LegacyAuthorizationServiceImpl.getPrivileges(LegacyAuthorizationServiceImpl.java:131)

at com.vmware.vise.vim.security.impl.LegacyAuthorizationServiceImpl.checkPrivileges(LegacyAuthorizationServiceImpl.java:98)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy353.checkPrivileges(Unknown Source)

at com.vmware.license.client.cis.ls.impl.VpxAuthorizationManagerImpl.isVpxLicenseAdminPrivilegeGranted(VpxAuthorizationManagerImpl.java:85)

at com.vmware.license.client.cis.ls.impl.VpxAuthorizationManagerImpl.isLicensingAdminPrivilegeGranted(VpxAuthorizationManagerImpl.java:64)

... 34 common frames omitted

Caused by: com.vmware.vim.binding.vmodl.MethodFault: Die Verbindung zu vCenter Server '93e45013-0c89-43ad-9680-152f09f10333' wurde unterbrochen.

at com.vmware.vise.vim.commons.MixedUtil.getMethodFault(MixedUtil.java:98)

at com.vmware.vise.vim.data.adapters.search.impl.PropertyCollectorDataAdapter.getResultSet(PropertyCollectorDataAdapter.java:747)

at com.vmware.vise.vim.data.adapters.search.impl.PropertyCollectorDataAdapter.getResultSet(PropertyCollectorDataAdapter.java:630)

at com.vmware.vise.vim.data.adapters.search.impl.PropertyCollectorDataAdapter.getProperties(PropertyCollectorDataAdapter.java:340)

at com.vmware.vise.data.query.impl.DataManager.getDataFromPropertyProvider(DataManager.java:1301)

at com.vmware.vise.data.query.impl.DataManager.getResultFromPropertyProvider(DataManager.java:1269)

at com.vmware.vise.data.query.impl.DataManager.access$000(DataManager.java:78)

at com.vmware.vise.data.query.impl.DataManager$1.call(DataManager.java:784)

at com.vmware.vise.data.query.impl.DataManager$1.call(DataManager.java:780)

... 7 common frames omitted

Caused by: com.vmware.vise.vim.commons.vcservice.PeerVcConnectionError: Die Verbindung zu vCenter Server '93e45013-0c89-43ad-9680-152f09f10333' wurde unterbrochen.

at com.vmware.vise.vim.data.adapters.search.impl.PropertyCollectorDataAdapter.createPropertyCollectorStub(PropertyCollectorDataAdapter.java:804)

at com.vmware.vise.vim.data.adapters.search.impl.PropertyCollectorDataAdapter.getResultSet(PropertyCollectorDataAdapter.java:688)

... 14 common frames omitted

[2016-12-01T15:55:03.501+01:00] [ERROR] data-service-pool-136 70000186 100003 200002 com.vmware.license.client.cis.impl.LicensingGlobalDataEditorImpl Error when retrieving licensing admin privileges. com.vmware.license.client.cis.ls.AuthorizationException: com.vmware.vim.binding.vmodl.RuntimeFault: Berechtigungen k n nicht berechnet werden. Weitere Details finden Sie im Protokoll.