Error in vsphere after fail over
ISO_Ocarrasco Aug 14, 2018 1:04 PMHi people,
I have a serius problem with the vCenter appliance after a fail over is done. (esxi 6.7, vsphere 6.7)
When the host server shutdowns all the VMs moves correctly to the backup host but the vcenter appliance never works again.
If I try to access the application with a internet browser I get this message:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005557ef930870] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
I connect to vcenter with ssh and list all the services status:
root@photon-machine [ ~ ]# service-control --status
Stopped:
pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
I think the problems comes from here, the vmware-cm didn't start. (and maybe all of the others services need to start too).
I tried to start it manually, but never starts again, in the log there is not error or something similar:
2018-08-14T16:39:31.107Z [main [] INFO com.vmware.cis.services.cm.service.Application ()] start: Starting Component Manager
2018-08-14T16:39:31.158Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:31.186Z [main [] INFO com.vmware.cis.services.cm.service.Application ()] start: Using Spring configuration file: 'http.xml'
2018-08-14T16:39:34.081Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:36.773Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:38.365Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] Using VECS store [TRUSTED_ROOTS]
2018-08-14T16:39:38.366Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] Using VECS store [machine]
2018-08-14T16:39:38.448Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] SecurityAgent: Connecting to SSO service
2018-08-14T16:39:39.731Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:39:39.755Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:39:39.868Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@530712d: startup date [Tue Aug 14 16:39:39 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.438Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@530712d: startup date [Tue Aug 14 16:39:39 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.439Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl loaded in 590 millis
2018-08-14T16:39:40.439Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@68ead359: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.814Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@68ead359: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.814Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl.query loaded in 375 millis
2018-08-14T16:39:40.847Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@47a5b70d: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@47a5b70d: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl loaded in 162 millis
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@56c698e3: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.098Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@56c698e3: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.098Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl.query loaded in 89 millis
2018-08-14T16:39:41.106Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@7fe7c640: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.277Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@7fe7c640: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.277Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.cis.cm loaded in 172 millis
2018-08-14T16:39:41.278Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@549949be: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.385Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@549949be: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.385Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.cis.common loaded in 107 millis
2018-08-14T16:39:41.386Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@3f0846c6: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.568Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@3f0846c6: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.570Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.lookup loaded in 184 millis
2018-08-14T16:39:43.394Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:39:45.559Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:32.462Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] createSsoAdminClient: Connecting to SSO service at https://10.161.254.155/sso-adminserver/sdk/vsphere.local
2018-08-14T16:40:32.532Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@ceb4bd2: startup date [Tue Aug 14 16:40:32 UTC 2018]; root of context hierarchy
2018-08-14T16:40:32.731Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@ceb4bd2: startup date [Tue Aug 14 16:40:32 UTC 2018]; root of context hierarchy
2018-08-14T16:40:32.734Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.sso loaded in 200 millis
2018-08-14T16:40:32.847Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.122Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.158Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl ()] Client was created successfully
2018-08-14T16:40:33.170Z [CertFetcher-0 [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.281Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.302Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] SecurityAgent: System domain is 'vsphere.local', CM solution UPN is 'machine-fb11c1f8-15c7-4484-8538-3567cac661b4@vsphere.local'
2018-08-14T16:40:33.907Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:40:34.412Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:40:34.421Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:40:34.426Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:40:34.427Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:34.488Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:42:32.620Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:42:32.625Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:42:32.626Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:42:32.688Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.806Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] createSsoClientHelper: Connecting to SSO service at https://10.161.254.155/sso-adminserver/sdk/vsphere.local (admin), https://10.161.254.155/sts/STSService/vsphere.local (STS)
2018-08-14T16:44:32.811Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.897Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.906Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl ()] Client was created successfully
2018-08-14T16:44:32.908Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.923Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient ()] Client was disposed successfully
2018-08-14T16:44:32.930Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:32.932Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:33.136Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:44:33.164Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:33.169Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:33.170Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:33.236Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:45:52.966Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:45:52.979Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:45:52.982Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:45:53.065Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
If I make another failover the problem persists, and the only way I can restore the vcenter functionallity is restoring a previus snapshot made before the first failover.
What's happening?
Regards, and sorry for poor english.