Hi people,
I have a serius problem with the vCenter appliance after a fail over is done. (esxi 6.7, vsphere 6.7)
When the host server shutdowns all the VMs moves correctly to the backup host but the vcenter appliance never works again.
If I try to access the application with a internet browser I get this message:
I connect to vcenter with ssh and list all the services status:
root@photon-machine [ ~ ]# service-control --status
Stopped:
pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
I think the problems comes from here, the vmware-cm didn't start. (and maybe all of the others services need to start too).
I tried to start it manually, but never starts again, in the log there is not error or something similar:
2018-08-14T16:39:31.107Z [main [] INFO com.vmware.cis.services.cm.service.Application ()] start: Starting Component Manager
2018-08-14T16:39:31.158Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:31.186Z [main [] INFO com.vmware.cis.services.cm.service.Application ()] start: Using Spring configuration file: 'http.xml'
2018-08-14T16:39:34.081Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:36.773Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:39:38.365Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] Using VECS store [TRUSTED_ROOTS]
2018-08-14T16:39:38.366Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] Using VECS store [machine]
2018-08-14T16:39:38.448Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] SecurityAgent: Connecting to SSO service
2018-08-14T16:39:39.731Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:39:39.755Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:39:39.868Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@530712d: startup date [Tue Aug 14 16:39:39 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.438Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@530712d: startup date [Tue Aug 14 16:39:39 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.439Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl loaded in 590 millis
2018-08-14T16:39:40.439Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@68ead359: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.814Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@68ead359: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:40.814Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl.query loaded in 375 millis
2018-08-14T16:39:40.847Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@47a5b70d: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@47a5b70d: startup date [Tue Aug 14 16:39:40 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl loaded in 162 millis
2018-08-14T16:39:41.009Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@56c698e3: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.098Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@56c698e3: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.098Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.vmodl.query loaded in 89 millis
2018-08-14T16:39:41.106Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@7fe7c640: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.277Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@7fe7c640: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.277Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.cis.cm loaded in 172 millis
2018-08-14T16:39:41.278Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@549949be: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.385Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@549949be: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.385Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.cis.common loaded in 107 millis
2018-08-14T16:39:41.386Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@3f0846c6: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.568Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@3f0846c6: startup date [Tue Aug 14 16:39:41 UTC 2018]; root of context hierarchy
2018-08-14T16:39:41.570Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.lookup loaded in 184 millis
2018-08-14T16:39:43.394Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:39:45.559Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:32.462Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] createSsoAdminClient: Connecting to SSO service at https://10.161.254.155/sso-adminserver/sdk/vsphere.local
2018-08-14T16:40:32.532Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Refreshing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@ceb4bd2: startup date [Tue Aug 14 16:40:32 UTC 2018]; root of context hierarchy
2018-08-14T16:40:32.731Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext ()] Closing com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@ceb4bd2: startup date [Tue Aug 14 16:40:32 UTC 2018]; root of context hierarchy
2018-08-14T16:40:32.734Z [main [] INFO com.vmware.vim.vmomi.core.types.impl.VmodlContextImpl ()] Package com.vmware.vim.binding.sso loaded in 200 millis
2018-08-14T16:40:32.847Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.122Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.158Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl ()] Client was created successfully
2018-08-14T16:40:33.170Z [CertFetcher-0 [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.281Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:33.302Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] SecurityAgent: System domain is 'vsphere.local', CM solution UPN is 'machine-fb11c1f8-15c7-4484-8538-3567cac661b4@vsphere.local'
2018-08-14T16:40:33.907Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:40:34.412Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:40:34.421Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:40:34.426Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:40:34.427Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:40:34.488Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:42:32.620Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:42:32.625Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:42:32.626Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:42:32.688Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.806Z [main [] INFO com.vmware.cis.services.cm.service.SecurityAgent ()] createSsoClientHelper: Connecting to SSO service at https://10.161.254.155/sso-adminserver/sdk/vsphere.local (admin), https://10.161.254.155/sts/STSService/vsphere.local (STS)
2018-08-14T16:44:32.811Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.897Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.906Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl ()] Client was created successfully
2018-08-14T16:44:32.908Z [main [] INFO com.vmware.vim.sso.admin.client.common.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:32.923Z [main [] INFO com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient ()] Client was disposed successfully
2018-08-14T16:44:32.930Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:32.932Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:33.136Z [main [] INFO com.vmware.cis.services.cm.service.util.Utils ()] Reading deployment properties from conf/cm.properties
2018-08-14T16:44:33.164Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:33.169Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:44:33.170Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:44:33.236Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:45:52.966Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:45:52.979Z [main [] INFO com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.161.254.155/lookupservice/sdk
2018-08-14T16:45:52.982Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2018-08-14T16:45:53.065Z [main [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
If I make another failover the problem persists, and the only way I can restore the vcenter functionallity is restoring a previus snapshot made before the first failover.
What's happening?
Regards, and sorry for poor english.
The issue also is that you have the vpxd service not running. Try to restart all the services from vCenter with these commands:
service-control --stop --all
service-control --start --all
Then do this and show us the output:
service-control --status
The normal behavior is that all the required services should be in running state.
Hi Lalegre
root@photon-machine [ ~ ]# service-control --stop --all
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service vmware-pod...
Successfully stopped service vmware-pod
Performing stop operation on profile: ALL...
Successfully stopped service vmware-vmon
Successfully stopped profile: ALL.
Performing stop operation on service vmdnsd...
Successfully stopped service vmdnsd
Performing stop operation on service vmware-stsd...
Successfully stopped service vmware-stsd
Performing stop operation on service vmware-sts-idmd...
Successfully stopped service vmware-sts-idmd
Performing stop operation on service vmcad...
Successfully stopped service vmcad
Performing stop operation on service vmdird...
Successfully stopped service vmdird
Performing stop operation on service vmafdd...
Successfully stopped service vmafdd
Performing stop operation on service lwsmd...
Successfully stopped service lwsmd
root@photon-machine [ ~ ]# service-control --status
Stopped:
applmgmt lwsmd pschealth vmafdd vmcad vmcam vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps vsphere-client vsphere-ui
root@photon-machine [ ~ ]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start cm services. Error: Operation timed out
root@photon-machine [ ~ ]# service-control --status
Stopped:
pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
cm.log didn't log anything.
Please execute df -Th and show the output here.
Hi Ocarrasco,
Make sure that after the failover the host (and the environment) hasn't unconfigured DNS and NTP.
Also, try following this previous post about this issue:
503 Service Unavailable (Failed to connect to endpoint:
Cheers,
mulcas
Hi lalegre
It is not a free space problem
root@photon-machine [ ~ ]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs tmpfs 4.9G 40K 4.9G 1% /dev/shm
tmpfs tmpfs 4.9G 708K 4.9G 1% /run
tmpfs tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 ext4 11G 5.2G 4.9G 52% /
tmpfs tmpfs 4.9G 2.2M 4.9G 1% /tmp
/dev/mapper/log_vg-log ext4 9.8G 455M 8.8G 5% /storage/log
/dev/mapper/autodeploy_vg-autodeploy ext4 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder ext4 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/core_vg-core ext4 25G 44M 24G 1% /storage/core
/dev/sda1 ext4 120M 28M 84M 25% /boot
/dev/mapper/netdump_vg-netdump ext4 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/db_vg-db ext4 9.8G 113M 9.1G 2% /storage/db
/dev/mapper/updatemgr_vg-updatemgr ext4 99G 69M 94G 1% /storage/updatemgr
/dev/mapper/dblog_vg-dblog ext4 15G 86M 14G 1% /storage/dblog
/dev/mapper/seat_vg-seat ext4 9.8G 66M 9.2G 1% /storage/seat
/dev/mapper/archive_vg-archive ext4 50G 150M 47G 1% /storage/archive
Hi mulcas,
As far I know, my network does not have a DNS provider.
When I Installed all hosts and vcenter I used the same ip of the machine as DNS.
NTP is well configurated in all machines.
Regards,
I can tell from your logs that this vCenter wasn't setup correct. If you don't have DNS in place with forward and reverse records then you cannot correctly or successfully deploy vCenter Server. Your first task is to build proper DNS infrastructure.
Thank you daphnissov for your reply.
But this is not the first VMWare system/solution we set up without DNS (more than 20) and this is the first time we have this issue.
I will try to build a DNS but I think that is not the problem.
Anyway Thank you again.
Regards,
I can tell you that it is the problem. You must have DNS. vCenter isn't designed to work without it as you have now discovered. I say again, you *must* have valid, working DNS infrastructure if you wish to use vSphere. If you can't/won't provide this basic infrastructure, do not plan on using vSphere
Hi Ocarrasco,
daphnissov is right, DNS is a must. However, first (just in case) make sure all your devices in the infrastructure are synced with the right time.
Cheers,
mulcas