vSphere client could not connect (503)

[celsoannes]

I am having the follow problem, when i try to log in using the vsphere client, I have this message:

web client:

Then I discover that /storage/log directory was full in vCenter and follow this article​ that solve my problem with space.

Now I have space but I am still having problem, I already try:

• reboot
• service-control --stop --all
• service-control --start --all

This is the output of service-control --status --all:

service-control --status --all

vcenter55:~ # service-control --status --all INFO:root:Service: vmafdd, Action: status Service: vmafdd, Action: status 2018-03-20T11:54:33.378Z   Running command: ['/sbin/service', u'vmafdd', 'status'] 2018-03-20T11:54:33.410Z   Done running command INFO:root:Service: vmware-rhttpproxy, Action: status Service: vmware-rhttpproxy, Action: status 2018-03-20T11:54:33.411Z   Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status'] 2018-03-20T11:54:33.531Z   Done running command INFO:root:Service: vmdird, Action: status Service: vmdird, Action: status 2018-03-20T11:54:33.532Z   Running command: ['/sbin/service', u'vmdird', 'status'] 2018-03-20T11:54:33.562Z   Done running command INFO:root:Service: vmcad, Action: status Service: vmcad, Action: status 2018-03-20T11:54:33.563Z   Running command: ['/sbin/service', u'vmcad', 'status'] 2018-03-20T11:54:33.590Z   Done running command INFO:root:Service: vmware-sts-idmd, Action: status Service: vmware-sts-idmd, Action: status 2018-03-20T11:54:33.591Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status'] 2018-03-20T11:54:33.612Z   Done running command INFO:root:Service: vmware-stsd, Action: status Service: vmware-stsd, Action: status 2018-03-20T11:54:33.613Z   Running command: ['/sbin/service', u'vmware-stsd', 'status'] 2018-03-20T11:54:33.668Z   Done running command INFO:root:Service: vmware-cm, Action: status Service: vmware-cm, Action: status 2018-03-20T11:54:33.669Z   Running command: ['/sbin/service', u'vmware-cm', 'status'] 2018-03-20T11:54:33.886Z   Done running command INFO:root:Service: vmware-cis-license, Action: status Service: vmware-cis-license, Action: status 2018-03-20T11:54:33.887Z   Running command: ['/sbin/service', u'vmware-cis-license', 'status'] 2018-03-20T11:54:34.100Z   Done running command INFO:root:Service: vmware-sca, Action: status Service: vmware-sca, Action: status 2018-03-20T11:54:34.101Z   Running command: ['/sbin/service', u'vmware-sca', 'status'] 2018-03-20T11:54:34.302Z   Done running command INFO:root:Service: applmgmt, Action: status Service: applmgmt, Action: status 2018-03-20T11:54:34.303Z   Running command: ['/sbin/service', u'applmgmt', 'status'] 2018-03-20T11:54:34.395Z   Done running command INFO:root:Service: vmware-netdumper, Action: status Service: vmware-netdumper, Action: status 2018-03-20T11:54:34.396Z   Running command: ['/sbin/service', u'vmware-netdumper', 'status'] 2018-03-20T11:54:34.485Z   Done running command INFO:root:Service: vmware-syslog, Action: status Service: vmware-syslog, Action: status 2018-03-20T11:54:34.486Z   Running command: ['/sbin/service', u'vmware-syslog', 'status'] 2018-03-20T11:54:34.584Z   Done running command INFO:root:Service: vmware-syslog-health, Action: status Service: vmware-syslog-health, Action: status 2018-03-20T11:54:34.585Z   Running command: ['/sbin/service', u'vmware-syslog-health', 'status'] 2018-03-20T11:54:34.805Z   Done running command INFO:root:Service: vmware-vapi-endpoint, Action: status Service: vmware-vapi-endpoint, Action: status 2018-03-20T11:54:34.806Z   Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status'] 2018-03-20T11:54:34.911Z   Done running command INFO:root:Service: vmware-vpostgres, Action: status Service: vmware-vpostgres, Action: status 2018-03-20T11:54:34.913Z   Running command: ['/sbin/service', u'vmware-vpostgres', 'status'] 2018-03-20T11:54:35.076Z   Done running command INFO:root:Service: vmware-invsvc, Action: status Service: vmware-invsvc, Action: status 2018-03-20T11:54:35.077Z   Running command: ['/sbin/service', u'vmware-invsvc', 'status'] 2018-03-20T11:54:35.264Z   Done running command INFO:root:Service: vmware-mbcs, Action: status Service: vmware-mbcs, Action: status 2018-03-20T11:54:35.265Z   Running command: ['/sbin/service', u'vmware-mbcs', 'status'] 2018-03-20T11:54:35.457Z   Done running command INFO:root:Service: vmware-vpxd, Action: status Service: vmware-vpxd, Action: status 2018-03-20T11:54:35.458Z   Running command: ['/sbin/service', u'vmware-vpxd', 'status'] 2018-03-20T11:54:36.134Z   Done running command INFO:root:Service: vmware-eam, Action: status Service: vmware-eam, Action: status 2018-03-20T11:54:36.135Z   Running command: ['/sbin/service', u'vmware-eam', 'status'] 2018-03-20T11:54:36.226Z   Done running command INFO:root:Service: vmware-rbd-watchdog, Action: status Service: vmware-rbd-watchdog, Action: status 2018-03-20T11:54:36.227Z   Running command: ['/sbin/service', u'vmware-rbd-watchdog', 'status'] 2018-03-20T11:54:36.315Z   Done running command INFO:root:Service: vmware-sps, Action: status Service: vmware-sps, Action: status 2018-03-20T11:54:36.316Z   Running command: ['/sbin/service', u'vmware-sps', 'status'] 2018-03-20T11:54:36.540Z   Done running command INFO:root:Service: vmware-vdcs, Action: status Service: vmware-vdcs, Action: status 2018-03-20T11:54:36.541Z   Running command: ['/sbin/service', u'vmware-vdcs', 'status'] 2018-03-20T11:54:36.722Z   Done running command INFO:root:Service: vmware-vpx-workflow, Action: status Service: vmware-vpx-workflow, Action: status 2018-03-20T11:54:36.723Z   Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status'] 2018-03-20T11:54:36.824Z   Done running command INFO:root:Service: vmware-vsm, Action: status Service: vmware-vsm, Action: status 2018-03-20T11:54:36.825Z   Running command: ['/sbin/service', u'vmware-vsm', 'status'] 2018-03-20T11:54:36.938Z   Done running command INFO:root:Service: vsphere-client, Action: status Service: vsphere-client, Action: status 2018-03-20T11:54:36.939Z   Running command: ['/sbin/service', u'vsphere-client', 'status'] 2018-03-20T11:54:37.144Z   Done running command INFO:root:Service: vmware-perfcharts, Action: status Service: vmware-perfcharts, Action: status 2018-03-20T11:54:37.145Z   Running command: ['/sbin/service', u'vmware-perfcharts', 'status'] 2018-03-20T11:54:37.264Z   Done running command INFO:root:Service: vmware-vws, Action: status Service: vmware-vws, Action: status 2018-03-20T11:54:37.265Z   Running command: ['/sbin/service', u'vmware-vws', 'status'] 2018-03-20T11:54:37.429Z   Done running command INFO:root:Running: applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vpostgres (VMware Postgres) Running: applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vpostgres (VMware Postgres) INFO:root:Stopped: vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client () Stopped: vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()

I am running a vCenter Server Applience 6.0.0.

Can someone help me please.

[pwilk]

How much RAM did you allocate to vCSA? This error shows up most of the time if the vCSA is still starting up. If nothing else works, leave it running for  about 30min and check again if the issue persists.

Please check RAM allocation best practices described in this KB article: vCenter Server Appliance Hardware Requirements

How big is your vSphere environment (VMs & hosts)?

[aadi369]

Go for latest bulid (know issue), so you might want to consider patching the host.

I think you're only fix is a host reboot.

[celsoannes]

I have 8Gb, it was running without a problem, I do not belive that the problem is memory, I already reboot and waited for it more than 12 hous.

have 2 hosts and about 22/25 VM each host.

[celsoannes]

I am considering go for vSphere 6.5 but I will try to fix it first.

I will try a host reboot and let you know if it work.

[pwilk]

Ok, in this case - before upgrading - you can try to run the following:

Search for duplicate keys in postgres database (run this from vCSA shell):

cd /var/log

grep -r already *  | grep device_key

An example result:

Key (id, device_key)=(66,9000) already exists

Connect to the embedded postgres database:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Remove the duplicate key and reboot the appliance:

DELETE FROM vc.vpx_vm_virtual_device where id='66' and device_key='9000';

Remember to change the values accordingly.

[Finikiez]

You have stopped vmware-vpxd (VMware vCenter Server) service and web client as well vsphere-client

First of all try to start vmware-vpxd manually using service-sontrol --start command.

If it doesn't start check the logs /var/log/vmware/vmware-vpxd/vpxd.log

The error should be in the bottom of log file. Or just attach it here to check.

[ashwin_prakash]

You might want to check the logs from /var/log/vmware/eam/wrapper.log

As it looks like the EAM service is not starting.

Also check eam.properties under /etc/vmware-eam/

Make sure there is content in eam.properties file.

[celsoannes]

I rebooted the host, but it did not work.

[celsoannes]

I tried stop and start vmware-vpxd manually it start the service but my problem still the same (503), I dont have the folder /var/log/vmware/vmware-vpxd I don't know if it is created if you don't get any errors.

[Finikiez]

Sorry, I made a typo.

Proper path is /var/log/vmware/vmware-vpx/vpxd.log

If you tried to start it manually, check if it didn't stop after that with service-control command again.

As well check the logs to find why vpxd stops after restart.

[celsoannes]

I have some errors on /var/log/vmware/eam/wrapper.log, but I have no idea what it means.

EAM is not running and when I force it to run I get an error.

The /etc/vmware-eam/eam.properties is empty, how can I fix this?

service-control --start vmware-eam

vcenter55:~ # service-control --start vmware-eam INFO:root:Service: vmware-eam, Action: start Service: vmware-eam, Action: start 2018-03-21T13:18:01.727Z   Running command: ['/sbin/chkconfig', u'vmware-eam'] 2018-03-21T13:18:02.013Z   Done running command 2018-03-21T13:18:02.014Z   Running command: ['/sbin/service', u'vmware-eam', 'status'] 2018-03-21T13:18:02.103Z   Done running command 2018-03-21T13:18:02.103Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-eam', 'on'] 2018-03-21T13:18:02.183Z   Done running command 2018-03-21T13:18:02.183Z   Running command: ['/sbin/service', u'vmware-eam', 'start'] 2018-03-21T13:18:07.009Z   Done running command 2018-03-21T13:18:07.009Z   Invoked command: ['/sbin/service', u'vmware-eam', 'start'] 2018-03-21T13:18:07.009Z   RC = 1 Stdout = Starting VMware ESX Agent Manager... Waiting for VMware ESX Agent Manager....... WARNING: VMware ESX Agent Manager may have failed to start. Last login: Wed Mar 21 13:17:12 UTC 2018 Stderr = 2018-03-21T13:18:07.009Z   {     "resolution": null,     "detail": [         {             "args": [                 "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "             ],             "id": "install.ciscommon.command.errinvoke",             "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",             "translatable": "An error occurred while invoking external command : '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } ERROR:root:Unable to start service vmware-eam, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-eam"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-eam'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } Unable to start service vmware-eam, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-eam"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-eam'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null }

tail -f /var/log/vmware/eam/eam.log
2018-03-21T13:18:03.596Z |  INFO | WrapperSimpleAppMain | Main.java | 47 | Initializing Spring from 'standalone.xml' and 'eam.properties' 2018-03-21T13:18:03.648Z |  INFO | WrapperSimpleAppMain | AbstractApplicationContext.java | 412 | Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@16f5b392: display name [org.springframework.context.support.ClassPathXmlApplicationContext@16f5b392]; startup date [Wed Mar 21 13:18:03 UTC 2018]; root of context hierarchy 2018-03-21T13:18:03.755Z |  INFO | WrapperSimpleAppMain | XmlBeanDefinitionReader.java | 323 | Loading XML bean definitions from class path resource [standalone.xml] 2018-03-21T13:18:03.925Z |  INFO | WrapperSimpleAppMain | XmlBeanDefinitionReader.java | 323 | Loading XML bean definitions from class path resource [eam-server-beans.xml] 2018-03-21T13:18:04.058Z |  INFO | WrapperSimpleAppMain | AbstractApplicationContext.java | 427 | Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@16f5b392]: org.springframework.beans.factory.support.DefaultListableBeanFactory@43f19bbe 2018-03-21T13:18:04.226Z |  INFO | WrapperSimpleAppMain | PropertiesLoaderSupport.java | 178 | Loading properties file from class path resource [eam.properties] 2018-03-21T13:18:04.232Z |  INFO | WrapperSimpleAppMain | DefaultSingletonBeanRegistry.java | 396 | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@43f19bbe: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,commonMapStore,vibServices,esxAgentMgr,vcTrustStore,eamKeyStore,eamKeyAccess,vimAndVumConnectionExecutor,vpxdConnection,vumConnection,vcOptions,serverAddr,serverAddrDepr,serverUrl,serverUrlDepr,vcListener,l10n,httpNfcLeaseTimeoutHandler,nameManager,vcDatastoreManager,vcEventManager,vcKeyValueStore,messageSource,vibServer,vibServerDepr,vibPackageDepot,vibPackageDepotDepr,secureHostsStartVersion,vibPackageDepots,vcInventory,springContext,scanForUnknownAgentVms,cmClient,stsCertManager,scheduler,heathStatusTaskScheduler,vmodlContext,versionFinder,clientAuthConnectionExecutor,clientAuthConnection,clientAuthenticator,localizationFilter,firstMobStep,mobConnectionExecutor,mobConnection,mobAuthenticator,localClient,localCorrelator,localFilterStartStep,localCompletionStep,sessionManager,dispatcher,adapterServer,eamVersion,bodyHandler,eamVersion2,bodyHandlerVersion2,eamVersion2_5,bodyHandlerVersion2_5,eamVersion3,bodyHandlerVersion3,eamVersion6,bodyHandlerVersion6,internalEamVersion1,internalBodyHandlerVersion1,internalEamVersion2,internalBodyHandlerVersion2,internalEamVersion2_5,internalBodyHandlerVersion2_5,internalEamVersion3,internalBodyHandlerVersion3,internalEamVersion6,internalBodyHandlerVersion6,httpResponseCorrelator,firstStep,validationStartStep,completionStep,filterStartStep,invocationStep,continuerStep,rejectedHandler,executor,httpServerEndpoint,listener,mob,jspServlet,fileServlet,health,managementUi,cmHealthServlet,cmResourceBundleServlet,vmomiService,mobService,jspService,filesService,healthService,managementService,vibService,vibdService,cmHealthService,cmResourceBundleService,httpServer]; root of factory hierarchy

[ashwin_prakash]

1. Browse to /etc/vmware-eam/.

2. Clone the eam.properties file to have a backup from the original file

cp eam.properties eam.properties.old

3. Determine the host ID,

cat /etc/vmware/install-defaults/sca.hostid:

The output should be a string like : 0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec

Check the vCneter FQDN: host -f

4. Edit the eam.properties file:

vi eam.properties

Note: If there is no executable permission run the command - "chmod 777 eam.properties"

5. Enter insert mode by pressing i, paste the below inside the file considering modifying the highlighted strings to fit the specifications of the environment:

vc.proxy.host=localhost

vc.proxy.port=80

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=15005

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=443

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=80

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Enable/disable VUM integration

vum.integration=true

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

# The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat

eam.web.root=/usr/lib/vmware-eam/web

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=vpxd-extension

eam.keystore.storename=vpxd-extension

# CM configuration

#

cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec #<====== same string from step 4

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService #<====== Keep this unchanged

vc.tunnelSdkUri=https://<vCenter_FQDN>:8089/sdk/vimService #<====== Change this to your vCenter FQDN/IP

drs.demandCapacityRatio=100

6. Save and exit the file by pressing <ESC> and enter :wq

Make sure you remove "<====== Keep this unchanged" "<====== Change this to your vCenter FQDN/IP" "<====== same string from step 4" once the required changes has been made.

7. Run this command to change the permission of the file:

chmod 0644 eam.properties

8. Start the EAM service:

Service vmware-eam start

[celsoannes]

My eam.properties was empty, so I copy and paste the code that you provide but:

The command host -f answer me with an error:

I am not sure about what to do with this line:

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService #<====== Keep this unchanged

I am not sure if I just remove the word #<====== Keep this unchanged and keep only:

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService

it wil work? is it right?

As I can't get an answer from host -f what do I do with it?

vc.tunnelSdkUri=https://<vCenter_FQDN>:8089/sdk/vimService #<====== Change this to your vCenter FQDN/IP

[ashwin_prakash]

You could ignore the command for host -h (This command is only to know the FQDN of your vCenter server)

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService #

This would work, in your reply you had removed the #

vc.tunnelSdkUri=https://<vCenter_FQDN>:8089/sdk/vimService #<====== Change this to your vCenter FQDN/IP

You should even get the fqdn by typing hostname

If your vCenter server FQDN is https://vmware.vsphere.local

You would have entry updated as:

vc.tunnelSdkUri=https://vmware.vsphere.local:8089/sdk/vimService #

[celsoannes]

vmware-eam looks good now...

but I am still with the same problem from the beginning:

[ashwin_prakash]

Run the command to start all the services.

service-control --start --all

[celsoannes]

I already try to reboot vcenter, them I tried the command that you suggest, below is the output of the command:

I still with problems to connect.

vcenter55:~ # service-control --start --all INFO:root:Service: vmafdd, Action: start Service: vmafdd, Action: start 2018-03-21T19:06:25.622Z   Running command: ['/sbin/chkconfig', u'vmafdd'] 2018-03-21T19:06:25.687Z   Done running command 2018-03-21T19:06:25.688Z   Running command: ['/sbin/service', u'vmafdd', 'status'] 2018-03-21T19:06:25.716Z   Done running command INFO:root:Service: vmware-rhttpproxy, Action: start Service: vmware-rhttpproxy, Action: start 2018-03-21T19:06:25.922Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy'] 2018-03-21T19:06:25.984Z   Done running command 2018-03-21T19:06:25.984Z   Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status'] 2018-03-21T19:06:26.089Z   Done running command INFO:root:Service: vmdird, Action: start Service: vmdird, Action: start 2018-03-21T19:06:26.091Z   Running command: ['/sbin/chkconfig', u'vmdird'] 2018-03-21T19:06:26.172Z   Done running command 2018-03-21T19:06:26.172Z   Running command: ['/sbin/service', u'vmdird', 'status'] 2018-03-21T19:06:26.201Z   Done running command INFO:root:Service: vmcad, Action: start Service: vmcad, Action: start 2018-03-21T19:06:26.203Z   Running command: ['/sbin/chkconfig', u'vmcad'] 2018-03-21T19:06:26.283Z   Done running command 2018-03-21T19:06:26.283Z   Running command: ['/sbin/service', u'vmcad', 'status'] 2018-03-21T19:06:26.319Z   Done running command INFO:root:Service: vmware-sts-idmd, Action: start Service: vmware-sts-idmd, Action: start 2018-03-21T19:06:26.321Z   Running command: ['/sbin/chkconfig', u'vmware-sts-idmd'] 2018-03-21T19:06:26.405Z   Done running command 2018-03-21T19:06:26.405Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status'] 2018-03-21T19:06:26.427Z   Done running command INFO:root:Service: vmware-stsd, Action: start Service: vmware-stsd, Action: start 2018-03-21T19:06:26.428Z   Running command: ['/sbin/chkconfig', u'vmware-stsd'] 2018-03-21T19:06:26.506Z   Done running command 2018-03-21T19:06:26.506Z   Running command: ['/sbin/service', u'vmware-stsd', 'status'] 2018-03-21T19:06:26.592Z   Done running command INFO:root:Service: vmware-cm, Action: start Service: vmware-cm, Action: start 2018-03-21T19:06:26.594Z   Running command: ['/sbin/chkconfig', u'vmware-cm'] 2018-03-21T19:06:26.674Z   Done running command 2018-03-21T19:06:26.675Z   Running command: ['/sbin/service', u'vmware-cm', 'status'] 2018-03-21T19:06:26.896Z   Done running command INFO:root:Service: vmware-cis-license, Action: start Service: vmware-cis-license, Action: start 2018-03-21T19:06:26.897Z   Running command: ['/sbin/chkconfig', u'vmware-cis-license'] 2018-03-21T19:06:26.962Z   Done running command 2018-03-21T19:06:26.962Z   Running command: ['/sbin/service', u'vmware-cis-license', 'status'] 2018-03-21T19:06:27.446Z   Done running command INFO:root:Service: vmware-sca, Action: start Service: vmware-sca, Action: start 2018-03-21T19:06:28.617Z   Running command: ['/sbin/chkconfig', u'vmware-sca'] 2018-03-21T19:06:28.779Z   Done running command 2018-03-21T19:06:28.779Z   Running command: ['/sbin/service', u'vmware-sca', 'status'] 2018-03-21T19:06:28.982Z   Done running command INFO:root:Service: applmgmt, Action: start Service: applmgmt, Action: start 2018-03-21T19:06:28.983Z   Running command: ['/sbin/chkconfig', u'applmgmt'] 2018-03-21T19:06:29.060Z   Done running command 2018-03-21T19:06:29.060Z   Running command: ['/sbin/service', u'applmgmt', 'status'] 2018-03-21T19:06:29.145Z   Done running command INFO:root:Service: vmware-netdumper, Action: start Service: vmware-netdumper, Action: start 2018-03-21T19:06:29.146Z   Running command: ['/sbin/chkconfig', u'vmware-netdumper'] 2018-03-21T19:06:29.227Z   Done running command 'vmware-netdumper' startMode is Manual, skipping to start: INFO:root:'vmware-netdumper' startMode is Manual, skipping to start: INFO:root:Service: vmware-syslog, Action: start Service: vmware-syslog, Action: start 2018-03-21T19:06:29.228Z   Running command: ['/sbin/chkconfig', u'vmware-syslog'] 2018-03-21T19:06:29.307Z   Done running command 2018-03-21T19:06:29.307Z   Running command: ['/sbin/service', u'vmware-syslog', 'status'] 2018-03-21T19:06:29.413Z   Done running command INFO:root:Service: vmware-syslog-health, Action: start Service: vmware-syslog-health, Action: start 2018-03-21T19:06:29.414Z   Running command: ['/sbin/chkconfig', u'vmware-syslog-health'] 2018-03-21T19:06:29.493Z   Done running command 2018-03-21T19:06:29.493Z   Running command: ['/sbin/service', u'vmware-syslog-health', 'status'] 2018-03-21T19:06:29.710Z   Done running command INFO:root:Service: vmware-vapi-endpoint, Action: start Service: vmware-vapi-endpoint, Action: start 2018-03-21T19:06:29.712Z   Running command: ['/sbin/chkconfig', u'vmware-vapi-endpoint'] 2018-03-21T19:06:29.790Z   Done running command 2018-03-21T19:06:29.790Z   Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status'] 2018-03-21T19:06:30.366Z   Done running command INFO:root:Service: vmware-vpostgres, Action: start Service: vmware-vpostgres, Action: start 2018-03-21T19:06:30.367Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres'] 2018-03-21T19:06:30.482Z   Done running command 2018-03-21T19:06:30.482Z   Running command: ['/sbin/service', u'vmware-vpostgres', 'status'] 2018-03-21T19:06:31.273Z   Done running command INFO:root:Service: vmware-invsvc, Action: start Service: vmware-invsvc, Action: start 2018-03-21T19:06:31.275Z   Running command: ['/sbin/chkconfig', u'vmware-invsvc'] 2018-03-21T19:06:31.346Z   Done running command 2018-03-21T19:06:31.347Z   Running command: ['/sbin/service', u'vmware-invsvc', 'status'] 2018-03-21T19:06:31.548Z   Done running command 2018-03-21T19:06:31.549Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-invsvc', 'on'] 2018-03-21T19:06:31.626Z   Done running command 2018-03-21T19:06:31.626Z   Running command: ['/sbin/service', u'vmware-invsvc', 'start'] 2018-03-21T19:06:49.343Z   Done running command 2018-03-21T19:06:49.343Z   Invoked command: ['/sbin/service', u'vmware-invsvc', 'start'] 2018-03-21T19:06:49.343Z   RC = 1 Stdout = Starting VMware Inventory Service... Waiting for VMware Inventory Service................... WARNING: VMware Inventory Service may have failed to start. Stderr = 2018-03-21T19:06:49.343Z   {     "resolution": null,     "detail": [         {             "args": [                 "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "             ],             "id": "install.ciscommon.command.errinvoke",             "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",             "translatable": "An error occurred while invoking external command : '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } ERROR:root:Unable to start service vmware-invsvc, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-invsvc"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-invsvc'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } Unable to start service vmware-invsvc, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-invsvc"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-invsvc'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null }

[ashwin_prakash · Accepted solution]

Check on the logs refereed to on the below KB. If you are receiving the same error. Follow the KB to reset the password.

VMware Knowledge Base

Keep generating vdcadmintool password until the password does not contain any of these characters "space" @, \ or ".

[celsoannes]

Unfortunately it did not work as you can see:

Symptoms:

Resolution:

Step 5

Step 11

Step 12

I tried stop and start all and it give me the same error:

vcenter55:/bin # service-control --start --all INFO:root:Service: vmafdd, Action: start Service: vmafdd, Action: start 2018-03-22T13:56:00.621Z   Running command: ['/sbin/chkconfig', u'vmafdd'] 2018-03-22T13:56:00.700Z   Done running command 2018-03-22T13:56:00.700Z   Running command: ['/sbin/service', u'vmafdd', 'status'] 2018-03-22T13:56:00.728Z   Done running command 2018-03-22T13:56:00.728Z   Running command: ['/sbin/chkconfig', '--force', u'vmafdd', 'on'] 2018-03-22T13:56:00.809Z   Done running command 2018-03-22T13:56:00.809Z   Running command: ['/sbin/service', u'vmafdd', 'start'] 2018-03-22T13:56:00.893Z   Done running command 2018-03-22T13:56:00.893Z   Successfully started service vmafdd INFO:root:Service: vmware-rhttpproxy, Action: start Service: vmware-rhttpproxy, Action: start 2018-03-22T13:56:00.894Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy'] 2018-03-22T13:56:00.975Z   Done running command 2018-03-22T13:56:00.976Z   Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status'] 2018-03-22T13:56:01.024Z   Done running command 2018-03-22T13:56:01.024Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-rhttpproxy', 'on'] 2018-03-22T13:56:01.100Z   Done running command 2018-03-22T13:56:01.100Z   Running command: ['/sbin/service', u'vmware-rhttpproxy', 'start'] 2018-03-22T13:56:01.139Z   Done running command 2018-03-22T13:56:01.139Z   Successfully started service vmware-rhttpproxy INFO:root:Service: vmdird, Action: start Service: vmdird, Action: start 2018-03-22T13:56:01.140Z   Running command: ['/sbin/chkconfig', u'vmdird'] 2018-03-22T13:56:01.216Z   Done running command 2018-03-22T13:56:01.217Z   Running command: ['/sbin/service', u'vmdird', 'status'] 2018-03-22T13:56:01.243Z   Done running command 2018-03-22T13:56:01.244Z   Running command: ['/sbin/chkconfig', '--force', u'vmdird', 'on'] 2018-03-22T13:56:01.320Z   Done running command 2018-03-22T13:56:01.320Z   Running command: ['/sbin/service', u'vmdird', 'start'] 2018-03-22T13:56:02.603Z   Done running command 2018-03-22T13:56:02.603Z   Successfully started service vmdird INFO:root:Service: vmcad, Action: start Service: vmcad, Action: start 2018-03-22T13:56:02.604Z   Running command: ['/sbin/chkconfig', u'vmcad'] 2018-03-22T13:56:02.680Z   Done running command 2018-03-22T13:56:02.680Z   Running command: ['/sbin/service', u'vmcad', 'status'] 2018-03-22T13:56:02.707Z   Done running command 2018-03-22T13:56:02.707Z   Running command: ['/sbin/chkconfig', '--force', u'vmcad', 'on'] 2018-03-22T13:56:02.785Z   Done running command 2018-03-22T13:56:02.785Z   Running command: ['/sbin/service', u'vmcad', 'start'] 2018-03-22T13:56:03.863Z   Done running command 2018-03-22T13:56:03.863Z   Successfully started service vmcad INFO:root:Service: vmware-sts-idmd, Action: start Service: vmware-sts-idmd, Action: start 2018-03-22T13:56:03.864Z   Running command: ['/sbin/chkconfig', u'vmware-sts-idmd'] 2018-03-22T13:56:03.941Z   Done running command 2018-03-22T13:56:03.941Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status'] 2018-03-22T13:56:03.961Z   Done running command 2018-03-22T13:56:03.962Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-sts-idmd', 'on'] 2018-03-22T13:56:04.039Z   Done running command 2018-03-22T13:56:04.039Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'start'] 2018-03-22T13:56:04.175Z   Done running command 2018-03-22T13:56:04.175Z   Successfully started service vmware-sts-idmd INFO:root:Service: vmware-stsd, Action: start Service: vmware-stsd, Action: start 2018-03-22T13:56:04.176Z   Running command: ['/sbin/chkconfig', u'vmware-stsd'] 2018-03-22T13:56:04.254Z   Done running command 2018-03-22T13:56:04.255Z   Running command: ['/sbin/service', u'vmware-stsd', 'status'] 2018-03-22T13:56:04.293Z   Done running command 2018-03-22T13:56:04.293Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-stsd', 'on'] 2018-03-22T13:56:04.373Z   Done running command 2018-03-22T13:56:04.374Z   Running command: ['/sbin/service', u'vmware-stsd', 'start'] 2018-03-22T13:56:08.533Z   Done running command 2018-03-22T13:56:08.533Z   Successfully started service vmware-stsd INFO:root:Service: vmware-cm, Action: start Service: vmware-cm, Action: start 2018-03-22T13:56:08.534Z   Running command: ['/sbin/chkconfig', u'vmware-cm'] 2018-03-22T13:56:08.617Z   Done running command 2018-03-22T13:56:08.617Z   Running command: ['/sbin/service', u'vmware-cm', 'status'] 2018-03-22T13:56:08.795Z   Done running command 2018-03-22T13:56:08.795Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-cm', 'on'] 2018-03-22T13:56:08.875Z   Done running command 2018-03-22T13:56:08.876Z   Running command: ['/sbin/service', u'vmware-cm', 'start'] 2018-03-22T13:58:10.939Z   Done running command 2018-03-22T13:58:10.939Z   Successfully started service vmware-cm INFO:root:Service: vmware-cis-license, Action: start Service: vmware-cis-license, Action: start 2018-03-22T13:58:10.940Z   Running command: ['/sbin/chkconfig', u'vmware-cis-license'] 2018-03-22T13:58:11.018Z   Done running command 2018-03-22T13:58:11.018Z   Running command: ['/sbin/service', u'vmware-cis-license', 'status'] 2018-03-22T13:58:11.204Z   Done running command 2018-03-22T13:58:11.204Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-cis-license', 'on'] 2018-03-22T13:58:11.282Z   Done running command 2018-03-22T13:58:11.282Z   Running command: ['/sbin/service', u'vmware-cis-license', 'start'] 2018-03-22T13:58:19.136Z   Done running command 2018-03-22T13:58:19.136Z   Successfully started service vmware-cis-license INFO:root:Service: vmware-sca, Action: start Service: vmware-sca, Action: start 2018-03-22T13:58:19.138Z   Running command: ['/sbin/chkconfig', u'vmware-sca'] 2018-03-22T13:58:19.218Z   Done running command 2018-03-22T13:58:19.218Z   Running command: ['/sbin/service', u'vmware-sca', 'status'] 2018-03-22T13:58:19.394Z   Done running command 2018-03-22T13:58:19.395Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-sca', 'on'] 2018-03-22T13:58:19.473Z   Done running command 2018-03-22T13:58:19.474Z   Running command: ['/sbin/service', u'vmware-sca', 'start'] 2018-03-22T14:00:21.197Z   Done running command 2018-03-22T14:00:21.197Z   Successfully started service vmware-sca INFO:root:Service: applmgmt, Action: start Service: applmgmt, Action: start 2018-03-22T14:00:21.198Z   Running command: ['/sbin/chkconfig', u'applmgmt'] 2018-03-22T14:00:21.262Z   Done running command 2018-03-22T14:00:21.262Z   Running command: ['/sbin/service', u'applmgmt', 'status'] 2018-03-22T14:00:21.343Z   Done running command 2018-03-22T14:00:21.343Z   Running command: ['/sbin/chkconfig', '--force', u'applmgmt', 'on'] 2018-03-22T14:00:21.406Z   Done running command 2018-03-22T14:00:21.406Z   Running command: ['/sbin/service', u'applmgmt', 'start'] 2018-03-22T14:00:21.801Z   Done running command 2018-03-22T14:00:21.801Z   Successfully started service applmgmt INFO:root:Service: vmware-netdumper, Action: start Service: vmware-netdumper, Action: start 2018-03-22T14:00:21.802Z   Running command: ['/sbin/chkconfig', u'vmware-netdumper'] 2018-03-22T14:00:21.867Z   Done running command 'vmware-netdumper' startMode is Manual, skipping to start: INFO:root:'vmware-netdumper' startMode is Manual, skipping to start: INFO:root:Service: vmware-syslog, Action: start Service: vmware-syslog, Action: start 2018-03-22T14:00:21.868Z   Running command: ['/sbin/chkconfig', u'vmware-syslog'] 2018-03-22T14:00:21.935Z   Done running command 2018-03-22T14:00:21.935Z   Running command: ['/sbin/service', u'vmware-syslog', 'status'] 2018-03-22T14:00:22.024Z   Done running command 2018-03-22T14:00:22.024Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-syslog', 'on'] 2018-03-22T14:00:22.094Z   Done running command 2018-03-22T14:00:22.094Z   Running command: ['/sbin/service', u'vmware-syslog', 'start'] 2018-03-22T14:00:24.721Z   Done running command 2018-03-22T14:00:24.721Z   Successfully started service vmware-syslog INFO:root:Service: vmware-syslog-health, Action: start Service: vmware-syslog-health, Action: start 2018-03-22T14:00:24.722Z   Running command: ['/sbin/chkconfig', u'vmware-syslog-health'] 2018-03-22T14:00:24.787Z   Done running command 2018-03-22T14:00:24.787Z   Running command: ['/sbin/service', u'vmware-syslog-health', 'status'] 2018-03-22T14:00:24.954Z   Done running command 2018-03-22T14:00:24.954Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-syslog-health', 'on'] 2018-03-22T14:00:25.030Z   Done running command 2018-03-22T14:00:25.031Z   Running command: ['/sbin/service', u'vmware-syslog-health', 'start'] 2018-03-22T14:00:26.771Z   Done running command 2018-03-22T14:00:26.771Z   Successfully started service vmware-syslog-health INFO:root:Service: vmware-vapi-endpoint, Action: start Service: vmware-vapi-endpoint, Action: start 2018-03-22T14:00:26.772Z   Running command: ['/sbin/chkconfig', u'vmware-vapi-endpoint'] 2018-03-22T14:00:26.836Z   Done running command 2018-03-22T14:00:26.837Z   Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status'] 2018-03-22T14:00:26.914Z   Done running command 2018-03-22T14:00:26.915Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-vapi-endpoint', 'on'] 2018-03-22T14:00:26.980Z   Done running command 2018-03-22T14:00:26.980Z   Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'start'] 2018-03-22T14:00:30.861Z   Done running command 2018-03-22T14:00:30.861Z   Successfully started service vmware-vapi-endpoint INFO:root:Service: vmware-vpostgres, Action: start Service: vmware-vpostgres, Action: start 2018-03-22T14:00:30.862Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres'] 2018-03-22T14:00:30.930Z   Done running command 2018-03-22T14:00:30.931Z   Running command: ['/sbin/service', u'vmware-vpostgres', 'status'] 2018-03-22T14:00:31.076Z   Done running command 2018-03-22T14:00:31.076Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-vpostgres', 'on'] 2018-03-22T14:00:31.143Z   Done running command 2018-03-22T14:00:31.143Z   Running command: ['/sbin/service', u'vmware-vpostgres', 'start'] 2018-03-22T14:00:32.377Z   Done running command 2018-03-22T14:00:32.377Z   Successfully started service vmware-vpostgres INFO:root:Service: vmware-invsvc, Action: start Service: vmware-invsvc, Action: start 2018-03-22T14:00:32.378Z   Running command: ['/sbin/chkconfig', u'vmware-invsvc'] 2018-03-22T14:00:32.444Z   Done running command 2018-03-22T14:00:32.445Z   Running command: ['/sbin/service', u'vmware-invsvc', 'status'] 2018-03-22T14:00:32.629Z   Done running command 2018-03-22T14:00:32.629Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-invsvc', 'on'] 2018-03-22T14:00:32.714Z   Done running command 2018-03-22T14:00:32.714Z   Running command: ['/sbin/service', u'vmware-invsvc', 'start'] 2018-03-22T14:00:50.341Z   Done running command 2018-03-22T14:00:50.341Z   Invoked command: ['/sbin/service', u'vmware-invsvc', 'start'] 2018-03-22T14:00:50.341Z   RC = 1 Stdout = Starting VMware Inventory Service... Waiting for VMware Inventory Service................... WARNING: VMware Inventory Service may have failed to start. Stderr = 2018-03-22T14:00:50.341Z   {     "resolution": null,     "detail": [         {             "args": [                 "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "             ],             "id": "install.ciscommon.command.errinvoke",             "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",             "translatable": "An error occurred while invoking external command : '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } ERROR:root:Unable to start service vmware-invsvc, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-invsvc"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-invsvc'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null } Unable to start service vmware-invsvc, Exception: {     "resolution": null,     "detail": [         {             "args": [                 "vmware-invsvc"             ],             "id": "install.ciscommon.service.failstart",             "localized": "An error occurred while starting service 'vmware-invsvc'",             "translatable": "An error occurred while starting service '%(0)s'"         }     ],     "componentKey": null,     "problemId": null }

I believe that I did everything right, is there something wrong?