Chiligee
Contributor
Contributor

503 Service Unavailable when accessing Vcenter.

Hello, and thank you for taking time to read my issue.

I have a very small vmware setup with 1 host  and a 4 VM's running on this.

Vcenter is an appliance running on the host.

ESXI Vsphere is version 6.0.0

It has been over a year since I had to log on to vcenter and i get the below error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7fe6e006a0c0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

I can log on to https://vcenterIP:5480/login.html . I did have a few issues prior where log files were full but I purged this following other KB. I did a password reset also via CLI over SSH to be able to log on via port 5480.

Once I log on I see:

Type:vCenter Server with an embedded Platform Services Controller

Product:VMware vCenter Server Appliance

Version:6.0.0.30800

Under Administration, I've set as Root password expires: No.

Port5480View.jpg

I'm not very hands ON regarding VMWARE and tried a few KB without success.

i'm hoping I can get some better direction to solve this issue.

thank you.

0 Kudos
14 Replies
Vijay2027
Expert
Expert

Get into bash shell and run the below command:

service-control --status --all 

and

df -Th

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

this is what I get on service-control --status --all :

VMware vCenter Server Appliance 6.0.0.30800

vcenter:~ # service-control --status --all

INFO:root:Service: vmafdd, Action: status

Service: vmafdd, Action: status

2019-04-03T13:55:59.176Z   Running command: ['/sbin/service', u'vmafdd', 'status                                                     ']

2019-04-03T13:55:59.260Z   Done running command

INFO:root:Service: vmware-rhttpproxy, Action: status

Service: vmware-rhttpproxy, Action: status

2019-04-03T13:55:59.262Z   Running command: ['/sbin/service', u'vmware-rhttpprox                                                     y', 'status']

2019-04-03T13:55:59.330Z   Done running command

INFO:root:Service: vmdird, Action: status

Service: vmdird, Action: status

2019-04-03T13:55:59.332Z   Running command: ['/sbin/service', u'vmdird', 'status                                                     ']

2019-04-03T13:55:59.362Z   Done running command

INFO:root:Service: vmcad, Action: status

Service: vmcad, Action: status

2019-04-03T13:55:59.364Z   Running command: ['/sbin/service', u'vmcad', 'status'                                                     ]

2019-04-03T13:55:59.386Z   Done running command

INFO:root:Service: vmware-sts-idmd, Action: status

Service: vmware-sts-idmd, Action: status

2019-04-03T13:55:59.387Z   Running command: ['/sbin/service', u'vmware-sts-idmd'                                                     , 'status']

2019-04-03T13:55:59.418Z   Done running command

INFO:root:Service: vmware-stsd, Action: status

Service: vmware-stsd, Action: status

2019-04-03T13:55:59.420Z   Running command: ['/sbin/service', u'vmware-stsd', 's                                                     tatus']

2019-04-03T13:55:59.487Z   Done running command

INFO:root:Service: vmware-cm, Action: status

Service: vmware-cm, Action: status

2019-04-03T13:55:59.488Z   Running command: ['/sbin/service', u'vmware-cm', 'sta                                                     tus']

2019-04-03T13:55:59.703Z   Done running command

INFO:root:Service: vmware-cis-license, Action: status

Service: vmware-cis-license, Action: status

2019-04-03T13:55:59.704Z   Running command: ['/sbin/service', u'vmware-cis-licen                                                     se', 'status']

2019-04-03T13:55:59.919Z   Done running command

INFO:root:Service: vmware-psc-client, Action: status

Service: vmware-psc-client, Action: status

2019-04-03T13:55:59.921Z   Running command: ['/sbin/service', u'vmware-psc-clien                                                     t', 'status']

2019-04-03T13:56:00.007Z   Done running command

INFO:root:Service: vmware-sca, Action: status

Service: vmware-sca, Action: status

2019-04-03T13:56:00.008Z   Running command: ['/sbin/service', u'vmware-sca', 'st                                                     atus']

2019-04-03T13:56:00.191Z   Done running command

INFO:root:Service: applmgmt, Action: status

Service: applmgmt, Action: status

2019-04-03T13:56:00.193Z   Running command: ['/sbin/service', u'applmgmt', 'stat                                                     us']

2019-04-03T13:56:00.302Z   Done running command

INFO:root:Service: vmware-netdumper, Action: status

Service: vmware-netdumper, Action: status

2019-04-03T13:56:00.302Z   Running command: ['/sbin/service', u'vmware-netdumper                                                     ', 'status']

2019-04-03T13:56:00.376Z   Done running command

INFO:root:Service: vmware-syslog, Action: status

Service: vmware-syslog, Action: status

2019-04-03T13:56:00.377Z   Running command: ['/sbin/service', u'vmware-syslog',                                                      'status']

2019-04-03T13:56:00.483Z   Done running command

INFO:root:Service: vmware-syslog-health, Action: status

Service: vmware-syslog-health, Action: status

2019-04-03T13:56:00.484Z   Running command: ['/sbin/service', u'vmware-syslog-he                                                     alth', 'status']

2019-04-03T13:56:00.692Z   Done running command

INFO:root:Service: vmware-vapi-endpoint, Action: status

Service: vmware-vapi-endpoint, Action: status

2019-04-03T13:56:00.693Z   Running command: ['/sbin/service', u'vmware-vapi-endp                                                     oint', 'status']

2019-04-03T13:56:00.845Z   Done running command

INFO:root:Service: vmware-vpostgres, Action: status

Service: vmware-vpostgres, Action: status

2019-04-03T13:56:00.846Z   Running command: ['/sbin/service', u'vmware-vpostgres                                                     ', 'status']

2019-04-03T13:56:01.079Z   Done running command

INFO:root:Service: vmware-invsvc, Action: status

Service: vmware-invsvc, Action: status

2019-04-03T13:56:01.080Z   Running command: ['/sbin/service', u'vmware-invsvc',                                                      'status']

2019-04-03T13:56:01.316Z   Done running command

INFO:root:Service: vmware-mbcs, Action: status

Service: vmware-mbcs, Action: status

2019-04-03T13:56:01.317Z   Running command: ['/sbin/service', u'vmware-mbcs', 's                                                     tatus']

2019-04-03T13:56:01.500Z   Done running command

INFO:root:Service: vmware-vpxd, Action: status

Service: vmware-vpxd, Action: status

2019-04-03T13:56:01.501Z   Running command: ['/sbin/service', u'vmware-vpxd', 's                                                     tatus']

2019-04-03T13:56:01.915Z   Done running command

INFO:root:Service: vmware-eam, Action: status

Service: vmware-eam, Action: status

2019-04-03T13:56:01.916Z   Running command: ['/sbin/service', u'vmware-eam', 'st                                                     atus']

2019-04-03T13:56:02.040Z   Done running command

INFO:root:Service: vmware-rbd-watchdog, Action: status

Service: vmware-rbd-watchdog, Action: status

2019-04-03T13:56:02.041Z   Running command: ['/sbin/service', u'vmware-rbd-watch                                                     dog', 'status']

2019-04-03T13:56:02.117Z   Done running command

INFO:root:Service: vmware-sps, Action: status

Service: vmware-sps, Action: status

2019-04-03T13:56:02.118Z   Running command: ['/sbin/service', u'vmware-sps', 'st                                                     atus']

2019-04-03T13:56:02.413Z   Done running command

INFO:root:Service: vmware-vdcs, Action: status

Service: vmware-vdcs, Action: status

2019-04-03T13:56:02.414Z   Running command: ['/sbin/service', u'vmware-vdcs', 's                                                     tatus']

2019-04-03T13:56:02.652Z   Done running command

INFO:root:Service: vmware-vpx-workflow, Action: status

Service: vmware-vpx-workflow, Action: status

2019-04-03T13:56:02.653Z   Running command: ['/sbin/service', u'vmware-vpx-workf                                                     low', 'status']

2019-04-03T13:56:02.782Z   Done running command

INFO:root:Service: vmware-vsan-health, Action: status

Service: vmware-vsan-health, Action: status

2019-04-03T13:56:02.782Z   Running command: ['/sbin/service', u'vmware-vsan-heal                                                     th', 'status']

2019-04-03T13:56:02.802Z   Done running command

INFO:root:Service: vmware-vsm, Action: status

Service: vmware-vsm, Action: status

2019-04-03T13:56:02.802Z   Running command: ['/sbin/service', u'vmware-vsm', 'st                                                     atus']

2019-04-03T13:56:02.965Z   Done running command

INFO:root:Service: vsphere-client, Action: status

Service: vsphere-client, Action: status

2019-04-03T13:56:02.966Z   Running command: ['/sbin/service', u'vsphere-client',                                                      'status']

2019-04-03T13:56:03.198Z   Done running command

INFO:root:Service: vmware-perfcharts, Action: status

Service: vmware-perfcharts, Action: status

2019-04-03T13:56:03.199Z   Running command: ['/sbin/service', u'vmware-perfchart                                                     s', 'status']

2019-04-03T13:56:03.368Z   Done running command

INFO:root:Service: vmware-vws, Action: status

Service: vmware-vws, Action: status

2019-04-03T13:56:03.371Z   Running command: ['/sbin/service', u'vmware-vws', 'st                                                     atus']

2019-04-03T13:56:03.480Z   Done running command

INFO:root:Running:

applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Fr                                                     amework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vm                                                     ware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance                                                      Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rh                                                     ttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Driven S                                                     torage Service) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd                                                      (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) v                                                     mware-syslog-health (VMware Syslog Health Service) vmware-vpostgres (VMware Post                                                     gres) vmware-vsan-health (VMware VSAN Health Service) vmware-vws (VMware System                                                      and Hardware Health Manager)

Running:

applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Fr                                                     amework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vm                                                     ware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance                                                      Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rh                                                     ttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Driven S                                                     torage Service) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd                                                      (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) v                                                     mware-syslog-health (VMware Syslog Health Service) vmware-vpostgres (VMware Post                                                     gres) vmware-vsan-health (VMware VSAN Health Service) vmware-vws (VMware System                                                      and Hardware Health Manager)

INFO:root:Stopped:

vmware-cm (VMware Component Manager) vmware-eam (VMware ESX Agent Manager) vmwa                                                     re-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configurati                                                     on Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-wat                                                     chdog (VMware vSphere Auto Deploy Waiter) vmware-sca (VMware Service Control Age                                                     nt) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Libr                                                     ary Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (                                                     VMware vCenter Server) vmware-vsm (VMware vService Manager) vsphere-client ()

Stopped:

vmware-cm (VMware Component Manager) vmware-eam (VMware ESX Agent Manager) vmwa                                                     re-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configurati                                                     on Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-wat                                                     chdog (VMware vSphere Auto Deploy Waiter) vmware-sca (VMware Service Control Age                                                     nt) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Libr                                                     ary Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (                                                     VMware vCenter Server) vmware-vsm (VMware vService Manager) vsphere-client ()

vcenter:~ #

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

On df -Th, i get the below:

vcenter:~ # df -Th

Filesystem                           Type   Size  Used Avail Use% Mounted on

/dev/sda3                            ext3    11G  5.5G  4.7G  55% /

udev                                 tmpfs  4.0G  164K  4.0G   1% /dev

tmpfs                                tmpfs  4.0G   44K  4.0G   1% /dev/shm

/dev/sda1                            ext3   128M   39M   83M  32% /boot

/dev/mapper/core_vg-core             ext3    25G  3.3G   21G  14% /storage/core

/dev/mapper/log_vg-log               ext3   9.9G  3.4G  6.0G  36% /storage/log

/dev/mapper/db_vg-db                 ext3   9.9G  227M  9.2G   3% /storage/db

/dev/mapper/dblog_vg-dblog           ext3   5.0G  203M  4.5G   5% /storage/dblog

/dev/mapper/seat_vg-seat             ext3   9.9G  356M  9.0G   4% /storage/seat

/dev/mapper/netdump_vg-netdump       ext3  1001M   18M  932M   2% /storage/netdump

/dev/mapper/autodeploy_vg-autodeploy ext3   9.9G  151M  9.2G   2% /storage/autodeploy

/dev/mapper/invsvc_vg-invsvc         ext3   5.0G  162M  4.6G   4% /storage/invsvc

0 Kudos
Vijay2027
Expert
Expert

Looking at the issue description looks like you are hitting an issue as per below KB:

https://kb.vmware.com/s/article/2147280

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

I followed the KB and reset account password.

Now I can see the landing page as below but when I click Log in to vSphere Web Client then I get the screenshot further below

vsphere webcl.jpg

webcl login.jpg

0 Kudos
Chiligee
Contributor
Contributor

Below are the services running now.

servicestatus1.jpg

0 Kudos
Chiligee
Contributor
Contributor

Vijay, I find this service fails to start: vmware-eam

service_vmware-eam.jpg

0 Kudos
Vijay2027
Expert
Expert

Follow the below KB for eam and start all services

VMware Knowledge Base

0 Kudos
Vijay2027
Expert
Expert

or you can ignore eam and start other services manually

Example: service-control --start vsphere-client

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

So far I had a few other services which i've fixed following vmware KB.

The only remaining ones that fail to start are :

vmware-mbcs

vmware-netdumper

vmware-rbd-watchdog

Currently when I run status on all services I get the below. When I try to manually start any of the above services, they fail to start.

servicestatus2.jpg

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

running the below manually started the services:

service-control --start vmware-netdumper

service-control --start vmware-rbd-watchdog

However, running service-control --start vmware-mbcs does not manually restart, but I get the below message:

vcenter:/var/log/vmware/vdcs # service-control --start vmware-mbcs

INFO:root:Service: vmware-mbcs, Action: start

Service: vmware-mbcs, Action: start

2019-04-03T16:28:41.735Z   Running command: ['/sbin/chkconfig', u'vmware-mbcs']

2019-04-03T16:28:41.783Z   Done running command

2019-04-03T16:28:41.783Z   Running command: ['/sbin/service', u'vmware-mbcs', 'status']

2019-04-03T16:28:41.958Z   Done running command

2019-04-03T16:28:41.959Z   Running command: ['/sbin/service', u'vmware-mbcs', 'start']

2019-04-03T16:28:43.865Z   Done running command

2019-04-03T16:28:43.866Z   Invoked command: ['/sbin/service', u'vmware-mbcs', 'start']

2019-04-03T16:28:43.866Z   RC = 1

Stdout = Starting VMware Message Bus Config Service...

Waiting for VMware Message Bus Config Service....

WARNING: VMware Message Bus Config Service may have failed to start.

Last login: Wed Apr  3 16:26:55 UTC 2019

Stderr =

2019-04-03T16:28:43.866Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-mbcs', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-mbcs', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-mbcs, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-mbcs"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-mbcs'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-mbcs, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-mbcs"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-mbcs'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

vcenter:/var/log/vmware/vdcs #

0 Kudos
Chiligee
Contributor
Contributor

Vijay

The good thing is that now I can see the webclient but none of the passwords I know work. I know the root password but this does not work. I've tried all passwords including the one I reset to via /usr/lib/vmware-vmdir/bin/vdcadmintool.

How can I reset this account and I'm not sure if the failing service is really required?

0 Kudos
Chiligee
Contributor
Contributor

Vijay,

I realised I can reset the password following this KB and now I've been able to log on to vsphere client and can see all the VM's !

VMware Knowledge Base

If it wasnt for your intervention and suggestions, I was at a complete loss.

What would be the best option to upgrade from 6.0 to 6.7? or should I go one version back ie to 6.5?

Thank you

0 Kudos
Vijay2027
Expert
Expert

I would say better option is to go to 6.7. Make sure any 3rd part solutions are compatible with 6.7

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered.

0 Kudos