VMware Cloud Community
Rambo21
Contributor
Contributor

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055ed2fdb3820] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055ed2fdb3820] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

VMware vCenter Server Appliance 6.7.0.10000

NSX manager 6.4.0

18 Replies
Rambo21
Contributor
Contributor

root@172 [ ~ ]# service-control --status --all

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-netdumper vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-perfcharts vmware-pod vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps

service-control --status --start  

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, cm services. Error: Operation timed out

Reply
0 Kudos
daphnissov
Immortal
Immortal

Did you install your vCenter using a FQDN and have DNS records (forward and reverse) present in your DNS server?

msripada
Virtuoso
Virtuoso

is it vcenter with external psc or embedded psc?

can you check the vmware-cm log - var/log/vmware/cm

Thanks,

MS

Reply
0 Kudos
meskirian
Contributor
Contributor

it looks like i got the same problem:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fbde8005fc0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

StartPending:

vmware-vsan-health

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-psc-client vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vcha vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vpxd vmware-vpxd-svcs vmware-vsm

Error executing start on service vpxd. Details {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vpxd"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vpxd'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Service-control failed. Error {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vpxd"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vpxd'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

Could any one help me?

vcenter is embedded

P.S.

i tried solution from the post VCSA 6.5 Web GUI Error

and it didn't help.

Reply
0 Kudos
Mokshir
Enthusiast
Enthusiast

Running into the same issue.  vCenter was running fine until a 2 weeks ago. The it died. I found out it ran out space to expan the swap file so I reserved all the memory for the VM. VM came up and ran into this error

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005650c59d5e50] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

In the VPXD logs I see the following:

76 sub=MoOptionMgr] [OptionManagerMo] Invoking callbacks for key log.level, pre commit

2018-05-24T18:50:27.852Z info vpxd[7F3B33476800] [Originator@6876 sub=MoOptionMgr] [OptionManagerMo] Invoking callbacks for key log.level, pre commit

2018-05-24T18:50:27.852Z info vpxd[7F3B33476800] [Originator@6876 sub=MoOptionMgr] No change to log.level

2018-05-24T18:50:27.858Z info vpxd[7F3B33476800] [Originator@6876 sub=Main] [VpxdMain] Setting OpenSSL verify locations CAFile= CAPath=/etc/ssl/certs

2018-05-24T18:50:27.858Z info vpxd[7F3B33476800] [Originator@6876 sub=Default] Creating SSL Contexts

2018-05-24T18:50:27.886Z info vpxd[7F3B33476800] [Originator@6876 sub=WorkQueue.vpxd.stats.persister.queue] Created: WorkQueue.vpxd.stats.persister.queue, type = serial, priority = 16 , itemWeight = 1

2018-05-24T18:50:27.891Z info vpxd[7F3B33476800] [Originator@6876 sub=DbParallelLoader] Num rows: 7, num CPUs: 4, threads: 1, step: 7

2018-05-24T18:50:27.941Z error vpxd[7F3B33476800] [Originator@6876 sub=DbBulkLoader] [VpxdDbBulkLoader::Load] Failed to load tableDef 41 from database: Column DVS_ID does not exist in table VPX_DVS_NRP

Tried restarting the service. verified FQDN resolution no luck. Seems like a DVS issue to which this vCenter VM is connected to but I have not clue how to fix this.

Help.

Reply
0 Kudos
grep_boy
Contributor
Contributor

I had the same issue, which finally turned out to be a certificate problem.

I fixed by logging into VCSA as root, and running the certificate manager: /usr/lib/vmware-vmca/bin/certificate-manager

Good KB article at https://kb.vmware.com/kb/2097936

I used option 3 to regenerate all of the SSL certificates which got me up and running.

josearpa
Contributor
Contributor

Hi.

I found that the process update failed because root password expired.  I changed root password and start update process from ssh.

Good luck.

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

For all our community members,

Error 503 is Service Unavailable when trying to connect to vCenter server service or it can be webclient as well.

This error is very generic in nature however possible troubleshootings include if the services are started or failed. If failed, look into respective logs.

In your case, the update failed so expected the service to be failed.

Other cases, service might be crashed or certificates expired or partition full issues.. It can vary from vCenter to vCenter and environment to environment.

Hope this helps for whoever trying to troubleshoot.

Thanks,

MS

ammadkazmi1
Contributor
Contributor

Thank you...... It has resolve my issue

Ammad Kazmi

Reply
0 Kudos
apatten
Contributor
Contributor

I'll add that I had this same issue and it turned out to be a issue with name resolution.

From the /var/log/vmware/cm/cm.log file:

2019-05-22T13:06:33.537Z [main [] WARN  com.vmware.cis.services.cm.service.util.LsUtils ()] Call to lookup service failed; uri:https://<my vcsa's hostname>/lookupservice/sdk [com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.UnknownHostException: <my vcsa's hostname>]

I added an entry in the /etc/hosts file and that resolved the issue after restarting the appliance.

blackcopy
Contributor
Contributor

In my case, the /storage/seat partition was filled over 95%-threshold. Increasing space helped.

I found it out by digging through the vpxd.log

huchord
Enthusiast
Enthusiast

Apatten: what entries did you put in your /etc/host file?

Here's mine right now - still having the problem as the OP.

pastedImage_0.png

Reply
0 Kudos
Jack_Moon
VMware Employee
VMware Employee

Thanks for sharing your experiences and i solve this issue to create a new certificate with running /usr/lib/vmware-vmca/bin/certificate-manager as you guided. Thank! grep_boy

Reply
0 Kudos
dineshbalendran
Contributor
Contributor

Thank you very much! This has worked!

running /usr/lib/vmware-vmca/bin/certificate-manager and regenerate all of the SSL certificates did the trick!

Reply
0 Kudos
psychobunny
Contributor
Contributor

We had a power outage and looks like vpxd service stopped along with a few other, this resolved my issue:

SSH into VCSA

shell.set --enabled true
shell
service-control --status --all
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha vmware-vpxd vsan-dps

service-control --start --all

Reply
0 Kudos
dineshbalendran
Contributor
Contributor

Different scenarios where this could occur.

If you had changed the VCSA DNS name or changed the IP then I encountered this error and the way I was able to resolve the issue is by re-generating a new VMCA Root Certificate and replace all certificates by running: /usr/lib/vmware-vmca/bin/certificate-manager

The reason why I had to change the IP was because I was deploying the appliance: VMware-VCSA-all-6.7.0-14070457_OVF10 onto Type 2 hypervisor with "bridged" connection - DHCP IP NOT static IP, hence I had to after deploying the appliance change to a static NAT IP.

Then i rebooted the appliance I had encountered this error. I had no other option other than re-generating a new VMCA Root Certificate and replace all certificates by running: /usr/lib/vmware-vmca/bin/certificate-manager to fix the issue.

 

This was though when I was running on version: VMware-VCSA-all-6.7.0-14070457

 

This error I did not encounter when I tested with anther version - attempted to deploy the appliance version: "VMware-VCSA-all-6.7.0-18485166 OVF" for the deployment scenario I explained here. I changed from DHCP to Static NAT IP after deployment and restarted the appliance I was able to browse the vCenter via the IP - did not have to re-generate a new VMCA Root Certificate. 

The service seem to start without any issues. 

 

 

Reply
0 Kudos
DUONGHO
Contributor
Contributor

How did you do?

Reply
0 Kudos
andrewckloh
Contributor
Contributor

andrewckloh_0-1685342454659.png

Regenerate all certiticates really can fix this.

 

Reply
0 Kudos