VMware Cloud Community
tangjiashiertu
Contributor
Contributor

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f285000c480] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Hi guys,

Last week, when I try to login to VCSA, it seem something wrong and I can't login. So I try to reboot the VCSA.

When it boot up,the website only show the text below:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f285000c480] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

And there is a lot of service can not be started.

root@dr-vcsa [ ~ ]# service-control --status

StartPending:

vmware-perfcharts

Stopped:

pschealth vmcam vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-cm vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sts-idmd vmware-stsd vmware-topologysvc vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

What should I do?

Reply
0 Kudos
19 Replies
Alex_Romeo
Leadership
Leadership

Hi,

You can follow these kb

ARomeo

Blog: https://www.aleadmin.it/
Reply
0 Kudos
berndweyand
Expert
Expert

first check if there is enough space left on the partitions - there must be free space escept on the /storage/archive

then try to start services manually and look in the correspondent logs for errors

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

Moderator: Moved to vCenter Server


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
IRIX201110141
Champion
Champion

If half of the services are not running its no surprise that vCenter isnt usable.  Your post lacks some information like VCSA version or how "old" the installation is. Is it a 6.0/6.5 than most likely one of your partions runs out of space (try to login and run df -h) in /storage/log or similar.

Check if DNS and Gateway  accessable from the VCSA because a 6.7 doenst restart without.

If you have planed a vHW upgrade by accident? If so revert to vHW10 .

If you have SnS you should call GSS.

Regards,
Joerg

Reply
0 Kudos
ChrisFD2
VMware Employee
VMware Employee

One of the partitions will be full and/or root account password expired is my bet.

Regards,
Chris
VCIX-DCV 2023 | VCIX-NV 2023 | vExpert *** | CCNA R&S
Reply
0 Kudos
tangjiashiertu
Contributor
Contributor

Thank for all answer. I have already solve my probem.

And its a certificate problem, the certificate of VCSA is expired. Somehow it infect the service, meaning time the service in not running,I can not change my certificate.

My solution is:

Reboot the VSCA. And run

/usr/lib/vmware-vmca/bin/certificate-manager

Choose the option 8 to reset all certificate,and reboot again.

Then the service return to normal, and I can change my certificate again.

johnzhou0806
Contributor
Contributor

very good !thinks

Reply
0 Kudos
adamefril
Contributor
Contributor

if my vcenter type vCenter Server with an external Platform Services Controller

which server do i have to run reset the certificate

 

Reply
0 Kudos
Ajay1988
Expert
Expert

Easiest is to run option 8 on both. First external PSC and then vCenter.

To find which which certs are expired on PSC/VCSA:
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

 

Do make sure ur STS cert is not expired. If STS is expired then u have to do fix STS on PSC first and then touch other certs.

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Tags (1)
adamefril
Contributor
Contributor

it seem already expire STS certificate

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Feb 14 01:09:54 2021 GMT
STORE TRUSTED_ROOTS
Alias : 9ea4dcd4e95fd9a50d1335670ed8689dde9e17e4
Not After : Feb 8 12:55:27 2029 GMT
Alias : 89c61f385128e7bd1c94949560d4d46c8c6aba04
Not After : Apr 20 00:28:51 2031 GMT
Alias : 9c7717657f17133087c434256b2fd1f710f4aa36
Not After : Apr 20 06:09:37 2031 GMT
Alias : 54fbc3e8209fbe1a235c4e02c72b6659b9c7ac8f
Not After : Apr 20 06:11:30 2031 GMT
STORE TRUSTED_ROOT_CRLS
Alias : 2371f2b38d600794a8a7ed596beb79f636b8a8c3
Alias : 7cc970e2054cc969bd2dc2310492d97a45457ddf
Alias : 31e0d6ea03c163712001ed9aa5f4cd743542f3b2
Alias : b4093de36799d9df57cdf88b7d11b1fba27dfcc8
STORE machine
Alias : machine
Not After : Feb 13 12:59:55 2021 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
Not After : Feb 13 12:59:56 2021 GMT
STORE vpxd
Alias : vpxd
Not After : Feb 13 12:59:57 2021 GMT
STORE vpxd-extension
Alias : vpxd-extension
Not After : Feb 13 12:59:57 2021 GMT
STORE SMS
Alias : sms_self_signed
Not After : Feb 14 13:15:30 2029 GMT
STORE BACKUP_STORE

Reply
0 Kudos
Ajay1988
Expert
Expert

For STS expiry  check   https://kb.vmware.com/s/article/79248.

From the output machine_ssl and solution users expired. Reset all certs using 8 better after sts(if expired). 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Tags (2)
adamefril
Contributor
Contributor

@Ajay1988correct:

1.cek STS -----expired

2.repair STS ------completed

3.reset all certificate using 8 option on psc server

4.reset all certificate (8) on vcsa server

solved

Ajay1988
Expert
Expert

Great .

If you think your queries have been answered
Mark the response as "Correct" or "Helpful".

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
jokanga
Contributor
Contributor

Thanks of this fix......i have had this problem since December 2021. Searched every where without luck but then today i bumped on this and it worked like a charm. You are my hero man 😀

rpurboyo
Contributor
Contributor

it works for my client vcenter problem.

same error and same problem solving.

thanx in advance, you make my day 😜😜

joshuacseldon
Contributor
Contributor

Im new to this but did you run /usr/lib/vmware-vmca/bin/certificate-manager in the VSCA server?

Reply
0 Kudos
IsuruSameera
Contributor
Contributor

 yes, you have to SSH to the VCSA appliance using root account and execute the above script/command.



please thumbs up if this helps.

https://www.iwadmin.info/
Reply
0 Kudos
JakeClaytonTiso
Contributor
Contributor

hi All

 

I have a similar problem, and these are my certificates that have expired. I did try and replace them with option 8 - and it asked me to confirm optional settings, i just skipped through these and rebooted vCentre Server - but is it something else perhaps? the Hard drive space is fine 

Certs.PNG

 

Reply
0 Kudos
JakeClaytonTiso
Contributor
Contributor

ignore it was an invalid certificate and this post helped me fix. so happy thank you! 

 

https://kb.vmware.com/s/article/76719

Reply
0 Kudos