VCSA no healthy upstream and services failing to start

[jmelcher21]

I had tried adding VCSA to a domain, then I started getting these issues.  I'm not certain that is the cause, but that was what I was doing right around the time it happened.  When I tried service-control --start --all, I see several fail.  Any help would be appreciated, thank you!

root@vcenter01 [ ~ ]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

[Ajay1988]

I suspect expired solution user certs. 

Run    this command on SSH fo find : 

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

If expired check   https://kb.vmware.com/s/article/2112283

[jmelcher21]

Here is the output, the look good.

root@vcenter01 [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Apr 25 06:09:33 2023 GMT
STORE TRUSTED_ROOTS
Alias : 8e02ecb5feb5b25ad64345dc430fe629fa20bff1
Not After : Apr 19 18:09:32 2031 GMT
STORE TRUSTED_ROOT_CRLS
Alias : e1eb875a4a5e9b69b64a73ec47ba4e459e3e2baf
STORE machine
Alias : machine
Not After : Apr 19 18:09:32 2031 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
Not After : Apr 19 18:09:32 2031 GMT
STORE vpxd
Alias : vpxd
Not After : Apr 19 18:09:32 2031 GMT
STORE vpxd-extension
Alias : vpxd-extension
Not After : Apr 19 18:09:32 2031 GMT
STORE hvc
Alias : hvc
Not After : Apr 19 18:09:32 2031 GMT
STORE data-encipherment
Alias : data-encipherment
Not After : Apr 19 18:09:32 2031 GMT
STORE APPLMGMT_PASSWORD
STORE SMS
Alias : sms_self_signed
Not After : Apr 24 18:18:34 2031 GMT
STORE wcp
Alias : wcp
Not After : Apr 24 18:11:47 2023 GMT

[Ajay1988]

cert seems fine. What version of vcsa is this ?

Investigate the below logs for issues. 

/var/log/vmware/vapi/endpoint/endpoint.log
/var/log/vmware/vpxd-svcs/vpxd-svcs.log

[jmelcher21]

VMware vCenter Server 7.0.2.00000

I attached the logs in a txt, in a zip.  I see some errors, but I'm not sure about them.

[Ajay1988]

Was this upgraded from an earlier version or new install of vcsa 7.0U2 ?

Noticed "Could not find any solution users from SSO" 

Download the attached script 'solutionusers_fixer.zip and extract  and upload the .sh file to /tmp folder on VCSA.
Make the file executable: chmod +rx /tmp/solutionusers_fixer.sh
Run the script : /tmp/solutionusers_fixer.sh

Restart all services : service-control --stop --all && service-control --start --all

[jmelcher21]

Did not work.  I really appreciate your help.  This is a new install.

root@vcenter01 [ /tmp ]# chmod +rx solutionusers_fixer.sh
root@vcenter01 [ /tmp ]# ./solutionusers_fixer.sh
Solution users health check started on Tue Apr 27 03:20:09 UTC 2021
SSO domain: ironrising.com

Enter SSO admin password:
Looking for machine user in groups

Looking in cn=administrators,cn=builtin
Looking in cn=SystemConfiguration.Administrators
Looking in cn=ComponentManager.Administrators

Looking for web client user in groups

Looking in cn=administrators,cn=builtin
Looking in cn=actasusers
Looking in cn=licenseservice.administrators

Looking for vpxd user in groups

Looking in cn=users,cn=builtin
Looking in cn=LicenseService.Administrators
Looking in cn=systemconfiguration.administrators
Looking in cn=componentmanager.administrators

Looking for vpxd-extension user in groups

Looking in cn=users,cn=builtin
Looking in cn=actasusers
Looking in cn=systemconfiguration.administrators
Looking in cn=componentmanager.administrators
Done on Tue Apr 27 03:20:14 UTC 2021. You may need to restart services...
root@vcenter01 [ /tmp ]# service-control --stop --all && service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service observability...
Successfully stopped service observability
Performing stop operation on service vmware-pod...
Successfully stopped service vmware-pod
Performing stop operation on service vmware-vdtc...
Successfully stopped service vmware-vdtc
Performing stop operation on profile: ALL...
Successfully stopped service vmware-vmon
Successfully stopped profile: ALL.
Performing stop operation on service vmcad...
Successfully stopped service vmcad
Performing stop operation on service vmdird...
Successfully stopped service vmdird
Performing stop operation on service vmafdd...
Successfully stopped service vmafdd
Performing stop operation on service lwsmd...
Successfully stopped service lwsmd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, vpxd-svcs services. Error: Operation timed out

[Ajay1988]

Seems to me as user missing in a sso group  .   

2021-04-26T03:08:54.918Z [pool-6-thread-1 ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user
: Administrator@ironrising.com

Check if above account(Administrator@ironrising.com) is present  in Administrators and SystemConfiguration.Administrators

[jmelcher21]

AJ,

How do I check that if I can't access vcenter?   I checked vCenter Server Management and didn't see that sections (on port 5480).  

[Ajay1988]

try to start the   vsphere-ui service and see if u can login to web-client and check it  . If not this needs a SR with VMware Support