When I try to login on my vCenter Server Appliance 6.0.0 with VMWare vSphere Client I am getting this error:
I have space:
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 11G 5.7G 4.6G 56% /
udev 4.0G 164K 4.0G 1% /dev
tmpfs 4.0G 84K 4.0G 1% /dev/shm
/dev/sda1 128M 38M 84M 31% /boot
/dev/mapper/core_vg-core 27G 277M 25G 2% /storage/core
/dev/mapper/log_vg-log 40G 9.3G 29G 25% /storage/log
/dev/mapper/db_vg-db 16G 1.1G 14G 7% /storage/db
/dev/mapper/dblog_vg-dblog 22G 847M 20G 5% /storage/dblog
/dev/mapper/seat_vg-seat 18G 777M 17G 5% /storage/seat
/dev/mapper/netdump_vg-netdump 2.0G 18M 1.9G 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 17G 170M 16G 2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc 8.9G 185M 8.3G 3% /storage/invsvc
The problem looks like to be an expired cert:
/var/log/vmware/sso/vmware-identity-sts.log
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.validateSigningCert(TokenLifetimeRemediator.java:92)
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:66)
at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:191)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:48)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator.issueToken(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.sts.impl.STSImpl.issueToken(STSImpl.java:342)
... 48 more
[2023-02-13T12:01:36.047Z tomcat-http--27 vsphere.local 0fc7d54d-516e-41b9-8cb4-e3ee88766677 INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Mon Feb 13 12:01:36 UTC 2023, endTime=Tue Feb 14 12:01:36 UTC 2023] :: Signing certificate is not valid at Mon Feb 13 12:01:36 UTC 2023, cert validity: TimePeriod [startTime=Wed Dec 12 16:59:36 UTC 2012, endTime=Sun Dec 11 16:59:36 UTC 2022]
This certs looks OK:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Mar 31 17:13:39 2024 GMT
[*] Store : TRUSTED_ROOTS
Alias : 3c05e1fd963e4fb30f87618ebcd549bc03c6216a
Not After : Mar 7 12:49:07 2025 GMT
Alias : d13256cbcbbee2d364c1d53eaf377b225d130ce5
Not After : Mar 31 17:13:39 2024 GMT
[*] Store : machine
Alias : machine
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vpxd
Alias : vpxd
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vpxd-extension
Alias : vpxd-extension
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vsphere-webclient
Alias : vsphere-webclient
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : SMS
Alias : sms_self_signed
Not After : Dec 13 17:05:18 2022 GMT
I didn't find a proper KB for my version, but I followed this one:
https://kb.vmware.com/s/article/76719
Every thing looks fine till this point:
./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for vcenter55 started on Mon Feb 13 12:30:48 UTC 2023
Detected DN: cn=vcenter55,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vcenter55
Detected PSC: vcenter55
Detected SSO domain name: vsphere.local
Detected Machine ID: 06887578-2176-4dec-a243-127ebd2fd6af
Detected IP Address: 10.101.255.247
Domain CN: dc=vsphere,dc=local
./fixsts.sh: line 55: administrator@${DOMAIN^^}: bad substitution
==================================
==================================
Detected Root's certificate expiration date: 2025 Mar 7
Detected today's date: 2023 Feb 13
==================================
Exporting and generating STS certificate
Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success
Enter password for administrator@vsphere.local:
Amount of tenant credentials:
Amount of trustedcertchains:
Applying newly generated STS certificate to SSO domain
ldap_add: No such object (32)
additional info: parent (cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local) not found, ((MDB_NOTFOUND: No matching key/data pair found)(cn=trustedcertificatechains,cn=vsphere.local,cn=tenants,cn=identitymanager,cn=services,dc=vsphere,dc=local))
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================
I dint't have problem on this command:
service-control --stop --all
But when I do start...
service-control --start --all
I get this error:
INFO:root:Service: vmafdd, Action: start
Service: vmafdd, Action: start
2023-02-13T12:34:57.560Z Running command: ['/sbin/chkconfig', u'vmafdd']
2023-02-13T12:34:57.599Z Done running command
2023-02-13T12:34:57.599Z Running command: ['/sbin/service', u'vmafdd', 'status']
2023-02-13T12:34:57.616Z Done running command
2023-02-13T12:34:57.616Z Running command: ['/sbin/chkconfig', '--force', u'vmafdd', 'on']
2023-02-13T12:34:57.652Z Done running command
2023-02-13T12:34:57.652Z Running command: ['/sbin/service', u'vmafdd', 'start']
2023-02-13T12:34:57.699Z Done running command
2023-02-13T12:34:57.699Z Successfully started service vmafdd
INFO:root:Service: vmware-rhttpproxy, Action: start
Service: vmware-rhttpproxy, Action: start
2023-02-13T12:34:57.700Z Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy']
2023-02-13T12:34:57.737Z Done running command
2023-02-13T12:34:57.737Z Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status']
2023-02-13T12:34:57.812Z Done running command
2023-02-13T12:34:57.812Z Running command: ['/sbin/chkconfig', '--force', u'vmware-rhttpproxy', 'on']
2023-02-13T12:34:57.847Z Done running command
2023-02-13T12:34:57.847Z Running command: ['/sbin/service', u'vmware-rhttpproxy', 'start']
2023-02-13T12:34:57.871Z Done running command
2023-02-13T12:34:57.871Z Successfully started service vmware-rhttpproxy
INFO:root:Service: vmdird, Action: start
Service: vmdird, Action: start
2023-02-13T12:34:57.872Z Running command: ['/sbin/chkconfig', u'vmdird']
2023-02-13T12:34:57.907Z Done running command
2023-02-13T12:34:57.907Z Running command: ['/sbin/service', u'vmdird', 'status']
2023-02-13T12:34:57.925Z Done running command
2023-02-13T12:34:57.925Z Running command: ['/sbin/chkconfig', '--force', u'vmdird', 'on']
2023-02-13T12:34:57.960Z Done running command
2023-02-13T12:34:57.960Z Running command: ['/sbin/service', u'vmdird', 'start']
2023-02-13T12:34:59.126Z Done running command
2023-02-13T12:34:59.126Z Successfully started service vmdird
INFO:root:Service: vmcad, Action: start
Service: vmcad, Action: start
2023-02-13T12:34:59.127Z Running command: ['/sbin/chkconfig', u'vmcad']
2023-02-13T12:34:59.164Z Done running command
2023-02-13T12:34:59.164Z Running command: ['/sbin/service', u'vmcad', 'status']
2023-02-13T12:34:59.179Z Done running command
2023-02-13T12:34:59.180Z Running command: ['/sbin/chkconfig', '--force', u'vmcad', 'on']
2023-02-13T12:34:59.217Z Done running command
2023-02-13T12:34:59.217Z Running command: ['/sbin/service', u'vmcad', 'start']
2023-02-13T12:35:00.259Z Done running command
2023-02-13T12:35:00.259Z Successfully started service vmcad
INFO:root:Service: vmware-sts-idmd, Action: start
Service: vmware-sts-idmd, Action: start
2023-02-13T12:35:00.260Z Running command: ['/sbin/chkconfig', u'vmware-sts-idmd']
2023-02-13T12:35:00.295Z Done running command
2023-02-13T12:35:00.295Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2023-02-13T12:35:00.307Z Done running command
2023-02-13T12:35:00.307Z Running command: ['/sbin/chkconfig', '--force', u'vmware-sts-idmd', 'on']
2023-02-13T12:35:00.344Z Done running command
2023-02-13T12:35:00.344Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'start']
2023-02-13T12:35:00.423Z Done running command
2023-02-13T12:35:00.423Z Successfully started service vmware-sts-idmd
INFO:root:Service: vmware-stsd, Action: start
Service: vmware-stsd, Action: start
2023-02-13T12:35:00.424Z Running command: ['/sbin/chkconfig', u'vmware-stsd']
2023-02-13T12:35:00.462Z Done running command
2023-02-13T12:35:00.462Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2023-02-13T12:35:00.488Z Done running command
2023-02-13T12:35:00.488Z Running command: ['/sbin/chkconfig', '--force', u'vmware-stsd', 'on']
2023-02-13T12:35:00.545Z Done running command
2023-02-13T12:35:00.545Z Running command: ['/sbin/service', u'vmware-stsd', 'start']
2023-02-13T12:35:05.223Z Done running command
2023-02-13T12:35:05.223Z Successfully started service vmware-stsd
INFO:root:Service: vmware-cm, Action: start
Service: vmware-cm, Action: start
2023-02-13T12:35:05.224Z Running command: ['/sbin/chkconfig', u'vmware-cm']
2023-02-13T12:35:05.262Z Done running command
2023-02-13T12:35:05.262Z Running command: ['/sbin/service', u'vmware-cm', 'status']
2023-02-13T12:35:05.374Z Done running command
2023-02-13T12:35:05.374Z Running command: ['/sbin/chkconfig', '--force', u'vmware-cm', 'on']
2023-02-13T12:35:05.412Z Done running command
2023-02-13T12:35:05.412Z Running command: ['/sbin/service', u'vmware-cm', 'start']
2023-02-13T12:35:22.263Z Done running command
2023-02-13T12:35:22.264Z Successfully started service vmware-cm
INFO:root:Service: vmware-cis-license, Action: start
Service: vmware-cis-license, Action: start
2023-02-13T12:35:22.264Z Running command: ['/sbin/chkconfig', u'vmware-cis-license']
2023-02-13T12:35:22.301Z Done running command
2023-02-13T12:35:22.301Z Running command: ['/sbin/service', u'vmware-cis-license', 'status']
2023-02-13T12:35:22.403Z Done running command
2023-02-13T12:35:22.403Z Running command: ['/sbin/chkconfig', '--force', u'vmware-cis-license', 'on']
2023-02-13T12:35:22.439Z Done running command
2023-02-13T12:35:22.439Z Running command: ['/sbin/service', u'vmware-cis-license', 'start']
2023-02-13T12:35:27.099Z Done running command
2023-02-13T12:35:27.099Z Successfully started service vmware-cis-license
INFO:root:Service: vmware-sca, Action: start
Service: vmware-sca, Action: start
2023-02-13T12:35:27.100Z Running command: ['/sbin/chkconfig', u'vmware-sca']
2023-02-13T12:35:27.136Z Done running command
2023-02-13T12:35:27.136Z Running command: ['/sbin/service', u'vmware-sca', 'status']
2023-02-13T12:35:27.237Z Done running command
2023-02-13T12:35:27.237Z Running command: ['/sbin/chkconfig', '--force', u'vmware-sca', 'on']
2023-02-13T12:35:27.274Z Done running command
2023-02-13T12:35:27.275Z Running command: ['/sbin/service', u'vmware-sca', 'start']
2023-02-13T12:35:31.930Z Done running command
2023-02-13T12:35:31.930Z Successfully started service vmware-sca
INFO:root:Service: applmgmt, Action: start
Service: applmgmt, Action: start
2023-02-13T12:35:31.931Z Running command: ['/sbin/chkconfig', u'applmgmt']
2023-02-13T12:35:31.969Z Done running command
2023-02-13T12:35:31.969Z Running command: ['/sbin/service', u'applmgmt', 'status']
2023-02-13T12:35:32.017Z Done running command
2023-02-13T12:35:32.017Z Running command: ['/sbin/chkconfig', '--force', u'applmgmt', 'on']
2023-02-13T12:35:32.053Z Done running command
2023-02-13T12:35:32.053Z Running command: ['/sbin/service', u'applmgmt', 'start']
2023-02-13T12:35:32.273Z Done running command
2023-02-13T12:35:32.273Z Successfully started service applmgmt
INFO:root:Service: vmware-netdumper, Action: start
Service: vmware-netdumper, Action: start
2023-02-13T12:35:32.274Z Running command: ['/sbin/chkconfig', u'vmware-netdumper']
2023-02-13T12:35:32.312Z Done running command
'vmware-netdumper' startMode is Manual, skipping to start:
INFO:root:'vmware-netdumper' startMode is Manual, skipping to start:
INFO:root:Service: vmware-syslog, Action: start
Service: vmware-syslog, Action: start
2023-02-13T12:35:32.313Z Running command: ['/sbin/chkconfig', u'vmware-syslog']
2023-02-13T12:35:32.351Z Done running command
2023-02-13T12:35:32.351Z Running command: ['/sbin/service', u'vmware-syslog', 'status']
2023-02-13T12:35:32.403Z Done running command
2023-02-13T12:35:32.403Z Running command: ['/sbin/chkconfig', '--force', u'vmware-syslog', 'on']
2023-02-13T12:35:32.442Z Done running command
2023-02-13T12:35:32.442Z Running command: ['/sbin/service', u'vmware-syslog', 'start']
2023-02-13T12:35:35.339Z Done running command
2023-02-13T12:35:35.339Z Successfully started service vmware-syslog
INFO:root:Service: vmware-syslog-health, Action: start
Service: vmware-syslog-health, Action: start
2023-02-13T12:35:35.340Z Running command: ['/sbin/chkconfig', u'vmware-syslog-health']
2023-02-13T12:35:35.376Z Done running command
2023-02-13T12:35:35.376Z Running command: ['/sbin/service', u'vmware-syslog-health', 'status']
2023-02-13T12:35:35.492Z Done running command
2023-02-13T12:35:35.493Z Running command: ['/sbin/chkconfig', '--force', u'vmware-syslog-health', 'on']
2023-02-13T12:35:35.529Z Done running command
2023-02-13T12:35:35.529Z Running command: ['/sbin/service', u'vmware-syslog-health', 'start']
2023-02-13T12:35:37.186Z Done running command
2023-02-13T12:35:37.186Z Successfully started service vmware-syslog-health
INFO:root:Service: vmware-vapi-endpoint, Action: start
Service: vmware-vapi-endpoint, Action: start
2023-02-13T12:35:37.187Z Running command: ['/sbin/chkconfig', u'vmware-vapi-endpoint']
2023-02-13T12:35:37.225Z Done running command
2023-02-13T12:35:37.225Z Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status']
2023-02-13T12:35:37.271Z Done running command
2023-02-13T12:35:37.271Z Running command: ['/sbin/chkconfig', '--force', u'vmware-vapi-endpoint', 'on']
2023-02-13T12:35:37.306Z Done running command
2023-02-13T12:35:37.306Z Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'start']
2023-02-13T12:35:41.118Z Done running command
2023-02-13T12:35:41.118Z Successfully started service vmware-vapi-endpoint
INFO:root:Service: vmware-vpostgres, Action: start
Service: vmware-vpostgres, Action: start
2023-02-13T12:35:41.119Z Running command: ['/sbin/chkconfig', u'vmware-vpostgres']
2023-02-13T12:35:41.155Z Done running command
2023-02-13T12:35:41.155Z Running command: ['/sbin/service', u'vmware-vpostgres', 'status']
2023-02-13T12:35:41.241Z Done running command
2023-02-13T12:35:41.241Z Running command: ['/sbin/chkconfig', '--force', u'vmware-vpostgres', 'on']
2023-02-13T12:35:41.278Z Done running command
2023-02-13T12:35:41.278Z Running command: ['/sbin/service', u'vmware-vpostgres', 'start']
2023-02-13T12:35:42.413Z Done running command
2023-02-13T12:35:42.413Z Successfully started service vmware-vpostgres
INFO:root:Service: vmware-invsvc, Action: start
Service: vmware-invsvc, Action: start
2023-02-13T12:35:42.414Z Running command: ['/sbin/chkconfig', u'vmware-invsvc']
2023-02-13T12:35:42.456Z Done running command
2023-02-13T12:35:42.456Z Running command: ['/sbin/service', u'vmware-invsvc', 'status']
2023-02-13T12:35:42.558Z Done running command
2023-02-13T12:35:42.558Z Running command: ['/sbin/chkconfig', '--force', u'vmware-invsvc', 'on']
2023-02-13T12:35:42.597Z Done running command
2023-02-13T12:35:42.597Z Running command: ['/sbin/service', u'vmware-invsvc', 'start']
2023-02-13T12:35:56.388Z Done running command
2023-02-13T12:35:56.388Z Successfully started service vmware-invsvc
INFO:root:Service: vmware-mbcs, Action: start
Service: vmware-mbcs, Action: start
2023-02-13T12:35:56.389Z Running command: ['/sbin/chkconfig', u'vmware-mbcs']
2023-02-13T12:35:56.425Z Done running command
'vmware-mbcs' startMode is Manual, skipping to start:
INFO:root:'vmware-mbcs' startMode is Manual, skipping to start:
INFO:root:Service: vmware-vpxd, Action: start
Service: vmware-vpxd, Action: start
2023-02-13T12:35:56.426Z Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2023-02-13T12:35:56.462Z Done running command
2023-02-13T12:35:56.462Z Running command: ['/sbin/service', u'vmware-vpxd', 'status']
2023-02-13T12:35:56.813Z Done running command
2023-02-13T12:35:56.813Z Running command: ['/sbin/chkconfig', '--force', u'vmware-vpxd', 'on']
2023-02-13T12:35:56.847Z Done running command
2023-02-13T12:35:56.847Z Running command: ['/sbin/service', u'vmware-vpxd', 'start']
2023-02-13T12:42:19.467Z Done running command
2023-02-13T12:42:19.467Z Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']
2023-02-13T12:42:19.467Z RC = 1
Stdout = vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
Waiting for the embedded database to start up: success
Executing pre-startup scripts...
vmware-vpxd: Starting vpxd by administrative request.
success
vmware-vpxd: Waiting for vpxd to start listening for requests on 8089
Waiting for vpxd to initialize: ......................................failed
failed
vmware-vpxd: vpxd failed to initialize in time.
vpxd is already starting up. Aborting the request.
Stderr =
2023-02-13T12:42:19.467Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-vpxd, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-vpxd, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
service-control --status --all
INFO:root:Service: vmafdd, Action: status
Service: vmafdd, Action: status
2023-02-13T12:45:28.047Z Running command: ['/sbin/service', u'vmafdd', 'status']
2023-02-13T12:45:28.062Z Done running command
INFO:root:Service: vmware-rhttpproxy, Action: status
Service: vmware-rhttpproxy, Action: status
2023-02-13T12:45:28.063Z Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status']
2023-02-13T12:45:28.119Z Done running command
INFO:root:Service: vmdird, Action: status
Service: vmdird, Action: status
2023-02-13T12:45:28.120Z Running command: ['/sbin/service', u'vmdird', 'status']
2023-02-13T12:45:28.135Z Done running command
INFO:root:Service: vmcad, Action: status
Service: vmcad, Action: status
2023-02-13T12:45:28.135Z Running command: ['/sbin/service', u'vmcad', 'status']
2023-02-13T12:45:28.150Z Done running command
INFO:root:Service: vmware-sts-idmd, Action: status
Service: vmware-sts-idmd, Action: status
2023-02-13T12:45:28.150Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2023-02-13T12:45:28.161Z Done running command
INFO:root:Service: vmware-stsd, Action: status
Service: vmware-stsd, Action: status
2023-02-13T12:45:28.162Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2023-02-13T12:45:28.192Z Done running command
INFO:root:Service: vmware-cm, Action: status
Service: vmware-cm, Action: status
2023-02-13T12:45:28.193Z Running command: ['/sbin/service', u'vmware-cm', 'status']
2023-02-13T12:45:28.313Z Done running command
INFO:root:Service: vmware-cis-license, Action: status
Service: vmware-cis-license, Action: status
2023-02-13T12:45:28.314Z Running command: ['/sbin/service', u'vmware-cis-license', 'status']
2023-02-13T12:45:28.428Z Done running command
INFO:root:Service: vmware-sca, Action: status
Service: vmware-sca, Action: status
2023-02-13T12:45:28.428Z Running command: ['/sbin/service', u'vmware-sca', 'status']
2023-02-13T12:45:28.536Z Done running command
INFO:root:Service: applmgmt, Action: status
Service: applmgmt, Action: status
2023-02-13T12:45:28.537Z Running command: ['/sbin/service', u'applmgmt', 'status']
2023-02-13T12:45:28.583Z Done running command
INFO:root:Service: vmware-netdumper, Action: status
Service: vmware-netdumper, Action: status
2023-02-13T12:45:28.584Z Running command: ['/sbin/service', u'vmware-netdumper', 'status']
2023-02-13T12:45:28.629Z Done running command
INFO:root:Service: vmware-syslog, Action: status
Service: vmware-syslog, Action: status
2023-02-13T12:45:28.630Z Running command: ['/sbin/service', u'vmware-syslog', 'status']
2023-02-13T12:45:28.680Z Done running command
INFO:root:Service: vmware-syslog-health, Action: status
Service: vmware-syslog-health, Action: status
2023-02-13T12:45:28.681Z Running command: ['/sbin/service', u'vmware-syslog-health', 'status']
2023-02-13T12:45:28.793Z Done running command
INFO:root:Service: vmware-vapi-endpoint, Action: status
Service: vmware-vapi-endpoint, Action: status
2023-02-13T12:45:28.794Z Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status']
2023-02-13T12:45:28.851Z Done running command
INFO:root:Service: vmware-vpostgres, Action: status
Service: vmware-vpostgres, Action: status
2023-02-13T12:45:28.852Z Running command: ['/sbin/service', u'vmware-vpostgres', 'status']
2023-02-13T12:45:28.936Z Done running command
INFO:root:Service: vmware-invsvc, Action: status
Service: vmware-invsvc, Action: status
2023-02-13T12:45:28.937Z Running command: ['/sbin/service', u'vmware-invsvc', 'status']
2023-02-13T12:45:29.052Z Done running command
INFO:root:Service: vmware-mbcs, Action: status
Service: vmware-mbcs, Action: status
2023-02-13T12:45:29.053Z Running command: ['/sbin/service', u'vmware-mbcs', 'status']
2023-02-13T12:45:29.152Z Done running command
INFO:root:Service: vmware-vpxd, Action: status
Service: vmware-vpxd, Action: status
2023-02-13T12:45:29.153Z Running command: ['/sbin/service', u'vmware-vpxd', 'status']
2023-02-13T12:45:29.489Z Done running command
INFO:root:Service: vmware-eam, Action: status
Service: vmware-eam, Action: status
2023-02-13T12:45:29.489Z Running command: ['/sbin/service', u'vmware-eam', 'status']
2023-02-13T12:45:29.541Z Done running command
INFO:root:Service: vmware-rbd-watchdog, Action: status
Service: vmware-rbd-watchdog, Action: status
2023-02-13T12:45:29.542Z Running command: ['/sbin/service', u'vmware-rbd-watchdog', 'status']
2023-02-13T12:45:29.589Z Done running command
INFO:root:Service: vmware-sps, Action: status
Service: vmware-sps, Action: status
2023-02-13T12:45:29.590Z Running command: ['/sbin/service', u'vmware-sps', 'status']
2023-02-13T12:45:29.706Z Done running command
INFO:root:Service: vmware-vdcs, Action: status
Service: vmware-vdcs, Action: status
2023-02-13T12:45:29.706Z Running command: ['/sbin/service', u'vmware-vdcs', 'status']
2023-02-13T12:45:29.804Z Done running command
INFO:root:Service: vmware-vpx-workflow, Action: status
Service: vmware-vpx-workflow, Action: status
2023-02-13T12:45:29.805Z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status']
2023-02-13T12:45:29.856Z Done running command
INFO:root:Service: vmware-vsm, Action: status
Service: vmware-vsm, Action: status
2023-02-13T12:45:29.857Z Running command: ['/sbin/service', u'vmware-vsm', 'status']
2023-02-13T12:45:29.911Z Done running command
INFO:root:Service: vsphere-client, Action: status
Service: vsphere-client, Action: status
2023-02-13T12:45:29.911Z Running command: ['/sbin/service', u'vsphere-client', 'status']
2023-02-13T12:45:30.009Z Done running command
INFO:root:Service: vmware-perfcharts, Action: status
Service: vmware-perfcharts, Action: status
2023-02-13T12:45:30.010Z Running command: ['/sbin/service', u'vmware-perfcharts', 'status']
2023-02-13T12:45:30.064Z Done running command
INFO:root:Service: vmware-vws, Action: status
Service: vmware-vws, Action: status
2023-02-13T12:45:30.065Z Running command: ['/sbin/service', u'vmware-vws', 'status']
2023-02-13T12:45:30.125Z Done running command
INFO:root:Running:
applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-invsvc (VMware Inventory Service) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vpostgres (VMware Postgres)
Running:
applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-invsvc (VMware Inventory Service) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vpostgres (VMware Postgres)
INFO:root:Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()
Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()
And I still get this error on my log:
vcenter55:/tmp # tail /var/log/vmware/sso/vmware-identity-sts.log
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.validateSigningCert(TokenLifetimeRemediator.java:92)
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:66)
at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:191)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:48)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator.issueToken(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.sts.impl.STSImpl.issueToken(STSImpl.java:342)
... 48 more
[2023-02-13T12:47:05.431Z tomcat-http--24 vsphere.local 3a8ff080-93d6-4453-9bd3-f648ed90b1aa INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Mon Feb 13 12:47:05 UTC 2023, endTime=Mon Feb 13 14:27:05 UTC 2023] :: Signing certificate is not valid at Mon Feb 13 12:47:05 UTC 2023, cert validity: TimePeriod [startTime=Wed Dec 12 16:59:36 UTC 2012, endTime=Sun Dec 11 16:59:36 UTC 2022]
I don't know what to do.
What certificate is this it is complaining about? Does anyone know?
How do I solve that?
Please, if anyone know, tell me.
STS certificate is still expired. That command doesn't pull STS certificate info . I think VMware also pulled down all docs related to 5.x so cannot find any STS replacement docs .
Better to reinstall VC and get a supported version up .
Thanks @Ajay1988,
first I tried to keep the installed version, because I wasn't sure if my key would work on a hight version so I installed version 6.0.0a and the fresh install started with an error, VC do not even started and produce some errors, so I tried VCSA 6.0U3j, then I could login, I was saing that my password was wrong.
You don't now how happy I was when you told me that I could use the same key on a version 6.7. I installed VCSA 7.5 without problems and put my key. Finally I could let behind the VMS Client becouse I couldn't use the web version after flash was discontiued from browsers.
Now I am looking for a window to upgrade vSphere 6 to 6.7 at least until we can buy a licence for version 7 or 8.
Can I use my actual key on a version 7 or 8?
VC 6.0 really. These are legacy versions. STS is indeed expired . I have never used that script to replace STS on 6.0 version. The kB even says for 6.5 and above versions. But looks like the script succeeded .
Signing certificate is not valid at Mon Feb 13 12:01:36 UTC 2023
Can you also check vmdird cert if its expired > /usr/lib/vmware-vmdir/share/config/vmdircert.pem
Also renew SMS certs : https://kb.vmware.com/s/article/2120105
Hi, thanks for reply.
I just used the script to replace STS for version 6.0 knowing that was made for 6.5 above only because I could not find a KB that explains how to do it for 6.0. Even though it didn't generate any error messages, it looks like didn't have any effect because the error in /var/log/vmware/sso/vmware-identity-sts.log didn't change.
If is there a proper KB for it let me know and I can revert my snapshot and try again.
I checked the expiration time of vmdircert.pem, and it still valid, here is the return:
vcenter55:/ # openssl x509 -enddate -noout -in /usr/lib/vmware-vmdir/share/config/vmdircert.pem
notAfter=Mar 7 12:49:07 2025 GMT
I follow the steps in https://kb.vmware.com/s/article/2120105
vcenter55:/ # cd /usr/lib/vmware-vmafd/bin
vcenter55:/usr/lib/vmware-vmafd/bin # ./vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vpxd
vpxd-extension
vsphere-webclient
SMS
vcenter55:/usr/lib/vmware-vmafd/bin # ./vecs-cli entry list --store sms --text | more
Number of entries in store : 1
Alias : sms_self_signed
Entry type : Private Key
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1355418318468 (0x13b953a3684)
Signature Algorithm: sha1WithRSAEncryption
Issuer: O=VMware, CN=SMS-121213170518468
Validity
Not Before: Dec 13 17:05:18 2011 GMT
Not After : Dec 13 17:05:18 2022 GMT
Subject: O=VMware, CN=SMS-121213170518468
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (1024 bit)
Modulus:
00:91:30:cd:e3:12:c5:93:1e:a4:d6:1d:0c:e3:b3:
4d:be:b7:b9:b8:ab:a2:d3:0c:7c:6b:ce:3b:0a:f1:
db:53:c3:6b:6f:ba:44:b6:23:49:e8:61:63:a1:9b:
0e:f3:a0:9a:0a:92:ff:5b:eb:db:68:2a:c0:8d:60:
05:fe:5f:a7:79:99:7a:74:d5:4b:82:c8:c9:a3:d7:
c9:c7:4e:a2:b8:f1:4f:24:d6:13:ec:86:26:e3:e3:
fd:16:33:ac:d9:d4:a1:48:a8:f2:ef:ec:33:94:41:
6b:77:2e:49:ac:e0:b0:04:ca:f3:2b:0f:9f:0f:64:
76:09:42:ea:0c:e4:3d:92:61
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
80:2d:65:1e:74:57:1b:29:1b:7b:29:d2:d2:68:cf:27:db:6a:
ea:c3:46:84:b3:b2:9b:9b:5d:40:a1:4a:83:f0:a6:b2:c5:5e:
a7:0b:2c:ee:df:82:54:9e:33:83:31:78:29:d0:d4:9e:e1:46:
10:8c:1b:71:d0:e6:cc:32:06:57:cb:b0:28:c5:a3:c5:0f:72:
40:31:68:5e:1a:47:f6:ef:9e:22:b0:16:bb:e0:23:24:ca:04:
bb:b3:10:ae:87:bc:3b:81:6d:82:be:6a:83:ad:ae:cd:16:33:
9d:8e:16:82:8b:f6:da:9e:6a:26:ce:1d:9f:66:ba:d1:1c:c5:
17:0c
vcenter55:/usr/lib/vmware-vmafd/bin # ./vecs-cli entry delete --store sms --alias sms_self_signed
Warning: This operation will delete entry [sms_self_signed] from store [sms]
Do you wish to continue? Y/N [N]
y
Deleted entry with alias [sms_self_signed] in store [sms] successfully
vcenter55:/usr/lib/vmware-vmafd/bin # service-control --stop vmware-sps
INFO:root:Service: vmware-sps, Action: stop
Service: vmware-sps, Action: stop
2023-02-16T11:12:03.216Z Running command: ['/sbin/service', u'vmware-sps', 'stop']
2023-02-16T11:12:03.354Z Done running command
2023-02-16T11:12:03.354Z Successfully stopped service vmware-sps
vcenter55:/usr/lib/vmware-vmafd/bin # service-control --start vmware-sps
INFO:root:Service: vmware-sps, Action: start
Service: vmware-sps, Action: start
2023-02-16T11:12:18.493Z Running command: ['/sbin/chkconfig', u'vmware-sps']
2023-02-16T11:12:18.529Z Done running command
2023-02-16T11:12:18.529Z Running command: ['/sbin/service', u'vmware-sps', 'status']
2023-02-16T11:12:18.643Z Done running command
2023-02-16T11:12:18.643Z Running command: ['/sbin/chkconfig', '--force', u'vmware-sps', 'on']
2023-02-16T11:12:18.677Z Done running command
2023-02-16T11:12:18.678Z Running command: ['/sbin/service', u'vmware-sps', 'start']
2023-02-16T11:12:22.376Z Done running command
2023-02-16T11:12:22.376Z Successfully started service vmware-sps
I waited more than ten minutes and run: ./vecs-cli entry list --store sms
Number of entries in store : 0
After that I try stop and when I started it aborted and exited with an error:
2023-02-16T11:38:58.013Z Done running command
2023-02-16T11:38:58.013Z Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']
2023-02-16T11:38:58.013Z RC = 1
Stdout = vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
Waiting for the embedded database to start up: success
Executing pre-startup scripts...
vmware-vpxd: Starting vpxd by administrative request.
success
vmware-vpxd: Waiting for vpxd to start listening for requests on 8089
Waiting for vpxd to initialize: ......................................failed
failed
vmware-vpxd: vpxd failed to initialize in time.
vpxd is already starting up. Aborting the request.
Stderr =
2023-02-16T11:38:58.014Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-vpxd, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-vpxd, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
When the machine reboot, I try to login, but I still with the same problem from the beginning:
vcenter55:~ # tail /var/log/vmware/sso/vmware-identity-sts.log
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.validateSigningCert(TokenLifetimeRemediator.java:92)
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:66)
at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:191)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:48)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator$1.call(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)
at com.vmware.identity.saml.impl.TokenAuthorityImplPerformanceDecorator.issueToken(TokenAuthorityImplPerformanceDecorator.java:45)
at com.vmware.identity.sts.impl.STSImpl.issueToken(STSImpl.java:342)
... 48 more
[2023-02-16T12:37:00.526Z tomcat-http--1 vsphere.local 39eb9793-c92e-44bb-babd-b4eb2147cca4 INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Thu Feb 16 12:37:00 UTC 2023, endTime=Fri Feb 17 12:37:00 UTC 2023] :: Signing certificate is not valid at Thu Feb 16 12:37:00 UTC 2023, cert validity: TimePeriod [startTime=Wed Dec 12 16:59:36 UTC 2012, endTime=Sun Dec 11 16:59:36 UTC 2022]
I checked the certs againg and looks good:
vcenter55:~ # for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Mar 31 17:13:39 2024 GMT
[*] Store : TRUSTED_ROOTS
Alias : 3c05e1fd963e4fb30f87618ebcd549bc03c6216a
Not After : Mar 7 12:49:07 2025 GMT
Alias : d13256cbcbbee2d364c1d53eaf377b225d130ce5
Not After : Mar 31 17:13:39 2024 GMT
[*] Store : machine
Alias : machine
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vpxd
Alias : vpxd
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vpxd-extension
Alias : vpxd-extension
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : vsphere-webclient
Alias : vsphere-webclient
Not After : Mar 7 12:49:07 2025 GMT
[*] Store : SMS
What to do next?
STS certificate is still expired. That command doesn't pull STS certificate info . I think VMware also pulled down all docs related to 5.x so cannot find any STS replacement docs .
Better to reinstall VC and get a supported version up .
Hi, I will try reinstall, two questions:
Thanks @Ajay1988,
first I tried to keep the installed version, because I wasn't sure if my key would work on a hight version so I installed version 6.0.0a and the fresh install started with an error, VC do not even started and produce some errors, so I tried VCSA 6.0U3j, then I could login, I was saing that my password was wrong.
You don't now how happy I was when you told me that I could use the same key on a version 6.7. I installed VCSA 7.5 without problems and put my key. Finally I could let behind the VMS Client becouse I couldn't use the web version after flash was discontiued from browsers.
Now I am looking for a window to upgrade vSphere 6 to 6.7 at least until we can buy a licence for version 7 or 8.
Can I use my actual key on a version 7 or 8?
6.x license would wok on 6.0/6.5/6.7 . For 7.0 or 8.0 you need new license .
I think if your license is still valid you can upgrade license leys from portal > https://kb.vmware.com/s/article/81665
Better to have word with VMware licensing Team .