VMware Cloud Community
Pollard
Enthusiast
Enthusiast

VCenter 6.7 Quit working

Below are the logs I pulled support package wont download. . 

Hostname:
vcenter
Type:
vCenter Server with an embedded Platform Services Controller
Product:
VMware vCenter Server Appliance
Version:
6.7.0.21000
Build number
11726888

 

 vmdird.service - LSB: Start and Stop vmdir
   Loaded: loaded (/etc/rc.d/init.d/vmdird; bad; vendor preset: enabled)
  Drop-In: /etc/systemd/system/vmdird.service.d
           └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf
   Active: failed (Result: exit-code) since Fri 2022-05-27 09:20:56 EDT; 1min 11s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 23980 ExecStart=/etc/rc.d/init.d/vmdird start (code=exited, status=1/FAILURE)

May 27 09:20:55 vcenter systemd[1]: Starting LSB: Start and Stop vmdir...
May 27 09:20:56 vcenter vmdird[23980]: Starting VMware Directory Servicefailed
May 27 09:20:56 vcenter systemd[1]: vmdird.service: Control process exited, code=exited status=1
May 27 09:20:56 vcenter systemd[1]: Failed to start LSB: Start and Stop vmdir.
May 27 09:20:56 vcenter systemd[1]: vmdird.service: Unit entered failed state.
May 27 09:20:56 vcenter systemd[1]: vmdird.service: Failed with result 'exit-code'.

 

 

 

 

 

 

May 27 10:01:01 vcenter run-parts[37945]: (/etc/cron.hourly) starting 0anacron
May 27 10:01:01 vcenter run-parts[37935][37951]: (/etc/cron.hourly) finished 0anacron
May 27 10:01:01 vcenter run-parts[37953]: (/etc/cron.hourly) starting netdumper
May 27 10:01:01 vcenter run-parts[37935][37959]: (/etc/cron.hourly) finished netdumper
May 27 10:01:03 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/ldap.c:219]
May 27 10:01:03 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/rootfetch.c:257]
May 27 10:01:03 vcenter vmafdd[1493]: t@140367794624256: Failed to update trusted roots. Error [9127]
May 27 10:01:19 vcenter vmcad[1578]: t@140335749359360: error code: 0xffffffff
May 27 10:01:19 vcenter vmcad[1578]: t@140335749359360: Failed to update root certs due to error [4294967295]
May 27 10:02:01 vcenter CROND[38073]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup
May 27 10:02:01 vcenter CROND[38074]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/nul
May 27 10:02:03 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 1.
May 27 10:02:08 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 2.
May 27 10:02:13 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 3.
May 27 10:02:18 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/ldap.c:219]
May 27 10:02:18 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/rootfetch.c:257]
May 27 10:02:18 vcenter vmafdd[1493]: t@140367794624256: Failed to update trusted roots. Error [9127]
May 27 10:02:19 vcenter vmcad[1578]: t@140335749359360: error code: 0xffffffff
May 27 10:02:19 vcenter vmcad[1578]: t@140335749359360: Failed to update root certs due to error [4294967295]
May 27 10:02:53 vcenter vmon[1920]: Executing service batch op API_HEALTH. IgnoreFail=1, service count=9
May 27 10:02:53 vcenter vmon[1920]: <vapi-endpoint> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <vmware-vpostgres> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <rhttpproxy-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-rhttppr
May 27 10:02:53 vcenter vmon[1920]: <vmware-postgres-archiver> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <vpxd-svcs> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <vpxd> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <sps> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <rbd> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:53 vcenter vmon[1920]: <pschealth> Skip service health check. State STOPPED, Curr request 0
May 27 10:02:54 vcenter vmon[1920]: Successfully executed service batch operation API_HEALTH.
May 27 10:03:01 vcenter CROND[38199]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/nul
May 27 10:03:01 vcenter CROND[38200]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup
May 27 10:03:18 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 1.
May 27 10:03:19 vcenter vmcad[1578]: t@140335749359360: error code: 0xffffffff
May 27 10:03:19 vcenter vmcad[1578]: t@140335749359360: Failed to update root certs due to error [4294967295]
May 27 10:03:23 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 2.
May 27 10:03:28 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 3.
May 27 10:03:33 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/ldap.c:219]
May 27 10:03:33 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/rootfetch.c:257]
May 27 10:03:33 vcenter vmafdd[1493]: t@140367794624256: Failed to update trusted roots. Error [9127]
May 27 10:04:01 vcenter CROND[38323]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/nul
May 27 10:04:01 vcenter CROND[38324]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup
May 27 10:04:19 vcenter vmcad[1578]: t@140335749359360: error code: 0xffffffff
May 27 10:04:19 vcenter vmcad[1578]: t@140335749359360: Failed to update root certs due to error [4294967295]

 

 

 

19 Replies
Pollard
Enthusiast
Enthusiast

Do I need to disassociate these esxi servers somehow or can I just spin up a new vcenter vm?

Reply
0 Kudos
compdigit44
Enthusiast
Enthusiast

I see the error below present. Have you check to see if the certificates on the VCSA expired? 

https://kb.vmware.com/s/article/6817

 

"Failed to update Trust Roots"

 

Ajay1988
Expert
Expert

Do you have an External PSC ?
Do you have more than 1 VC in linked mode ? 

Check /var/log/vmware/vmdird/vmdird-syslog.log to see what it is failing or share the last log ? 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Pollard
Enthusiast
Enthusiast

Attempting to reset the cert with certificate-manager it wants the administrator@vsphere.local password. However the password I know is correct isn't working anymore. Attempted to reset administrator@vsphere.local with the vdcadmintool I get vmdirforceresetpassword failed (9126). Kind of stuck here cant win 😩 can I just create a new vcenter and relink the server's to the new one?

Ajay1988
Expert
Expert

That's mostly because ur vmdird service itself is down. Please help us with the earlier queries

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

What's the easiest way to pull the log out? I opened it with vim in my ssh session but its rather large and couldn't copy and paste it all. 

Reply
0 Kudos
compdigit44
Enthusiast
Enthusiast

Have you tried connecting to the VCSA using WinSCP to get the logs?

 

https://www.johnborhek.com/vmware/vmware-vsphere/vmware-vcenter/using-winscp-vmware-vcenter-server-a...

 

Pollard
Enthusiast
Enthusiast

That tools awesome thanks!

 

Here is my log 

Them Modify Entry line just repeats above

 

2022-05-20T18:11:34.079018+00:00 info vmdird  t@140688826033920: Modify Entry (CN=E0EB9FBD0654BEEB41B7DF28B85E6C4C0450C980,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local)(from 127.0.0.1)(by vcenter.mydomainwashere.com@vsphere.local)(via Ext)(USN 11489,0)
2022-05-20T18:11:39.634268+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: making database snapshot with file size 21Mb; will take approximate 1 seconds; 1 updates occurred since last snapshot.
2022-05-20T18:11:39.666987+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: completed making snapshot with file size 21Mb in 1 seconds; data transfer rate: 21.0MB/sec, db last tid: 22985
2022-05-20T23:11:34.061464+00:00 info vmdird  t@140688809248512: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----
MIICIjCCAQoCAQEwDQYJKoZIhvcNAQELBQAwgaUxCzAJBgNVBAMMAkNBMRcwFQYK
CZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkWBWxvY2FsMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEnMCUGA1UECgwedmNlbnRlci5j
aGVzYXBlYWtlY29udHJvbHMuY29tMRswGQYD)
2022-05-20T23:11:34.063270+00:00 info vmdird  t@140688809248512: Modify Entry (CN=E0EB9FBD0654BEEB41B7DF28B85E6C4C0450C980,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local)(from 127.0.0.1)(by vcenter.mydomainwashere.com@vsphere.local)(via Ext)(USN 11490,0)
2022-05-20T23:11:43.015196+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: making database snapshot with file size 21Mb; will take approximate 1 seconds; 1 updates occurred since last snapshot.
2022-05-20T23:11:43.031573+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: completed making snapshot with file size 21Mb in 1 seconds; data transfer rate: 21.0MB/sec, db last tid: 22987
2022-05-21T04:11:34.063095+00:00 info vmdird  t@140688809248512: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----
MIICIjCCAQoCAQEwDQYJKoZIhvcNAQELBQAwgaUxCzAJBgNVBAMMAkNBMRcwFQYK
CZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkWBWxvY2FsMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEnMCUGA1UECgwedmNlbnRlci5j
aGVzYXBlYWtlY29udHJvbHMuY29tMRswGQYD)
2022-05-21T04:11:34.072729+00:00 info vmdird  t@140688809248512: Modify Entry (CN=E0EB9FBD0654BEEB41B7DF28B85E6C4C0450C980,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local)(from 127.0.0.1)(by vcenter.mydomainwashere.com@vsphere.local)(via Ext)(USN 11491,0)
2022-05-21T04:11:42.701553+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: making database snapshot with file size 21Mb; will take approximate 1 seconds; 1 updates occurred since last snapshot.
2022-05-21T04:11:42.741278+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: completed making snapshot with file size 21Mb in 1 seconds; data transfer rate: 21.0MB/sec, db last tid: 22989
2022-05-21T09:11:34.071117+00:00 info vmdird  t@140689388050176: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----
MIICIjCCAQoCAQEwDQYJKoZIhvcNAQELBQAwgaUxCzAJBgNVBAMMAkNBMRcwFQYK
CZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkWBWxvY2FsMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEnMCUGA1UECgwedmNlbnRlci5j
aGVzYXBlYWtlY29udHJvbHMuY29tMRswGQYD)
2022-05-21T09:11:34.080934+00:00 info vmdird  t@140689388050176: Modify Entry (CN=E0EB9FBD0654BEEB41B7DF28B85E6C4C0450C980,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local)(from 127.0.0.1)(by vcenter.mydomainwashere.com@vsphere.local)(via Ext)(USN 11492,0)
2022-05-21T09:11:42.449940+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: making database snapshot with file size 21Mb; will take approximate 1 seconds; 1 updates occurred since last snapshot.
2022-05-21T09:11:42.491629+00:00 info vmdird  t@140689539053312: _VmDirCpMdbFile: completed making snapshot with file size 21Mb in 1 seconds; data transfer rate: 21.0MB/sec, db last tid: 22991
2022-05-23T13:17:25.563294+00:00 info vmdird  t@140708393096960: VmDir State (1)
2022-05-23T13:17:25.563884+00:00 info vmdird  t@140708393096960: Lotus Vmdird: starting...
2022-05-23T13:17:25.647760+00:00 info vmdird  t@140708393096960: Openssl Ctx disable protocol (TLSv1)
2022-05-23T13:17:25.648279+00:00 info vmdird  t@140708393096960: Openssl Ctx disable protocol (TLSv1.1)
2022-05-23T13:17:25.650678+00:00 info vmdird  t@140708393096960: SSL cipher list (!aNULL:kECDH+AES:ECDH+AES:RSA+AES:@STRENGTH).
2022-05-23T13:17:25.665809+00:00 info vmdird  t@140708393096960: Acquired SSL Cert from VECS
2022-05-23T13:17:25.668613+00:00 info vmdird  t@140708393096960: SASL2PATH=/opt/likewise/lib64/sasl2:/usr/lib/vmware-vmdir/lib64/sasl2
2022-05-23T13:17:25.673182+00:00 info vmdird  t@140708393096960: Use default max-database-size 21474836480
2022-05-23T13:17:25.729239+00:00 info vmdird  t@140708393096960: _VmDirOpenDbEnv: use default database file; its tid (22993) >= snapshot tid (22991)
2022-05-23T13:17:25.777155+00:00 info vmdird  t@140708393096960: mdb stats: last_pgno 5271, max_pgs 5242880
2022-05-23T13:17:25.799556+00:00 info vmdird  t@140708393096960: New schema instance (0x2c0d5a0)
2022-05-23T13:17:25.828919+00:00 info vmdird  t@140708393096960: VmDirReadSubSchemaSubEntry subschema subentry not found (9703)
2022-05-23T13:17:25.829356+00:00 info vmdird  t@140708393096960: New schema instance (0x2c0eba0)
2022-05-23T13:17:25.847734+00:00 info vmdird  t@140708393096960: Indexing Progress: Attribute = attributeMetaData, Status = COMPLETE
2022-05-23T13:17:27.612640+00:00 info vmdird  Starting VMware Directory Servicefailed
2022-05-23T13:17:27.660941+00:00 info vmdird  t@139701632558848: VmDir State (1)
2022-05-23T13:17:27.661592+00:00 info vmdird  t@139701632558848: Lotus Vmdird: starting...
2022-05-23T13:17:27.719747+00:00 info vmdird  t@139701632558848: Openssl Ctx disable protocol (TLSv1)
2022-05-23T13:17:27.720298+00:00 info vmdird  t@139701632558848: Openssl Ctx disable protocol (TLSv1.1)
2022-05-23T13:17:27.722410+00:00 info vmdird  t@139701632558848: SSL cipher list (!aNULL:kECDH+AES:ECDH+AES:RSA+AES:@STRENGTH).
2022-05-23T13:17:27.733948+00:00 info vmdird  t@139701632558848: Acquired SSL Cert from VECS
2022-05-23T13:17:27.736056+00:00 info vmdird  t@139701632558848: SASL2PATH=/opt/likewise/lib64/sasl2:/usr/lib/vmware-vmdir/lib64/sasl2
2022-05-23T13:17:27.739142+00:00 info vmdird  t@139701632558848: Use default max-database-size 21474836480
2022-05-23T13:17:27.740858+00:00 info vmdird  t@139701632558848: _VmDirOpenDbEnv: use default database file; its tid (22993) >= snapshot tid (22991)
2022-05-23T13:17:27.741420+00:00 info vmdird  t@139701632558848: mdb stats: last_pgno 5271, max_pgs 5242880
2022-05-23T13:17:27.745493+00:00 info vmdird  t@139701632558848: New schema instance (0x29ef5a0)
2022-05-23T13:17:27.746044+00:00 info vmdird  t@139701632558848: VmDirReadSubSchemaSubEntry subschema subentry not found (9703)
2022-05-23T13:17:27.746881+00:00 info vmdird  t@139701632558848: New schema instance (0x29f0ba0)
2022-05-23T13:17:27.747553+00:00 info vmdird  t@139701632558848: Indexing Progress: Attribute = attributeMetaData, Status = COMPLETE
2022-05-23T13:17:27.861805+00:00 info vmdird  t@140480272967424: VmDir State (1)
2022-05-23T13:17:27.862322+00:00 info vmdird  t@140480272967424: Lotus Vmdird: starting...
2022-05-23T13:17:27.920531+00:00 info vmdird  t@140480272967424: Openssl Ctx disable protocol (TLSv1)
2022-05-23T13:17:27.920965+00:00 info vmdird  t@140480272967424: Openssl Ctx disable protocol (TLSv1.1)
2022-05-23T13:17:27.922934+00:00 info vmdird  t@140480272967424: SSL cipher list (!aNULL:kECDH+AES:ECDH+AES:RSA+AES:@STRENGTH).
2022-05-23T13:17:27.934641+00:00 info vmdird  t@140480272967424: Acquired SSL Cert from VECS
2022-05-23T13:17:27.936748+00:00 info vmdird  t@140480272967424: SASL2PATH=/opt/likewise/lib64/sasl2:/usr/lib/vmware-vmdir/lib64/sasl2
2022-05-23T13:17:27.939540+00:00 info vmdird  t@140480272967424: Use default max-database-size 21474836480
2022-05-23T13:17:27.940084+00:00 info vmdird  t@140480272967424: _VmDirOpenDbEnv: use default database file; its tid (22993) >= snapshot tid (22991)
2022-05-23T13:17:27.940484+00:00 info vmdird  t@140480272967424: mdb stats: last_pgno 5271, max_pgs 5242880
2022-05-23T13:17:27.943819+00:00 info vmdird  t@140480272967424: New schema instance (0x2f1d5a0)
2022-05-23T13:17:27.944321+00:00 info vmdird  t@140480272967424: VmDirReadSubSchemaSubEntry subschema subentry not found (9703)
2022-05-23T13:17:27.944731+00:00 info vmdird  t@140480272967424: New schema instance (0x2f1eba0)
2022-05-23T13:17:27.945290+00:00 info vmdird  t@140480272967424: Indexing Progress: Attribute = attributeMetaData, Status = COMPLETE

 

Reply
0 Kudos
compdigit44
Enthusiast
Enthusiast

I see the VMDir service is not running. Please run the following command:  service-control –status which will list the current status of all services. Next try to start vmdir using service-control –start vmdir

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

Just hard to restore a snapshot quit letting me login at all. Can I just re-deploy a new vcenter and trash this one?

 

root@vcenter [ ~ ]# service-control --status
Stopped:
pschealth vmcam vmdird vmdnsd vmonapi vmware-cm vmware-content-library vmware-i magebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-postgres-archi ver vmware-rbd-watchdog vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vm ware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpostgres vmware-vpxd vmw are-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps
Running:
applmgmt lwsmd vmafdd vmcad vmware-analytics vmware-cis-license vmware-eam vmwa re-pod vmware-rhttpproxy vmware-sts-idmd vmware-vmon vsphere-client vsphere-ui


root@vcenter [ ~ ]# service-control -–start vmdir
usage: service-control [-h] [--start] [--stop] [--restart] [--status] [--list]
                       [--list-services] [--list-dependencies]
                       [--vmon-profile VMON_PROFILE] [--all] [--ignore]
                       [--verbose]
                       ...
service-control: error: unrecognized arguments: -–start

service-control --start vmdird
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmdird...
2022-06-11T12:37:45.395Z  RC = 1
Stdout =
Stderr = Job for vmdird.service failed because the control process exited with e                                                                                                    rror code. See "systemctl status vmdird.service" and "journalctl -xe" for detail                                                                                                    s.

2022-06-11T12:37:45.395Z  {
    "detail": [
        {
            "args": [
                "Stderr: Job for vmdird.service failed because the control proce                                                                                                    ss exited with error code. See \"systemctl status vmdird.service\" and \"journal                                                                                                    ctl -xe\" for details.\n"
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'S                                                                                                    tderr: Job for vmdird.service failed because the control process exited with err                                                                                                    or code. See \"systemctl status vmdird.service\" and \"journalctl -xe\" for deta                                                                                                    ils.\n'",
            "translatable": "An error occurred while invoking external command :                                                                                                     '%(0)s'"
        }
    ],
    "problemId": null,
    "componentKey": null,
    "resolution": null
}
Error executing start on service vmdird. Details {
    "detail": [
        {
            "args": [
                "vmdird"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmdird'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "componentKey": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "args": [
                "vmdird"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmdird'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "componentKey": null,
    "resolution": null
root@vcenter [ ~ ]# systemctl status vmdird.service
● vmdird.service - LSB: Start and Stop vmdir
   Loaded: loaded (/etc/rc.d/init.d/vmdird; bad; vendor preset: enabled)
  Drop-In: /etc/systemd/system/vmdird.service.d
           └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf
   Active: failed (Result: exit-code) since Sat 2022-06-11 08:37:45 EDT; 1min 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 57672 ExecStart=/etc/rc.d/init.d/vmdird start (code=exited, status=1/FAILURE)

Jun 11 08:37:44 vcenter systemd[1]: Starting LSB: Start and Stop vmdir...
Jun 11 08:37:45 vcenter vmdird[57672]: Starting VMware Directory Servicefailed
Jun 11 08:37:45 vcenter systemd[1]: vmdird.service: Control process exited, code=exited status=1
Jun 11 08:37:45 vcenter systemd[1]: Failed to start LSB: Start and Stop vmdir.
Jun 11 08:37:45 vcenter systemd[1]: vmdird.service: Unit entered failed state.
Jun 11 08:37:45 vcenter systemd[1]: vmdird.service: Failed with result 'exit-code'.
root@vcenter [ ~ ]# journalctl -xe
Jun 11 08:38:21 vcenter vmon[1920]:         "id": "install.default.failure.resolution"
Jun 11 08:38:21 vcenter vmon[1920]:     }
Jun 11 08:38:21 vcenter vmon[1920]: }
Jun 11 08:38:21 vcenter vmon[1920]:
Jun 11 08:38:22 vcenter vmon[1920]: <cm> Re-check service health since it is still initializing.
Jun 11 08:38:24 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 1.
Jun 11 08:38:26 vcenter vmon[1920]: <vapi-endpoint-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vapi-endpoint -u /vapiendpoint/health
Jun 11 08:38:27 vcenter vmon[1920]: <vapi-endpoint> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:12346/vapiendpoint/health.
Jun 11 08:38:27 vcenter vmon[1920]:
Jun 11 08:38:27 vcenter vmon[1920]: <vapi-endpoint> Re-check service health since it is still initializing.
Jun 11 08:38:29 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 2.
Jun 11 08:38:30 vcenter vmon[1920]: <cm-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-cm/bin/cmhealth.py
Jun 11 08:38:33 vcenter vmon[1920]: <cm> Service api-health command's stderr: ERROR:root:CM search failed. ProductId=com.vmware.cis, TypeId=cs.identity
Jun 11 08:38:33 vcenter vmon[1920]:
Jun 11 08:38:33 vcenter vmon[1920]: <cm> Service api-health command's stderr: ERROR:root:Error connecting to CM: {
Jun 11 08:38:33 vcenter vmon[1920]:     "problemId": null,
Jun 11 08:38:33 vcenter vmon[1920]:     "detail": [
Jun 11 08:38:33 vcenter vmon[1920]:         {
Jun 11 08:38:33 vcenter vmon[1920]:             "args": [
Jun 11 08:38:33 vcenter vmon[1920]:                 "CM search failed. ProductId=com.vmware.cis, TypeId=cs.identity:[Errno 111] Connection refused"
Jun 11 08:38:33 vcenter vmon[1920]:             ],
Jun 11 08:38:33 vcenter vmon[1920]:             "translatable": "Encountered an internal error.\n\n%(0)s",
Jun 11 08:38:33 vcenter vmon[1920]:             "id": "install.ciscommon.internal.error",
Jun 11 08:38:33 vcenter vmon[1920]:             "localized": "Encountered an internal error.\n\nCM search failed. ProductId=com.vmware.cis, TypeId=cs.identity:[Errno 111] Connectio
Jun 11 08:38:33 vcenter vmon[1920]:         }
Jun 11 08:38:33 vcenter vmon[1920]:     ],
Jun 11 08:38:33 vcenter vmon[1920]:     "resolution": {
Jun 11 08:38:33 vcenter vmon[1920]:         "translatable": "This is an unrecoverable error, please retry install. If you encounter this error again, please search for these sympto
Jun 11 08:38:33 vcenter vmon[1920]:         "id": "install.default.failure.resolution",
Jun 11 08:38:33 vcenter vmon[1920]:         "localized": "This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms
Jun 11 08:38:33 vcenter vmon[1920]:     },
Jun 11 08:38:33 vcenter vmon[1920]:     "componentKey": null
Jun 11 08:38:33 vcenter vmon[1920]: }
Jun 11 08:38:33 vcenter vmon[1920]:
Jun 11 08:38:33 vcenter vmon[1920]: <cm> Re-check service health since it is still initializing.
Jun 11 08:38:34 vcenter vmafdd[1493]: t@140367794624256: [VmAfdLDAPConnect] Vmdir server is down. Attempting retry 3.
Jun 11 08:38:35 vcenter vmon[1920]: <vapi-endpoint-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vapi-endpoint -u /vapiendpoint/health
Jun 11 08:38:36 vcenter vmon[1920]: <vapi-endpoint> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:12346/vapiendpoint/health.
Jun 11 08:38:36 vcenter vmon[1920]:
Jun 11 08:38:36 vcenter vmon[1920]: <vapi-endpoint> Re-check service health since it is still initializing.
Jun 11 08:38:39 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/ldap.c:219]
Jun 11 08:38:39 vcenter vmafdd[1493]: t@140367794624256: [Error - 9127, ../../../server/vmafd/rootfetch.c:257]
Jun 11 08:38:39 vcenter vmafdd[1493]: t@140367794624256: Failed to update trusted roots. Error [9127]
Jun 11 08:38:40 vcenter vmcad[1578]: t@140335749359360: error code: 0xffffffff
Jun 11 08:38:40 vcenter vmcad[1578]: t@140335749359360: Failed to update root certs due to error [4294967295]
Jun 11 08:38:41 vcenter vmon[1920]: <cm-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-cm/bin/cmhealth.py
Jun 11 08:38:43 vcenter vmon[1920]: <cm> Service exited. Exit code 0
Jun 11 08:38:43 vcenter vmon[1920]: <cm-healthcmd> Cancelling execution of pid 57926
Jun 11 08:38:43 vcenter vmon[1920]: A non tracked pid 57926 terminated.
Jun 11 08:38:44 vcenter vmon[1920]: <vapi-endpoint-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vapi-endpoint -u /vapiendpoint/health
Jun 11 08:38:44 vcenter vmon[1920]: <vapi-endpoint> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:12346/vapiendpoint/health.
Jun 11 08:38:44 vcenter vmon[1920]:
Jun 11 08:38:44 vcenter vmon[1920]: <vapi-endpoint> Re-check service health since it is still initializing.
Jun 11 08:38:52 vcenter vmon[1920]: <vapi-endpoint-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vapi-endpoint -u /vapiendpoint/health
Jun 11 08:38:53 vcenter vmon[1920]: <vapi-endpoint> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:12346/vapiendpoint/health.
Jun 11 08:38:53 vcenter vmon[1920]:
Jun 11 08:38:53 vcenter vmon[1920]: <vapi-endpoint> Re-check service health since it is still initializing.

 

 

Reply
0 Kudos
compdigit44
Enthusiast
Enthusiast

Do you have any backups of your vcsa? I would suggest contacting Vmware support so they can advise you on how to best proceed so you do not lose any data  / configs on your vcsa.

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

Nope unfortunately. There's not a whole lot configured on there what would I loose just making a new one? 

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

I actually just found an older backup I had from veeam. Not as many issues stopped on starting vAPI endpoint but vmdird started no problem. I was able to reset the administrator@vsphere.local account password also which still says it's incorrect when trying to run the certificate-manager to update certs.

 

 

 

2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2022-06-14T15:34:10UTC and 2022-06-14T15:34:10UTC.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from Lookup Service.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2022-06-14T15:34:10UTC and 2022-06-14T15:34:10UTC.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | DefaultStateManager            | lock
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.
2022-06-14T15:34:10.861Z | INFO  | state-manager1            | DefaultStateManager            | unlock
2022-06-14T15:34:15.862Z | INFO  | state-manager1            | DefaultStateManager            | Start initializing API Endpoint runtime state. State check interval 240 sec.
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | DefaultStateManager            | Creating runtime state context.
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | DefaultStateManager            | Invoking log4j-config-builder
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | Log4jConfigBuilder             | Watching log4j configuration at /etc/vmware-vapi/endpoint-log4j.properties for changes.
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | DefaultStateManager            | Invoking init-configuration-builder
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | DefaultStateManager            | Invoking properties-builder
2022-06-14T15:34:15.863Z | INFO  | state-manager1            | LocalPropertiesFileStateBuilder | Will be configuring from following locations: file:/etc/vmware-vapi/endpoint.properties
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | LocalPropertiesFileStateBuilder | Resource URL [file:/etc/vmware-vapi/endpoint.properties] was parsed successfully into set of properties.
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | Invoking feature-state-switch-builder
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | Invoking state-manager
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | State check interval property found 240.
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | State check delay property found 240.
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | Invoking keystore-builder
2022-06-14T15:34:15.864Z | INFO  | state-manager1            | DefaultStateManager            | Invoking endpoint-settings-builder
2022-06-14T15:34:15.867Z | INFO  | state-manager1            | DefaultStateManager            | Invoking lookup-service-client-builder
2022-06-14T15:34:15.867Z | INFO  | state-manager1            | DefaultStateManager            | Invoking http-settings-builder
2022-06-14T15:34:15.867Z | INFO  | state-manager1            | HttpSettingsBuilder            | HTTP Endpoint default=http://127.0.0.1, ::1:12346
2022-06-14T15:34:15.867Z | INFO  | state-manager1            | DefaultStateManager            | Invoking static-configuration-utilities
2022-06-14T15:34:15.867Z | INFO  | state-manager1            | DefaultStateManager            | Invoking configuration-utilities
2022-06-14T15:34:15.883Z | INFO  | state-manager1            | DefaultStateManager            | Invoking http-server
2022-06-14T15:34:15.883Z | INFO  | state-manager1            | BaseServerBuilder              | Server instances already started. Do nothing.
2022-06-14T15:34:15.883Z | INFO  | state-manager1            | DefaultStateManager            | Invoking cis-sso-settings-builder
2022-06-14T15:34:15.889Z | ERROR | state-manager1            | SsoSettingsBuilder             | Failded to retrieve SSO settings.
com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:292)
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:206)
	at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
	at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
	at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
	at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
	at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
	at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
	at com.sun.proxy.$Proxy38.list(Unknown Source)
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:288)
	... 13 more
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from Lookup Service.
2022-06-14T15:34:15.889Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
	at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)
	at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
	at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
	at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:292)
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:206)
	at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
	at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
	... 10 more
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
	at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
	at com.sun.proxy.$Proxy38.list(Unknown Source)
	at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:288)
	... 13 more
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2022-06-14T15:34:15UTC and 2022-06-14T15:34:15UTC.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from Lookup Service.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2022-06-14T15:34:15UTC and 2022-06-14T15:34:15UTC.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | DefaultStateManager            | lock
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.
2022-06-14T15:34:15.889Z | INFO  | state-manager1            | DefaultStateManager            | unlock

 

Reply
0 Kudos
compdigit44
Enthusiast
Enthusiast

Thank god for a backup!!! So you were able to log in to your VCSA correct? Are you able to start any stopped services?

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

No still throwing "503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb63400bd10] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)" when I try to get into the VCSA webserver. When I attempt to renew the certs it acts like my administrator@vsphere.local password is incorrect right after I reset it. Is there anyway to fake the time out so it thinks the certs are still valid? I attempted that with no luck. Ready to just dump this thing I think it's todo with my sso I setup with my ldap. 

 

I can get here though but everything wont start

Pollard_0-1655230726482.png

 

 

 

 

Reply
0 Kudos
BigMike23
Enthusiast
Enthusiast

Can you Putty into the VCSA and see if all the services are started. I had an issue with one service that didn't start and once I started the service manually everything worked great. I see your going down the certificate path so maybe it isn;t services.

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

They all wont start even manually always some kind of error

 

Reply
0 Kudos
BigMike23
Enthusiast
Enthusiast

If you restored the backup were you able to get it working again?

Reply
0 Kudos
Pollard
Enthusiast
Enthusiast

No that one was full of issues also. I didn't have much configured on there I deleted it and made a new one. Should have done that sooner I was only using vcenter for some of the VM moving features. 

Reply
0 Kudos