Reply to Message

Thanks to all.

We had the same problem with autodeploy stopping with "Fatal error: 15". Finding this page I checked rbd-cgi.log and vmcad-syslog.log ...

root@vc1 [ ~ ]# cat /var/log/vmware/rbd/rbd-cgi.log | grep -E "rror|ERROR"
2021-03-03T09:21:23.536 [6150]ERROR:vmcacertutil:Could not generate certificates for: 10.2.2.1
out: b'Error: 5, VMCAGetSignedCertificatePrivate() failedError Code : 5\nMessage :UNKNOWN\n'
2021-03-03T09:21:23.553 [6150]ERROR:pluginmaster:exception:rbdplugins.sslcert.vmwWaiterTgz -- 0:b'Error: 5, VMCAGetSignedCertificatePrivate() failedError Code : 5\nMessage :UNKNOWN\n':b"Operation Failed: exception <class 'vmca.vmca_exception'> not a BaseException subclass"
Exception: 0:b'Error: 5, VMCAGetSignedCertificatePrivate() failedError Code : 5\nMessage :UNKNOWN\n':b"Operation Failed: exception <class 'vmca.vmca_exception'> not a BaseException subclass"
2021-03-03T09:21:23.554 [6150]WARNING:waitertgz:retrying waiter tgz because of rc: [None, None, None], except: [Exception('0:b\'Error: 5, VMCAGetSignedCertificatePrivate() failedError Code : 5\\nMessage :UNKNOWN\\n\':b"Operation Failed: exception <class \'vmca.vmca_exception\'> not a BaseException subclass"',)]

root@vc1 [ ~ ]# tail /var/log/vmware/vmcad/vmcad-syslog.log
2021-03-03T09:34:52.765706+01:00 info vmcad t@140664742344338: VMCACheckAccessKrb: Authenticated user waiter-a67cf497-3462-48bb-868d-866c983aa484@vsphere.local
2021-03-03T09:34:52.770946+01:00 info vmcad t@140664742344338: Checking upn: cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local against CA admin group: waiter-a67cf497-3462-48bb-868d-866c983aa484@vsphere.local
2021-03-03T09:34:52.771150+01:00 warning vmcad t@140664742344338: error code: 0x00000005
2021-03-03T09:34:52.771329+01:00 warning vmcad t@140664742344338: error code: 0x00000005
2021-03-03T09:34:52.771497+01:00 warning vmcad t@140664742344338: error code: 0x00000005

Using dir-cli in vCenter shell I checked users in CAAdmins group and found out that two waiter accounts are there but the one from vmcad-syslog.log (waiter-a67cf497-3462-48bb-868d-866c983aa484) is missing.

root@vc1 [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli group list --name CAAdmins
Enter password for administrator@vsphere.local:
cn=Administrator,cn=Users,dc=vsphere,dc=local
cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local
cn=DCClients,cn=Builtin,dc=vsphere,dc=local
CN=waiter 0af35be1-fc4b-427a-8181-1a25dbaa1270,cn=users,dc=vsphere,dc=local
CN=waiter 5a882302-063f-4bb1-9eac-6cbd662d5130,cn=users,dc=vsphere,dc=local

I looked for this particular user in other user groups (Users, Administrators ...) hoping I will find it somewhere but I did not. So I tried to create it and found out that it actually exists:

root@vc1 [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli user create --account waiter-a67cf497-3462-48bb-868d-866c983aa484 --first-name waiter --last-name a67cf497-3462-48bb-868d-866c983aa484 --user-password 'testpass'
Enter password for administrator@vsphere.local:
dir-cli failed. Error 9706: Possible errors:
LDAP error: Already exists
Win Error: Operation failed with error ERROR_TOO_MANY_NAMES (68)

Great, because I had no idea what password to give to the new user. Now I just had to add existing user to CAAdmins group:

root@vc1 [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name CAAdmins --add waiter-a67cf497-3462-48bb-868d-866c983aa484

Adding user to CAAdmins group was successful and Autodeploy started working immediately.