a_sand
Contributor
Contributor

Upgrade vCenter Server Appliance (VCSA) from 6.5 (6.5.0.23200) to 6.7U2 failed with "Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store."

I have VCSA 6.5 and try to upgrade it to 6.7U2

During migration prerequsites check phase I have error "Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store. Make sure that the vmafd service is reachable and started before continuing."

I have check cert store on new VCSA installation by:

service-control --status certificatemanagement

The service is running

Then I check cert store by "vecs-cli store list" and get "Unable to connect to vmafd service"

Then I check syslog and found

Error opening Certificate /etc/vmware/vmware-vmafd/machine-ssl.crt

140367761127064:error:02001002:system library:fopen:No such file or directory:bss_file.c:406:fopen('/etc/vmware/vmware-vmafd/machine-ssl.crt','r')

140367761127064:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:408:

unable to load certificate

Mon, 29 Apr 2019 09:44:24 +0000 [ERROR] CA file /etc/ssl/certs/.0 not found

How I can generate machine-ssl.crt ?

PS I cannot use certificate-manager as I cannot authenticate as administrator@vsphere.local at this point of time.

UPD: I have copied /etc/vmware/vmware-vmafd/* from old VCSA but without success. "vecs-cli store list" still cannot connect to vmafd service

Сообщение отредактировано: a_sand

0 Kudos
17 Replies
Raj1988
Enthusiast
Enthusiast

Make sure all the required services on source is up and running fine.

Check VMware Knowledge Base   and validate if you are trying any back-in-time upgrade .

Regards,

AJ

0 Kudos
birend1988
Hot Shot
Hot Shot

please share the service-control --status --all output. It seems vmafdd service is not running.

VCIX, NCAP
0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

Got the exact same error when trying to update from 6.5 U2c to 6.7.0-13643870.

vmafd service is running.

vmafdd log gives some errors:

19-06-12T19:42:30.112292+02:00 err vmafdd  t@140302530627328: [Error - 5, ../../../server/vmafd/authutil.c:471]

19-06-12T19:42:30.126924+02:00 notice vmafdd  t@140302530627328: User root changed permission of Store with ID: 9

Permission read  was granted to user vpxd

19-06-12T19:43:08.954684+02:00 err vmafdd  t@140302507448064: [Error - 183, ../../../server/vmafd/vecsserviceapi.c:189]

19-06-12T19:43:08.954962+02:00 err vmafdd  t@140302507448064: [Error - 183, ../../../server/vmafd/authservice.c:36]

19-06-12T19:43:08.955093+02:00 err vmafdd  t@140302507448064: ERROR! [VecsIpcCreateCertStore] is returning  [183]

19-06-12T19:43:09.066357+02:00 err vmafdd  t@140302507448064: [Error - 5, ../../../server/vmafd/authutil.c:471]

19-06-12T19:43:09.083582+02:00 notice vmafdd  t@140302507448064: User root changed permission of Store with ID: 8

Permission read  was granted to user vpxd

19-06-12T19:46:23.254544+02:00 err vmafdd  t@140302499055360: [Error - 4312, ../../../server/vmafd/vecsserviceapi.c:836]

19-06-12T19:46:23.254850+02:00 err vmafdd  t@140302499055360: ERROR! [VecsIpcGetEntryByAlias] is returning  [4312]

0 Kudos
birend1988
Hot Shot
Hot Shot

As per the log snipped it seems vmdir in read only mode. Try to restart the PSC and try again. Also check the service status on VC and PSC servers.

VCIX, NCAP
0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

We have already tried restarting both entire VC and only the related services.

Also tried redeploying the new VCSA a few times but we get the same error.

0 Kudos
ydheeraj
Contributor
Contributor

Are you using External PSC or Embedded setup??

0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

Embedded

0 Kudos
birend1988
Hot Shot
Hot Shot

Can you share the vmdird.log

VCIX, NCAP
0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

vmdird from time of upgrade attached.

0 Kudos
birend1988
Hot Shot
Hot Shot

Current version of vCenter server??

I suspect this is the issue,

19-06-12T19:46:08.699450+02:00 err vmdird  t@139758168700672: VmDirSendLdapResult: Request (Search), Error (32), Message (DNToEID (9703)((MDB_NOTFOUND: No matching key/data pair found)(cn=b01ebf2b-e6ef-46d0-b14b-fe951be3ade7_com.vmware.migrate-connector.127.0.0.1,cn=serviceregistrations,cn=lookupservice,cn=Site2,cn=sites,cn=configuration,dc=vsphere,dc=local))), (0) socket (127.0.0.1)

19-06-12T19:46:52.902459+02:00 err vmdird  t@139758462215936: VmDirSendLdapResult: Request (Search), Error (32), Message (DNToEID (9703)((MDB_NOTFOUND: No matching key/data pair found)(cn=b01ebf2b-e6ef-46d0-b14b-fe951be3ade7_com.vmware.migrate-connector.127.0.0.1,cn=serviceregistrations,cn=lookupservice,cn=Site2,cn=sites,cn=configuration,dc=vsphere,dc=local))), (0) socket (127.0.0.1)

VCIX, NCAP
0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

Current version is 6.5.0-13834586

0 Kudos
birend1988
Hot Shot
Hot Shot

You need to open a case with VMware as it seems to be issue with the vmdir and vecs store.

VCIX, NCAP
NiklasAndersson
Enthusiast
Enthusiast

I have opened a case this morning. Thanks for your help so far.

I will post back when I've talked to vmware support.

0 Kudos
NiklasAndersson
Enthusiast
Enthusiast

Hi,

According to VMware support the issue is with two trusted root certificates on the vCenter server that installed by HPE Nimble VASA providers.

I will have to schedule a new maintenance window to confirm that removing those certificates solves the problem.

Probably won't happen until after the summer but I will post back as soon as I have verified.

0 Kudos
johager
Contributor
Contributor

Hi,

we had the same problems when using VCSA 6.7U2 for the upgrade (also Nimble Storage). After switching to VCSA 6.7U1 we could upgrade our VCSA 6.5. After the upgrade we updated the vcsa 6.7 manually to U2.

Greetings.

0 Kudos
GregSmid
Enthusiast
Enthusiast

Ran into the same problem with my upgrade from 6.5 U3 to 6.7 U3. Removing the invalid Nimble cert(s) as outlined in KB 70902 (VMware Knowledge Base) solved the issue and I was able to proceed with the upgrade.

Using the 6.7 U1 installer was not a work-around option for us since that would be an unsupported 'back-in-time' upgrade.

0 Kudos
jstefani
Contributor
Contributor

I too had the same issue going from 6.5 to 6.7U3g, but was successful in getting to 6.7U1 (6.7.0.21000). Here is my question...Within the VMware vCenter Appliance Manager, I have staged 6.7.0.44000 which is 6.7U3g.  I can't afford to take down this environment for maintenance so I need to know if this will work without issue. I've always performed vCenter upgrades from major versions (e.g. 6.0 to 6.5, 6.5 to 6.7) using the installer from the .iso image which is pretty interactive.  I'm hoping installing from the VC Appliance Manager will be the same.

Sincere Regards, Gio Stefani CIO/CISO
0 Kudos