Ian2498
Enthusiast
Enthusiast

vCenter Upgrade - Unexpected error 87 during certificate pre-check

Hi, I am experiencing a vCenter upgrade error in a customer's environment and wondered if anyone else in the community had come across anything similair.  I am receiving the following error message when performing an upgrade from vCenter 6.5 VCSA build 10964411 to 7.0 U2 VCSA at stage 2 pre-checks:

Error: Unexpected error 87 during certificate pre-check 

Resolution: Please contact support

 

The upgrade-source-requirements.log shows the following error:

2021-06-16T10:14:54.851Z ERROR __main__ FAILED: Found upgrade requirements mismatches.

 

The requirements-upgrade-runner.log shows the following:

2021-06-16T10:14:07.414Z INFO upgrade.states.component_states vmafd:CollectRequirements: 2021-06-16T10:14:07.71Z ERROR vmafdupgrade.collectRequirements Additional Certificate found at VMware Endpoint Certificate Store

2021-06-16T10:14:07.444Z INFO upgrade.states.component_states sso:CollectRequirements: 2021-06-16T10:14:07.136Z INFO ssoCollectValidate certDetails {} , certMismatch {'text': Certificate validation failed during pre-upgrade check., 'severity': 'ERROR', 'description': Unexpected error 87 during certificate pre-check., 'resolution': Please contact support, 'problemId': None}

2021-06-16T10:14:09.715Z INFO upgrade.states.component_states sso:CollectRequirements: 2021-06-16T10:14:08.828Z INFO extensions The component script returned '{'installArguments': {}, 'userOptionSpecs': [], 'extraArguments': {'appliance.net.pnid': ‘**********.***.AC.UK'}, 'coreRequirement': {'requiredDstDiskSpace': {'/storage/core': 0.01}, 'requiredSrcDiskSpace': 0.01, 'exportEstimationTime': 1, 'importEstimationTime': 1}, 'srcPorts': ['7444', '7080', '12721'], 'requirementMismatchSpecs': [{'text': Certificate validation failed during pre-upgrade check., 'severity': 'ERROR', 'description': Unexpected error 87 during certificate pre-check., 'resolution': Please contact support, 'problemId': None}], 'dstPortSpecs': [{'reconfigure': False, 'installationKey': 'sts.int.port1', 'destinationPort': '7080'}, {'reconfigure': False, 'installationKey': 'sts.int.port2', 'destinationPort': '12721'}, {'reconfigure': True, 'installationKey': 'sts.ext.port1', 'destinationPort': '7444'}]}'

2021-06-16T10:14:09.718Z INFO upgrade.states.component_states sso:CollectRequirements: Remote Command Returned: {'coreRequirement': {'importEstimationTime': 1, 'exportEstimationTime': 1, 'requiredDstDiskSpace': {'/storage/core': 0.01}, 'requiredSrcDiskSpace': 0.01}, 'requirementMismatchSpecs': [{'severity': 'ERROR', 'description': Unexpected error 87 during certificate pre-check., 'resolution': Please contact support, 'problemId': None, 'text': Certificate validation failed during pre-upgrade check.}], 'srcPorts': ['7444', '7080', '12721'], 'extraArguments': {'appliance.net.pnid': '**********.***.AC.UK'}, 'dstPortSpecs': [{'installationKey': 'sts.int.port1', 'destinationPort': '7080', 'reconfigure': False}, {'installationKey': 'sts.int.port2', 'destinationPort': '12721', 'reconfigure': False}, {'installationKey': 'sts.ext.port1', 'destinationPort': '7444', 'reconfigure': True}], 'installArguments': {}, 'userOptionSpecs': []}

                        "id": "upgrade.sso.precheck.error.resolution",

                        "id": "upgrade.sso.precheck.error.text",

                        "localized": "Unexpected error 87 during certificate pre-check.",

                        "id": "upgrade.sso.precheck.error.description",

                        "translatable": "Unexpected error %(0)d during certificate pre-check.",

                    "severity": "ERROR"

2021-06-16T10:14:53.977Z INFO __main__ Collected local upgrade runner requirements: [{'installedOn': '127.0.0.1', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'core': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 10, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [], 'srcPorts': ['8190', '22000'], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.sps'}, {'installedOn': '127.0.0.1', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {}, 'requiredSrcDiskSpace': 0.0, 'importEstimationTime': 0, 'exportEstimationTime': 0}, 'requirementMismatchSpecs': [], 'srcPorts': [], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.vsan-health'}, {'installedOn': '127.0.0.1', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {}, 'requiredSrcDiskSpace': 0.0, 'importEstimationTime': 0, 'exportEstimationTime': 0}, 'requirementMismatchSpecs': [], 'srcPorts': [], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.ngc'}, {'installedOn': '127.0.0.1', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'netdump': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 1, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [], 'srcPorts': ['6500'], 'extraArguments': {}, 'dstPortSpecs': [{'destinationPort': '6500', 'installationKey': 'netdumper.ext.serviceport', 'reconfigure': True}], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.netdump'}, {'installedOn': '127.0.0.1', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'core': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 1, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [{'resolution': {'localized': 'Please ensure extensions are compatible with the new vCenter Server  ...

2021-06-16T10:14:54.261Z INFO __main__ Prechecks aggregated result: [{'installedOn': 'localhost', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'core': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 10, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [], 'srcPorts': ['8190', '22000'], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.sps'}, {'installedOn': 'localhost', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {}, 'requiredSrcDiskSpace': 0.0, 'importEstimationTime': 0, 'exportEstimationTime': 0}, 'requirementMismatchSpecs': [], 'srcPorts': [], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.vsan-health'}, {'installedOn': 'localhost', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {}, 'requiredSrcDiskSpace': 0.0, 'importEstimationTime': 0, 'exportEstimationTime': 0}, 'requirementMismatchSpecs': [], 'srcPorts': [], 'extraArguments': {}, 'dstPortSpecs': [], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.ngc'}, {'installedOn': 'localhost', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'netdump': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 1, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [], 'srcPorts': ['6500'], 'extraArguments': {}, 'dstPortSpecs': [{'destinationPort': '6500', 'installationKey': 'netdumper.ext.serviceport', 'reconfigure': True}], 'installArguments': {}, 'userOptionSpecs': []}, 'name': 'com.vmware.netdump'}, {'installedOn': 'localhost', 'optional': False, 'requirements': {'coreRequirement': {'requiredDstDiskSpace': {'core': 0.01}, 'requiredSrcDiskSpace': 0.01, 'importEstimationTime': 1, 'exportEstimationTime': 1}, 'requirementMismatchSpecs': [{'resolution': {'localized': 'Please ensure extensions are compatible with the new vCenter Server and re-register  ...

The only 3rd party extensions installed are for Netapp Ontap Tools 9.8 for VMware vSphere, which according to the NetApp interoperability matrix are compatible with 7.0U2.

Thanks in advance!

0 Kudos
7 Replies
Ank_S
Enthusiast
Enthusiast

Hello @Ian2498 

 

Based on the log snippet shared , following KB may help you resolve the issue :

 

https://kb.vmware.com/s/article/68155


PS: Mark kudos or correct answer as appropriate

Ian2498
Enthusiast
Enthusiast

Hi, I did see this article which describes an upgrade error due to an expired STS cert.  My STS cert checks out as ok so unfortunately this article doesn't provide a solution in this instance.  

0 Kudos
Ank_S
Enthusiast
Enthusiast

Could you share the output of the file :

 

var/log/vmware/upgrade/prechecks.json

0 Kudos
Ian2498
Enthusiast
Enthusiast

Sure, info as requested, thanks.

{
"sourceInfo": {
"adDcAddress": null,
"isolatedNodes": [],
"diskInfo": [
{
"size": 12.0,
"mountedOn": "/dev",
"free": 12.0,
"used": 0.0
},
{
"size": 12.0,
"mountedOn": "/dev/shm",
"free": 12.0,
"used": 0.01
},
{
"size": 12.0,
"mountedOn": "/run",
"free": 12.0,
"used": 0.01
},
{
"size": 12.0,
"mountedOn": "/sys/fs/cgroup",
"free": 12.0,
"used": 0.0
},
{
"size": 11.0,
"mountedOn": "/",
"free": 4.5,
"used": 5.7
},
{
"size": 12.0,
"mountedOn": "/tmp",
"free": 12.0,
"used": 0.13
},
{
"size": 0.12,
"mountedOn": "/boot",
"free": 0.08,
"used": 0.04
},
{
"size": 25.0,
"mountedOn": "/storage/dblog",
"free": 17.0,
"used": 6.6
},
{
"size": 50.0,
"mountedOn": "/storage/seat",
"free": 40.0,
"used": 7.1
},
{
"size": 25.0,
"mountedOn": "/storage/db",
"free": 20.0,
"used": 3.9
},
{
"size": 50.0,
"mountedOn": "/storage/core",
"free": 47.0,
"used": 0.06
},
{
"size": 25.0,
"mountedOn": "/storage/imagebuilder",
"free": 24.0,
"used": 0.05
},
{
"size": 9.8,
"mountedOn": "/storage/netdump",
"free": 9.2,
"used": 0.03
},
{
"size": 25.0,
"mountedOn": "/storage/autodeploy",
"free": 24.0,
"used": 0.05
},
{
"size": 25.0,
"mountedOn": "/storage/log",
"free": 20.0,
"used": 3.5
},
{
"size": 99.0,
"mountedOn": "/storage/updatemgr",
"free": 94.0,
"used": 0.08
}
],
"domainName": null,
"adDcIpv6Addresses": null,
"deploymentType": "embedded",
"deploymentSize": "small",
"inventoryInfo": {
"vmCount": 249,
"hostCount": 7
},
"adDcIpv4Addresses": null,
"managementComponent": {
"version": "6.5"
},
"infrastructureComponent": {
"httpsPort": 443,
"version": "6.5",
"domainName": "vsphere.local",
"fqdn": "127.0.0.1"
},
"adDcName": null,
"dnsAddresses": ".*.*.*,*.*.*.*"
},
"optionalData": [
{
"requirements": {
"requiredSrcDiskSpace": 1.1,
"requiredDstDiskSpace": {
"seat": 8.7
},
"exportEstimationTime": 24,
"importEstimationTime": 43
},
"optionId": "vcdb.migrateSet",
"requirementsMismatch": {
"warning": [],
"error": []
},
"answerId": "all",
"valid": true
},
{
"requirements": {
"requiredSrcDiskSpace": 0.12,
"requiredDstDiskSpace": {
"seat": 0.89
},
"exportEstimationTime": 3,
"importEstimationTime": 6
},
"optionId": "vcdb.migrateSet",
"requirementsMismatch": {
"warning": [],
"error": []
},
"answerId": "core_events_tasks",
"valid": true
},
{
"requirements": {
"requiredSrcDiskSpace": 0.0,
"requiredDstDiskSpace": {},
"exportEstimationTime": 0,
"importEstimationTime": 0
},
"optionId": "vcdb.migrateSet",
"requirementsMismatch": {
"warning": [],
"error": []
},
"answerId": "core",
"valid": true
}
],
"componentRequirements": [],
"requirements": {
"requiredSrcDiskSpace": 1.96,
"requiredDstDiskSpace": {
"vtsdb": 0.12,
"db": 4.65,
"updatemgr": 0.08,
"lifecycle": 0.12,
"archive": 0.08,
"seat": 0.08,
"core": 0.1,
"autodeploy": 0.05,
"imagebuilder": 0.03,
"log": 0.06,
"netdump": 0.03,
"vtsdblog": 0.12,
"dblog": 0.12
},
"exportEstimationTime": 13,
"importEstimationTime": 29
},
"requirementsMismatch": {
"warning": [
{
"text": {
"id": "ur.esx.managedby.vc.warning.text",
"translatable": "This ESXi host [%(0)s] is managed by vCenter Server [%(1)s].",
"args": [
"//*.*.*.*:443",
"*.*.*.*"
],
"localized": "This ESXi host [//*.*.*.*:443] is managed by vCenter Server [*.*.*.*]."
},
"description": null,
"resolution": {
"id": "ur.esx.managedby.source.vc.resolution",
"translatable": "Make sure the cluster where this ESXi host resides is not set to Fully Automated DRS for the duration of the upgrade process.",
"localized": "Make sure the cluster where this ESXi host resides is not set to Fully Automated DRS for the duration of the upgrade process."
},
"problemId": null
},
{
"text": {
"localized": "vCenter External Extensions",
"translatable": "vCenter External Extensions",
"id": "upgrade.vpxd.external.extension.title"
},
"description": {
"args": [
"\nVDP on https://*.*.*.*:8543/vdp-plugin-package.zip, \nONTAP tools for VMware vSphere (by NetApp Inc.) on https://*.*.*.*:8143/htmlclient_deployment_bundle, \nONTAP tools for VMware vSphere (by NetApp Inc.) on https://*.*.*.*:8143/vSphereExtensionDescriptor.xml, \n"
],
"localized": "This vCenter Server has extensions registered that cannot be upgraded to or may not work with the new vCenter Server.\n\nExtensions:\n\nVDP on https://*.*.*.*:8543/vdp-plugin-package.zip, \nONTAP tools for VMware vSphere (by NetApp Inc.) on https://*.*.*.*:8143/htmlclient_deployment_bundle, \nONTAP tools for VMware vSphere (by NetApp Inc.) on https://*.*.*.*:8143/vSphereExtensionDescriptor.xml, \n",
"translatable": "This vCenter Server has extensions registered that cannot be upgraded to or may not work with the new vCenter Server.\n\nExtensions:\n%(0)s",
"id": "upgrade.vpxd.external.extension.msg"
},
"resolution": {
"localized": "Please ensure extensions are compatible with the new vCenter Server and re-register extensions with the new vCenter Server after upgrade.\nPlease refer to the vSphere documentation on extensions, and the upgrade and interoperability guides.",
"translatable": "Please ensure extensions are compatible with the new vCenter Server and re-register extensions with the new vCenter Server after upgrade.\nPlease refer to the vSphere documentation on extensions, and the upgrade and interoperability guides.",
"id": "upgrade.vpxd.external.extension.resolution.msg"
},
"problemId": "upgrade.vpxd.external.extension"
},
{
"text": {
"args": [
"7.0.2"
],
"localized": "Files that cannot be used with Lifecycle Manager 7.0.2 will not be copied from the source. These files include VM guest OS patch baselines, host upgrade baselines and files, and ESXi 6.0 and lower version host patches baselines.",
"translatable": "Files that cannot be used with Lifecycle Manager %(0)s will not be copied from the source. These files include VM guest OS patch baselines, host upgrade baselines and files, and ESXi 6.0 and lower version host patches baselines.",
"id": "upgrade.vcIntegrity.warning.description"
},
"description": {
"args": [
"7.0.2"
],
"localized": "Files that cannot be used with Lifecycle Manager 7.0.2 will not be copied from the source. These files include VM guest OS patch baselines, host upgrade baselines and files, and ESXi 6.0 and lower version host patches baselines.",
"translatable": "Files that cannot be used with Lifecycle Manager %(0)s will not be copied from the source. These files include VM guest OS patch baselines, host upgrade baselines and files, and ESXi 6.0 and lower version host patches baselines.",
"id": "upgrade.vcIntegrity.warning.description"
},
"resolution": {
"args": [
"7.0.2"
],
"localized": "Please review VMware Lifecycle Manager 7.0.2 Documentation for details",
"translatable": "Please review VMware Lifecycle Manager %(0)s Documentation for details",
"id": "upgrade.vcIntegrity.warning.resolution"
},
"problemId": null
},
{
"text": {
"args": [
"7.0.2"
],
"localized": "The proxy settings for the vSphere Update Manager will not be retained. The Lifecycle Manager 7.0.2 will now use proxy settings configured from vCenter Server Appliance.",
"translatable": "The proxy settings for the vSphere Update Manager will not be retained. The Lifecycle Manager %(0)s will now use proxy settings configured from vCenter Server Appliance.",
"id": "upgrade.vcIntegrity.proxy.warning"
},
"description": {
"localized": "The vSphere Update Manager specific proxy settings is deprecated from vSphere 7.0 GA. The Lifecycle Manager service will now use the proxy settings configured at VMware vCenter Server Appliance (VCSA) Interface.",
"translatable": "The vSphere Update Manager specific proxy settings is deprecated from vSphere 7.0 GA. The Lifecycle Manager service will now use the proxy settings configured at VMware vCenter Server Appliance (VCSA) Interface.",
"id": "upgrdae.vcIntegrity.proxy.warning.description"
},
"resolution": {
"args": [
"7.0.2"
],
"localized": "Please log in to the vCenter Server Appliance Management Interface to view and configure proxy settings. Refer to VMware Lifecycle Manager 7.0.2 release notes for details.",
"translatable": "Please log in to the vCenter Server Appliance Management Interface to view and configure proxy settings. Refer to VMware Lifecycle Manager %(0)s release notes for details.",
"id": "upgrade.vcIntegrity.proxy.warning.resolution"
},
"problemId": null
},
{
"text": {
"id": "ur.cpu.cores.warning.text",
"translatable": "The source vCenter Server instance is configured with more CPU cores than the target appliance.",
"localized": "The source vCenter Server instance is configured with more CPU cores than the target appliance."
},
"description": null,
"resolution": {
"id": "ur.cpu.cores.warning.resolution",
"translatable": "If you need to increase the number of CPU cores on the target vCenter Server Appliance, you can do that manually, after the upgrade or migration finishes",
"localized": "If you need to increase the number of CPU cores on the target vCenter Server Appliance, you can do that manually, after the upgrade or migration finishes"
},
"problemId": null
}
],
"error": [
{
"text": {
"localized": "Certificate validation failed during pre-upgrade check.",
"translatable": "Certificate validation failed during pre-upgrade check.",
"id": "upgrade.sso.precheck.error.text"
},
"description": {
"args": [
87
],
"localized": "Unexpected error 87 during certificate pre-check.",
"translatable": "Unexpected error %(0)d during certificate pre-check.",
"id": "upgrade.sso.precheck.error.description"
},
"resolution": {
"localized": "Please contact support",
"translatable": "Please contact support",
"id": "upgrade.sso.precheck.error.resolution"
},
"problemId": null
}
]
},
"componentLocations": [
{
"installedOn": "localhost",
"name": "com.vmware.sps"
},
{
"installedOn": "localhost",
"name": "com.vmware.vsan-health"
},
{
"installedOn": "localhost",
"name": "com.vmware.ngc"
},
{
"installedOn": "localhost",
"name": "com.vmware.netdump"
},
{
"installedOn": "localhost",
"name": "com.vmware.vpxd"
},
{
"installedOn": "localhost",
"name": "com.vmware.license"
},
{
"installedOn": "localhost",
"name": "com.vmware.vmafd"
},
{
"installedOn": "localhost",
"name": "com.vmware.common_upgrade"
},
{
"installedOn": "localhost",
"name": "com.vmware.vcIntegrity"
},
{
"installedOn": "localhost",
"name": "com.vmware.sso"
},
{
"installedOn": "localhost,*.*.*.*",
"name": "com.vmware.rbd"
},
{
"installedOn": "localhost",
"name": "com.vmware.rhttpproxy"
},
{
"installedOn": "localhost",
"name": "com.vmware.syslog"
},
{
"installedOn": "localhost",
"name": "com.vmware.vcdb"
},
{
"installedOn": "localhost",
"name": "upgrade_framework"
}
],
"extraArguments": {
"ur.appliance.sts.url": "https://************.AC.UK/sts/STSService/vsphere.local",
"ur.appliance.sso.port": 443,
"ur.appliance.sso.admin.endpoint.path": "/sso-adminserver/sdk/vsphere.local",
"ur.appliance.net.pnid": "************AC.UK",
"ur.appliance.sso.ipaddresses": [
"127.0.0.1"
],
"ur.appliance.sso.pnid": "***********.ac.uk",
"ur.appliance.vmdir.nodes.list": null,
"ur.appliance.cpu.cores": 8
}
}

0 Kudos
Ank_S
Enthusiast
Enthusiast

Hello @Ian2498 


The log snippet uploaded does not contain the info i am looking for.


Considering SSL certs are valid and STS does have the correct FQDN of the vCenter in its Subject Alternate Name (SAN)  and matches the FQDN in the Machine_SSL cert , the only other thing we can look for is the sslTrust mismatch in corresponding service registrations with VMware Lookup Service.


Try to follow the below KB and see if it resolves the issue.


https://kb.vmware.com/s/article/78552 


If it does not work , would advise opening a case with VMware support to advise further based on the log review.

Ian2498
Enthusiast
Enthusiast

I contacted VMware support & an escalation engineer identified an issue with the certificates in use.

The certificates available on the vCenterserver 6.5 were older certificates generated some time ago.  They had not expired however they were missing a few required parameters like Subject Key Identifier, Authority Key identifier. 

We used Certificate Manager option 8 to reset all certificates.

We did have STS_INTERNAL_SSL_CERT available in the store which was holding the older certificates.

Below are the commands we used to match the sts_ssl_cert to the existing machine_ssl_cert:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/sts.cer
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/sts.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/machine.cer
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/machine.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/machine.cer --key /var/tmp/machine.key

We also used ls.doctor which cleaned up a number of legacy 5.5 entries in the config

After the changes were made, the 'Unexpected error 87 during certificate pre-check' error was no longer present and I was able to proceed with the upgrade.

Ank_S
Enthusiast
Enthusiast

Hello @Ian2498 

 

Thanks for the update and sharing the steps.

Glad to hear the issue is sorted now.

 

Regards,

Ank_S

0 Kudos