VMware Cloud Community
bascheew
Enthusiast
Enthusiast
Jump to solution

Error when upgrading from VCSA 6.7 to 7.0

Running VCSA 6.7.0.42300 and attempting to upgrade to 7.0.0.10100.  The new VCSA is deployed fine, and during stage 2 right after the pre-upgrade tests are run I receive the following error.

vCenter 7 Upgrade Error.png

Internal error occurs during execution of upgrade process.

Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

I reviewed the logs and in the "requirements-upgrade-runner.log" the last error is this:

2020-04-03T05:27:57.786Z INFO utils.loginspec_utils User credentials based login specification.

2020-04-03T05:27:57.787Z INFO utils.loginspec_utils The vCenter Server username and password are specified.

2020-04-03T05:27:57.804Z INFO bindings.vc_binding User credentials based authentication.

2020-04-03T05:27:57.863Z INFO deployer.migration_env_deployer Updating extension -- com.vmware.migrate-connector.127.0.0.1

2020-04-03T05:27:57.932Z ERROR UpgradeRunner Upgrade Runner has encountered an exception

Traceback (most recent call last):

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/UpgradeRunner.py", line 1889, in main

    envDeployer.setup(srcTargets, destTargets, isRegVcExt)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 563, in setup

    self._setupTargets(srcTargetsToSetup, False, isRegVcExt)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 535, in _setupTargets

    self._registerTargetAsExtension(targetConfig)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 345, in _registerTargetAsExtension

    _registerExtension(srcSi, extKey, extCertificate)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 78, in _registerExtension

    em.UpdateExtension(ext)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 556, in <lambda>

    self.f(*(self.args + (obj,) + args), **kwargs)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 368, in _InvokeMethod

    return self._stub.InvokeMethod(self, info, args)

  File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1481, in InvokeMethod

    raise obj # pylint: disable-msg=E0702

pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = 'lookup.fault.EntryNotFoundFault',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   reason = 'Invalid fault'

}

Please advise on how to further troubleshoot the upgrade.

1 Solution

Accepted Solutions
bascheew
Enthusiast
Enthusiast
Jump to solution

I was finally able to pass the pre-upgrade checks! 

First I had to remove the "com.vmware.migrate-connector.127.0.0.1" extension in order to resolve the "Internal error occurs during execution of upgrade process" error.  To do this I had to:

  1. Go to https://<vcenter server name or IP>/mob/?moid=ExtensionManager
  2. Verify that "com.vmware.migrate-connector.127.0.0.1" was in the list of extensions (I had to click "more" at the bottom of the list to see it).
  3. In the bottom section of the page click UnregisterExtension. (This will open a new window).
  4. In the value field paste "com.vmware.migrate-connector.127.0.0.1" and click "Invoke Method"  (I also removed an old EqualLogic extension that was no longer in use.)

I then restarted the vCenter 6.7 VCSA and ran stage 2 of the upgrade again.  This time upon completing the pre-upgrade checks I received a different error: "Certificate validation failed during pre-upgrade check".  I reset all certs (/usr/lib/vmware-vmca/bin/certificate-manager" option 8 ), but this itself did not fix anything, so that step is likely optional.  I then SSH'd into the VCSA and ran the following the commands as instructed in the this KB article: VMware Knowledge Base

  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.crt
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.key
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.crt
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.key
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/MachineSSL.crt --key /var/tmp/MachineSSL.key

I then restarted the VCSA again, ran stage 2 upgrade and it passed the checks and successfully upgraded to 7.0.

Many thanks to tomosimeonov​ for assisting in this problem!

View solution in original post

19 Replies
berndweyand
Expert
Expert
Jump to solution

i think you should open a support request.

vpshere 7.0 was released yesterday so you will find no one here who has experience with 7.0

Reply
0 Kudos
akodenkiri
VMware Employee
VMware Employee
Jump to solution

Hi bascheew,

Can you please provide some information about your source VCSA 6.7 env ? whether any solutions or extension registered to it ? Is it possible to share the requirements-upgrade-runner.log ?

Regards

---------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards ak
Reply
0 Kudos
bascheew
Enthusiast
Enthusiast
Jump to solution

We don't have any extensions that I'm aware of using vCenter.  The environment is fairly simple - one Datacenter, 3 Clusters, a couple hundred VMs, DRS, distributed switches, SAN storage.  We originally started with vCenter 5 on Windows and have upgraded it over the years to VCSA 6.7.  I wonder if there is some old garbage lingering around from previous versions. 

Attached is a sanitized log file.  Names and IPs have been changed.

I will add that the first time we ran the prerequisite check I had 2 errors - one was cert related so I renewed all the certs on the old VCSA and that error went away.  The second error was about vswitches and folders having the same name.  I just renamed the switches or folders and then that error went away -- leaving me with the error I've posted about.

bascheew
Enthusiast
Enthusiast
Jump to solution

The following article seems awfully close to the problem I'm having: https://kb.vmware.com/s/article/78296?lang=en_US%3Flang%3Den_US

However I don't see any VMs showing up as "invalid" when I run the following command across all my hosts.

vim-cmd vmsvc/getallvms

I'm not sure this is the same issue...

Reply
0 Kudos
jmhalder
Contributor
Contributor
Jump to solution

My issue has thrown the same exception at the same place. I'm clueless at this point, I've been hammering on it for a few hours, exploring every avenue.

Reply
0 Kudos
akodenkiri
VMware Employee
VMware Employee
Jump to solution

Thanks . Can you please share the Upgraderunner.log as well?

---------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards ak
Reply
0 Kudos
tomosimeonov
VMware Employee
VMware Employee
Jump to solution

Hi,

I think you might have stale entries in your ExtensionManager on the 6.7 environment. Can you check if you have Extension called com.vmware.migrate-connector.127.0.0.1 and if so remove it. If that does not solve your problem I would highly recommend raising  a support request.

Thanks,

Tomo

jmhalder
Contributor
Contributor
Jump to solution

https://vcenter.domain.local/mob/?moid=ExtensionManager I did find com.vmware.migrate-connector.127.0.0.1 there and was able to remove it. As the other person had problems with Certs, so did I. They seemingly went away... After unregistering that extension, I had the same cert problem... I'm sure it didn't help that I probably tried the upgrade 10+ times last night... Okay:

"A vCenter Single Sign-On endpoint certificate validation error has occurred. Resolution Ensure that the endpoint service registrations in vmdir match their corrsponding machine SSL certificates in VECS. For more information, see Knowledge Base article KB 2121701"

VMware Knowledge Base

I was able to get the "old cert" and run the ls_update_certs.py script. Now it seems to be working fine. What a nightmare, hopefully this helps somebody.

*To clarify, I'm running a 6.7 vcsa with embedded psc*

Reply
0 Kudos
pradeepjigalur
Enthusiast
Enthusiast
Jump to solution

HI, jmhalder​,

Now what's the state of upgrade now. are you able to proceed with stage2.

Reply
0 Kudos
jmhalder
Contributor
Contributor
Jump to solution

Correct, stage 2 completed, and I'm upgraded to 7.0 now.

Reply
0 Kudos
pradeepjigalur
Enthusiast
Enthusiast
Jump to solution

jmhalder​ thanks for confirmation.

Can you share more information regarding the history of VC.

you have told it started from 5.0. we want to know what all path you traversed to reach 67 like (5.0->5.1->5.5->6.7/5,0->5,1->5.5->6.0->6.5->6.7 etc) . where did you do the migration etc.

Reply
0 Kudos
bascheew
Enthusiast
Enthusiast
Jump to solution

Thank you for your assistance!

It's been a long time, but I believe the upgrade path was 5.0 Windows -> 5.1 Windows -> 5.5 Windows -> 6.0 VCSA -> 6.5 VCSA -> 6.7 VCSA.

I found that I also had the extension "com.vmware.migrate-connector.127.0.0.1" installed so I removed it.  Now I am getting a cert error instead:

Error:  Certificate validation failed during pre-upgrade check.
Resolution:  Please contact support

I will start troubleshooting this error and report back on my findings.
vCenter 7 Upgrade Error 2.png

Reply
0 Kudos
bascheew
Enthusiast
Enthusiast
Jump to solution

The requirement-upgrade-runner.log contain the following line:

INFO ssoCollectValidate certDetails {} , certMismatch {'resolution': Please contact support, 'problemId': None, 'description': Unexpected error 87 during certificate pre-check., 'severity': 'ERROR', 'text': Certificate validation failed during pre-upgrade check.}

Under "Certificate Management" of vCenter all certs show a green check and are valid.  This error came from the ssoCollectVaildate stage, which would indicate an SSO cert perhaps?  Any idea where I should look next?

Reply
0 Kudos
bascheew
Enthusiast
Enthusiast
Jump to solution

I ran the certificate manager on the 6.7 vCenter CLI "/usr/lib/vmware-vmca/bin/certificate-manager" and chose option 8 - Reset all Certificates.  After that command successfully ran, I rebooted the vCenter server and restarted stage 2 of the VCSA 7.0 upgrade.  I still receive the same pre-upgrade certificate error.  Any other suggestions on what I can do?

Reply
0 Kudos
bascheew
Enthusiast
Enthusiast
Jump to solution

I was finally able to pass the pre-upgrade checks! 

First I had to remove the "com.vmware.migrate-connector.127.0.0.1" extension in order to resolve the "Internal error occurs during execution of upgrade process" error.  To do this I had to:

  1. Go to https://<vcenter server name or IP>/mob/?moid=ExtensionManager
  2. Verify that "com.vmware.migrate-connector.127.0.0.1" was in the list of extensions (I had to click "more" at the bottom of the list to see it).
  3. In the bottom section of the page click UnregisterExtension. (This will open a new window).
  4. In the value field paste "com.vmware.migrate-connector.127.0.0.1" and click "Invoke Method"  (I also removed an old EqualLogic extension that was no longer in use.)

I then restarted the vCenter 6.7 VCSA and ran stage 2 of the upgrade again.  This time upon completing the pre-upgrade checks I received a different error: "Certificate validation failed during pre-upgrade check".  I reset all certs (/usr/lib/vmware-vmca/bin/certificate-manager" option 8 ), but this itself did not fix anything, so that step is likely optional.  I then SSH'd into the VCSA and ran the following the commands as instructed in the this KB article: VMware Knowledge Base

  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.crt
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.key
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.crt
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.key
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
  • /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/MachineSSL.crt --key /var/tmp/MachineSSL.key

I then restarted the VCSA again, ran stage 2 upgrade and it passed the checks and successfully upgraded to 7.0.

Many thanks to tomosimeonov​ for assisting in this problem!

fsaguilieri
Contributor
Contributor
Jump to solution

  Hello,

Me too. Somebody have some solution?

Thanks

Reply
0 Kudos
MrChris87
Contributor
Contributor
Jump to solution

I login first time to the community with my account (as far as i remember) just to say thank you to you  bascheew

I worked since the morning on updating the vcsa and already fixed some else problems. But this last one i could not figure out. Your hint was the final solution i needed. Thank you VERY MUCH!

Reply
0 Kudos
nahenry
Contributor
Contributor
Jump to solution

amazing. I finally got it to work and I've been on this for 3 days now.

I think the important part for us was to stop and restart the services with 

service-control --stop --all
service-control --start --all

or just reboot the appliance.

FYI: we have an appliance running 6.7 and upgrading to 7.0. 

I tried initially to go from 6.5 to 6.7 when making the jump straight to 7.0 gave the 87 error, but even with the interim upgrade, it still failed. ultimately I think the cert manager option 8, restarting services, then running the commands listed in the solution worked!

thanks for using time to add the info here and help others.
 

Tags (1)
Reply
0 Kudos
mgiurgeu
Contributor
Contributor
Jump to solution

I had the same problem upgrading from 7.0 to 8. The solution worked in my case, also. I did not restart the source VCSA nor restarted the services after removing the com.vmware.migrate-connector (I did not touch the certificates.

Thanks again for sharing

Reply
0 Kudos