Solved: VMWare vCenter Upgrade 6.7 (18831049) to 7.0U3a

MKey90 · ‎11-30-2021

Hello,

we are trying to Upgrade our vCenter 6.7 (18831049) to vCenter 7.0U3a.

Stage 1 in the Migration Process is working normally.

Stage 2 fails with the following Error:

Internal error occurs during Export of VMware vSphere Update Manager.
Error from the Support Bundle Logfile - "Export_com.vmware.vcIntegrity_..."
- 2021-11-29T14:25:50.586Z ERROR __main__ Upgrade Phase 'vcIntegrity:Export' failed. Exception: list index out of range
  Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/payload/componentPhaseLauncher.py", line 461, in main
  executionResult = systemExtension(exeContext)
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/libs/sdk/extensions.py", line 94, in __call__
  result = self.extension(*args)
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/libs/sdk/extensions.py", line 110, in _func
  return func(*args)
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/payload/component-scripts/vcIntegrity/__init__.py", line 50, in doExport
  return vcIntegrity_upgrade.exportData(context)
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 379, in exportData
  exportDataLin(context)
  File "/tmp/vmware-upgrade-temp-dirBpwKTL4VL0/tmpFXZ1iaTIKD/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 357, in exportDataLin
  vumDBPassword = vumDBPasswordElem[0].text
  IndexError: list index out of range

What i tried so far:

Update our vCenter 6.7 (18831049) to 7.0 U2d -> same error

Resetting the Update Manager Database (KB2147284) -> same error

Does anyone have any more ideas on what else I could try?

best regards

Quickpaw · ‎12-02-2021

Exactly the same boat here. Support told us that we can't upgrade because of the vSphere Back-in-time release upgrade restriction.

kb.vmware.com/s/article/67077
kb.vmware.com/s/article/2143838

EDIT: vCenter was incorrectly reporting my build number. VAMI correctly reports it as 18831133, and that currently does not support upgrade to 7, per their matrix.

Nothing to do now but wait, I guess.

View solution in original post

MKey90 · ‎12-02-2021

Thats correct, i also got this answer.

Here is a good list for the Release Dates:

https://www.virten.net/vmware/esxi-release-build-number-history/

My Current Version - vCenter Server Appliance 6.7 U3p 2021-11-23 6.7.0.51000 18831133 18831049

The 7.0U3a Version - vCenter Server 7.0 U3a 2021-10-21 7.0.3.00100 18778458 18778458

So we have to wait for a 7.0 Version with a Release Date after 2021-11-23

View solution in original post

mastamme · ‎01-30-2022

vCenter 7U3c is now supportet to upgrade from 6.7U3p https://kb.vmware.com/s/article/67077?lang=en_us

View solution in original post

dhierhol · ‎12-01-2021

Same problem here: trying to upgrade from 6.7.0.51000 (a.k.a. vCenter 6.7 build 18831133) to 7.0.3a

/var/log/vmware/upgrade/export-upgrade-runner.log

reported message 'Exporting VMware vSphere Update Manager data...'
2021-12-01T14:21:23.906Z INFO upgrade.states.component_states vcIntegrity:Export: 
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.811Z INFO vcIntegrity_upgrade Add VUM schema to search path on destination vSphere...
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.825Z INFO vcIntegrity_upgrade Add VUM schema to search path return code: 0
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.826Z INFO vcIntegrity_upgrade Add VUM schema to search path StdOut:
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.826Z INFO vcIntegrity_upgrade
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.826Z INFO vcIntegrity_upgrade Add VUM schema to search path StdErr:
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.826Z INFO vcIntegrity_upgrade
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.826Z INFO vcIntegrity_upgrade Add VUM schema to search path completed
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.828Z INFO vcIntegrity_upgrade Export VUM scheam version on destination vSphere...
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.842Z INFO vcIntegrity_upgrade Export VUM scheam version return code: 0
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.842Z INFO vcIntegrity_upgrade Export VUM scheam version StdOut:
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.842Z INFO vcIntegrity_upgrade                   120
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.843Z INFO vcIntegrity_upgrade Export VUM scheam version StdErr:
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.843Z INFO vcIntegrity_upgrade
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.843Z INFO vcIntegrity_upgrade Export VUM scheam version completed
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: 2021-12-01T14:21:23.845Z ERROR __main__ Upgrade Phase 'vcIntegrity:Export' failed. Exception: list index out of range
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export: Traceback (most recent call last):
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/payload/componentPhaseLauncher.py", line 461, in main
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:     executionResult = systemExtension(exeContext)
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/libs/sdk/extensions.py", line 94, in __call__
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:     result = self.extension(*args)
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/libs/sdk/extensions.py", line 110, in _func
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:     return func(*args)
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/payload/component-scripts/vcIntegrity/__init__.py", line 50, in doExport
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:     return vcIntegrity_upgrade.exportData(context)
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 379, in exportData
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:     exportDataLin(context)
2021-12-01T14:21:23.907Z INFO upgrade.states.component_states vcIntegrity:Export:   File "/tmp/vmware-upgrade-temp-dirmVcHlWFesC/tmp9ITrSUohLi/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 357, in exportDataLin
2021-12-01T14:21:23.908Z INFO upgrade.states.component_states vcIntegrity:Export:     vumDBPassword = vumDBPasswordElem[0].text
2021-12-01T14:21:23.908Z INFO upgrade.states.component_states vcIntegrity:Export: IndexError: list index out of range
2021-12-01T14:21:23.908Z INFO upgrade.states.component_states vcIntegrity:Export: Script completed for 11.041993379592896 secs with return-code='1', and executionId=4200274c-005b-47fb-9bf8-e95b0d90007d
2021-12-01T14:21:23.908Z ERROR upgrade.states.component_states vcIntegrity:Export: Remote script failed with an error [InternalError()]
2021-12-01T14:21:23.908Z ERROR upgrade.states.component_states vcIntegrity:Export: failed with internal error. For details take a look at Export_com.vmware.vcIntegrity_2021_12_01_14_21.log.
2021-12-01T14:21:23.914Z WARNING wf.wf_processor State Export com.vmware.vcIntegrity has failed.
2021-12-01T14:21:23.914Z ERROR wf.wf_processor Workflow execution has failed
2021-12-01T14:21:23.914Z WARNING root stopping status aggregation...
2021-12-01T14:21:23.919Z INFO UpgradeRunner Persisting upgrade workflow context..
2021-12-01T14:21:23.919Z INFO deployer.migration_env_deployer Cleaning-up migration environment...
2021-12-01T14:21:23.919Z INFO utils.loginspec_utils User credentials based login specification.
2021-12-01T14:21:23.919Z INFO utils.loginspec_utils The vCenter Server username and password are specified.
2021-12-01T14:21:23.926Z INFO bindings.vc_binding User credentials based authentication.
2021-12-01T14:21:24.15Z INFO deployer.migration_env_deployer The extension 'com.vmware.migrate-connector.127.0.0.1' has been uninstalled
2021-12-01T14:21:24.15Z INFO deployer.migration_env_deployer The components located on 127.0.0.1 has been successfully unregistered as extension.
2021-12-01T14:21:24.98Z INFO deployer.migration_env_deployer Cleaning up target: None target with address=127.0.0.1
2021-12-01T14:21:24.99Z INFO deployer.migration_env_deployer The target None target with address=127.0.0.1 is already cleaned up
2021-12-01T14:21:24.100Z ERROR UpgradeRunner Upgrade mode export failed
2021-12-01T14:21:24.100Z INFO root Exiting with exit-code 1

dhierhol · ‎12-01-2021

Created a ticket with VMware support: SR #21283013412

DM_80 · ‎12-01-2021

We ran into the exact same issue with exactly the same versions (pre and post) and identical logging as displayed.

Have created an support request as well: SR# 21283120212.

dhierhol · ‎12-01-2021

Many thanks for sharing this - I've mentioned your case number in my ticket so that the two engineers can share their findings.

MKey90 · ‎12-01-2021

I also mentioned your Ticket Number in my Ticket.

Quickpaw · ‎12-02-2021

Exactly the same boat here. Support told us that we can't upgrade because of the vSphere Back-in-time release upgrade restriction.

kb.vmware.com/s/article/67077
kb.vmware.com/s/article/2143838

EDIT: vCenter was incorrectly reporting my build number. VAMI correctly reports it as 18831133, and that currently does not support upgrade to 7, per their matrix.

Nothing to do now but wait, I guess.

MKey90 · ‎12-02-2021

Thats correct, i also got this answer.

Here is a good list for the Release Dates:

https://www.virten.net/vmware/esxi-release-build-number-history/

My Current Version - vCenter Server Appliance 6.7 U3p 2021-11-23 6.7.0.51000 18831133 18831049

The 7.0U3a Version - vCenter Server 7.0 U3a 2021-10-21 7.0.3.00100 18778458 18778458

So we have to wait for a 7.0 Version with a Release Date after 2021-11-23

NLYNN · ‎12-20-2021

Same issue. Why is this not a pre-check requirement so I don't go through the upgrade just to find that it fails?

para92 · ‎12-20-2021

I agree, very frustrating.

mpwerner · ‎01-04-2022

Same thing happened to me. Same versions.

My Current Version - vCenter Server Appliance 6.7 U3p 2021-11-23 6.7.0.51000 18831133 18831049

The 7.0U3a Version - vCenter Server 7.0 U3a 2021-10-21 7.0.3.00100 18778458 18778458

Opened a case with VMware as well. Seems like no questions this should have been included in a pre-check.

Trying to find out any specifics for backing out and canceling the upgrade with no impact. VMware support must be getting quite a few cases with this same issue

Support Request #22292359201

Quickpaw · ‎01-04-2022

Agreed, it should be a pre-check. At the very least it would save a lot of time and head scratching.

Backing out shouldn't be an issue though, the upgrade creates a new VM, so just power on the original one and get rid of the new one it created.

ChadMarkley · ‎01-07-2022

OMG, this is so lame. I am having the same issue. Like an idiot, i patched my Vcenter before deciding to move to v7. Banging my head against the wall trying to figure out WTF was going on. I am trying to move from 6.7.0.51000 build 18831133, with a 11-21-2021 date via Vcenter 7.0u3a Build 18778458, with a 10-21-2021 date.

I am literally punishing myself for patching my current vcenter server!

Oh, the current builds for vcenter are here. Includes the latest and greatest https://kb.vmware.com/s/article/2143838

ftaormina · ‎01-11-2022

Just Wow!! Having the exact same issue. Waiting for support call. I used VRanger to restore the appliance back prior to doing the 11/23 patch - 6.7 Update 3p (6.7.0.51000) but then I started having issues with my hosts not syncing and menu options not available so I nixed that idea and went back to my va67 and then deleted the va70 instance that was made during the failed upgrade. Still can't log into the virtual appliance manager but at least vcenter seems to be normal and I can manage all my host. Guess will have to wait to see what support does or if we just need to wait for a patch/fix.

tomwerf · ‎01-14-2022

Same problem here.
After the upgrade failure, the vCenter appliance manager of the 6.7 applicance was stuck in the installer screen.

I've "reset" the VM Appliance manager through SSH by creating 3 empty files:

shell
cd /var/log/vmware/upgrade
touch prechecks-succeeded
touch import-succeeded
touch export-succeeded

DM_80 · ‎01-15-2022

Indeed, see KB article "Accessing the VAMI returns error "Update installation in progress" after recovering from a failed update (67179)"

https://kb.vmware.com/s/article/67179

Been there as well

echio22669 · ‎01-18-2022

Any update on this? I just ran into the same issue

echio22669 · ‎01-18-2022

Any Update for this ? I just ran into the same issue. No Workaround from what VMware is telling us

ftaormina · ‎01-18-2022

We will need to wait until VMWare comes out with a newer rev for 7.0 so that the upgrade doesn't fail due to the back in time patch for 6.7. I just backed out and created the 3 files to get back to where I was prior to doing the upgrade. You will need to make sure SSH Bash is enabled and log into your 6.7VA and create the three files so you can log back into the VAMI.

mastamme · ‎01-20-2022

I also got the answer to wait until 7.0u3c appears. the only options are restore to 6.7U3o or a new installation to 7.0 SR#22297001201

All

VMWare vCenter Upgrade 6.7 (18831049) to 7.0U3a