Workaround: VCSA 7.02 -> 7.03 upgrade succeeds, but VAMI stuck in loop "Error in method invocation"

[Sqrl]

Hi All,

A colleague encountered an interesting issue during a standard upgrade of the VCSA 7.02 to 7.03 in our lab environment, and since this specific issue/error encountered appears to be undocumented I thought I'd share the work around I put in place.

After what appeared to be a successful upgrade of the VCSA to 7.03, upon login to the VAMI you are redirected to "https://[vcsa.fqdn_or_IP]:5480/configurev2/" and presented with the following prompt "Error in method invocation". Attempts to bypass this URL by direct access to a known URL (eg #/ui/summary) will result in redirection back to /configurev2.

Suggestions to remove the update state file (/etc/applmgmt/appliance/software_update_state.conf) had already been tried by my colleague and had proven unsuccessful.

Examination of the applmgmt log (tail -f /var/log/vmware/applmgmt/applmgmt.log) showed instances of failed file open operations each page refresh that led to the install stage being set to an incorrect stage (termed incorrect as it was known that the VCSA had fully completed its upgrade).

First file open failure:
INFO:vcenter.fileutil:read_json_file: [Errno 2] No such file or directory: '/var/log/firstboot/rpmInstall.json'
Followed shortly by a log entry "INFO:vcenter.deploy_status:adding phase rpminstall to pending list"

Second file open failure:
INFO:vcenter.fileutil:read_json_file: [Errno 2] No such file or directory: '/var/log/vmware/deploy/validate-status.json'
Followed shortly by a log entry "INFO:vcenter.deploy_status:adding phase validate to pending list"

Third file open failure:
INFO:vcenter.fileutil:read_json_file: [Errno 2] No such file or directory: '/var/log/firstboot/fbInstall.json'
Followed shortly by a log entry "INFO:vcenter.deploy_status:adding phase firstboot to pending list"

To me this would indicate normal behaviour, that the installation script uses these files as progress markers, however in our case the status was not being set in the expected place for the installation script as shown by this error in the log "ERROR:vcenter.deploy_status:DeploymentStatus.get_status: no current or last task found!!" being thrown followed by a stack trace:

ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.vcenter.deployment in get -
Traceback (most recent call last):
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
service_id, operation_id, input_value, ctx)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
method_result = iface.invoke(ctx, method_id, input_value)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
meth_output = method(**meth_args)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter_impl.py", line 24, in get
return ds.get_status()
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/deploy_status.py", line 1159, in get_status
assert False
AssertionError

As we could reliably determine that the VCSA had in fact completed its upgrade and was functioning correctly, and that the issues appeared to be isolated only to the VAMI, I manually created a valid 'rpmInstall.json' file in the expected location with vi

{
  "progress": 100,
  "question": null,
  "status": "success",
  "progress_message": {
    "localized": "",
    "translatable": ""
  },
  "error": null,
  "warning": [],
  "start_time": "2021-12-31T12:28:30.2321Z",
  "info": [],
  "end_time": "2021-12-31T12:38:30.2321Z"
}

Following the creation of this file the VAMI stopped throwing the "Error in method invocation", however it only progressed to the post upgrade/install firstboot page (setup, upgrade, migrate). This was not applicable, and since as before any attempt to access other links on the VAMI were redirected the the firstboot page it was not a desired outcome. To resolve this I manually created a valid 'fbInstall.json' file in the expected location with vi

{
    "progress": 100,
    "progress_message": {"id":
      "install.ciscommon.component.starting",
      "translatable": "Starting %(0)s...",
      "args": [
          "VMware Performance Charts"
       ],
       "localized": "Starting VMware Performance Charts..."
    },
    "status": "success",
    "info": [],
    "warning": [],
    "question": null,
    "error": null,
    "start_time": "2021-12-31T12:39:30.321Z",
    "end_time": "2021-12-31T12:59:30.321Z"
}

After this file was present the experience logging into the VAMI and subsequent navigation within the VAMI was as per expectation.

A reboot is not required to implement this workaround.

Please note that this has only been tested in a lab environment and for production environments you should be contacting GSS for support.

Further research has returned the following KB (https://kb.vmware.com/s/article/84259) Please refer to it as I believe that what was observed may just a different symptom of the cap first boot service "bug".