VIO 5.1 Patch hangs

[jet1981]

Has anyone had issues with patching 5.0 to 5.1? Our installation keeps hanging at "TASK [nova-api : run online data migrations]" for many hours at a time. I have let it go for over 6 hours but all I see is viocli maxed out at 100% CPU usage on the OMS and on controller01 the nova-manage process maintains a solid 60-70% CPU usage.

In the nova-manage.log this is the last entry:

DEBUG migrate.versioning.repository [-] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'nova_api'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))]) __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:83

I realize this can take a long time, but 6+ hours? There isn't anything helpful inthe viocli.log either just reating entries for:

2019-05-09 15:30:45,330 urllib3.connectionpool DEBUG    Resetting dropped connection: 127.0.0.1

2019-05-09 15:30:45,336 urllib3.connectionpool DEBUG    http://127.0.0.1:48620 "GET /runs/767fb729-f909-4283-b202-33569cfe9237 HTTP/1.1" 200 574

Is this amount of time normal? Any help would be appreciated.

[jet1981 · Accepted solution]

For anyone else that stumbles upon this.

This is a known issue for 5.0 to 5.1 upgrades. The dev team is aware of it and is working on a fix according to support rep I spoke with today. Also the revert process is known to be an issue as well. The revert process ends up with Ansible plays for 5.1 that try to start services like Barbican (even though 5.0 doesn't have it) and bombs out. Leaving your deployment in an "Configuration Error" state. Once the revert process has bombed out, we were able to restore the OMS from the pre-upgrade snapshot in vCenter (you did take the snapshot right?), manually set the services to 'stopped' in the OMS database, then start the deployment from the viocli. Once the control plane was started, we were able to complete the"Roll back VIO patch" step 4 (stop vSphere Client service, delete residual files, and restart the service). After that, reconnecting the OMS to vCenter was all that was needed and we were back to a green state. This worked in our case so YMMV...

Hope this helps someone!