services updatemgr failed to start, following upgrade to vCenter 8

[Arnaud_L]

I upgraded my vCenter Appliance 7 to 8 and since then, i'm not able to backup the configuration anymore (evertyhing else seems to work nicely).

The import of the data did fail, but mainly because of the missing 2nd adapter on the new appliance in v8 (which I corrected manually once the upgrade process finished).

My issue raise when I want to backup my configuration, I got the following error :

Invalid vCenter Server Status: All required services are not up! Stopped services: 'updatemgr'.

Trying to start the service from the Shell, give me this :

root@vcenter02 [ ~ ]# systemctl start updatemgr
Job for updatemgr.service failed because the control process exited with error code.
See "systemctl status updatemgr.service" and "journalctl -xe" for details.

root@vcenter02 [ ~ ]# systemctl status updatemgr.service
● updatemgr.service - Service for updatemgr service
   Loaded: loaded (/lib/systemd/system/updatemgr.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2023-06-06 10:55:10 UTC; 5s ago
  Process: 13183 ExecStopPost=/usr/bin/sh -c /usr/lib/vmware-vmon/vmon-cli --inform-stopped updatemgr --pid ${MAINPID} --return-code ${EXIT_STATUS} (code=exited, status=4)
  Process: 13180 ExecStartPost=/usr/bin/sh -c /usr/lib/vmware-vmon/vmon-cli --inform-started updatemgr --pid ${MAINPID} --return-code 0 (code=exited, status=4)
  Process: 13179 ExecStart=/usr/bin/sh -l -c /usr/lib/vmware-updatemgr/bin/vmware-updatemgr /usr/lib/vmware-updatemgr/bin/vci-integrity.xml (code=killed, signal=TERM)
  Process: 13135 ExecStartPre=/usr/bin/sh -l -c ${VMWARE_PYTHON_BIN} /usr/lib/vmware-updatemgr-root/sbin/updatemgr-vmon-prestart.py; +sh -c /usr/lib/vmware-vmon/vmon-cli --inform-prestarted updatemgr --pid 0 > Main PID: 13179 (code=killed, signal=TERM)

Jun 06 10:55:09 vcenter02.domain.tld systemd[1]: Starting Service for updatemgr service...
Jun 06 10:55:10 vcenter02.domain.tld systemd[1]: updatemgr.service: Control process exited, code=exited status=4
Jun 06 10:55:10 vcenter02.domain.tld systemd[1]: updatemgr.service: Control process exited, code=exited status=4
Jun 06 10:55:10 vcenter02.domain.tld systemd[1]: updatemgr.service: Failed with result 'exit-code'.
Jun 06 10:55:10 vcenter02.domain.tld systemd[1]: Failed to start Service for updatemgr service.

I will be happy to provide more information, but can't find any relevant one on /var/log/vmware/vmware-updatemgr/

[mannharry]

Hello ,

Kindly check this 

https://kb.vmware.com/s/article/2151622?lang=en_US

Regards

Harry

[Arnaud_L]

Thanks Harry,

On the Mob page, I can see this, on the properties section : 

extensionList["com.vmware.vcIntegrity"] Extension

I assume then that it's not missing ?!

I anyhow tried to add the extension using the InvokeMethod and it doesn't work.

Most of the log file path are not existing, and the command "refresh-certs" command doesn't really work on my version (8) :

root@vcenter02 [ ~ ]# /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py refresh-certs
usage: updatemgr-utility.py [-h] {register-vc,reset-db,config,install-cert,uninstall-cert} ...
updatemgr-utility.py: error: invalid choice: 'refresh-certs' (choose from 'register-vc', 'reset-db', 'config', 'install-cert', 'uninstall-cert')

root@vcenter02 [ ~ ]# /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py
Exception while executing updatemgr-utility. err: 'Namespace' object has no attribute 'func'

[mannharry]

can you share the update manager log file

[Arnaud_L]

Where / which file exactly do you may need ?

root@vcenter02 [ /var/log/vmware/vmware-updatemgr ]# ls -l
total 160
-rw-r--r-- 1 updatemgr updatemgr 14352 Jun  6 06:32 refreshCerts-utility.log
-rw-r--r-- 1 root      root        125 Jun  6 11:38 updatemgr-utility.log
-rw-r--r-- 1 root      root        179 Jun  6 10:55 updatemgr-vmon.log
-rw------- 1 updatemgr updatemgr    41 Jun  5 16:06 updatemgr-vmon.log-0.stderr
-rw------- 1 updatemgr updatemgr    41 Jun  5 16:06 updatemgr-vmon.log-0.stdout
-rw------- 1 updatemgr updatemgr    41 Jun  5 15:56 updatemgr-vmon.log-1.stderr
-rw------- 1 updatemgr updatemgr    41 Jun  5 15:56 updatemgr-vmon.log-1.stdout
-rw------- 1 updatemgr updatemgr    42 Jun  5 15:52 updatemgr-vmon.log-2.stderr
-rw------- 1 updatemgr updatemgr    42 Jun  5 15:52 updatemgr-vmon.log-2.stdout
-rw------- 1 updatemgr updatemgr    42 Jun  5 15:40 updatemgr-vmon.log-3.stderr
-rw------- 1 updatemgr updatemgr    42 Jun  5 15:40 updatemgr-vmon.log-3.stdout
-rw------- 1 updatemgr updatemgr    41 Jun  5 14:39 updatemgr-vmon.log-4.stderr
-rw------- 1 updatemgr updatemgr    41 Jun  5 14:39 updatemgr-vmon.log-4.stdout
-rw------- 1 updatemgr updatemgr    42 Jun  6 06:32 updatemgr-vmon.log.stderr
-rw------- 1 updatemgr updatemgr    42 Jun  6 06:32 updatemgr-vmon.log.stdout
-rw-r--r-- 1 root      root      29386 Jun  6 10:55 updatemgr-vmon-prestart.log
-rw-r--r-- 1 updatemgr updatemgr  1231 Jun  5 15:52 vmware-vci-vciInstallUtils-10.log.gz
-rw-r--r-- 1 updatemgr updatemgr   973 Jun  5 15:56 vmware-vci-vciInstallUtils-11.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1262 Jun  5 15:56 vmware-vci-vciInstallUtils-12.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1000 Jun  5 16:06 vmware-vci-vciInstallUtils-13.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1278 Jun  5 16:06 vmware-vci-vciInstallUtils-14.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1053 Jun  6 06:32 vmware-vci-vciInstallUtils-15.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1227 Jun  6 06:32 vmware-vci-vciInstallUtils-16.log.gz
-rw-r--r-- 1 updatemgr updatemgr  4349 Jun  6 06:32 vmware-vci-vciInstallUtils-17.log
-rw-r--r-- 1 updatemgr updatemgr  1196 Jun  5 15:40 vmware-vci-vciInstallUtils-8.log.gz
-rw-r--r-- 1 updatemgr updatemgr  1008 Jun  5 15:52 vmware-vci-vciInstallUtils-9.log.gz
lrwxrwxrwx 1 updatemgr updatemgr    66 Jun  6 06:32 vmware-vci-vciInstallUtils.log -> /var/log/vmware/vmware-updatemgr/vmware-vci-vciInstallUtils-17.log
drwx------ 2 updatemgr updatemgr  4096 Jun  6 06:32 vum-server

On the meantime, during the service start, only this log is getting updated :

==> updatemgr-vmon-prestart.log <==
2023-06-06 13:16:33,065 INFO updatemgr-vmon-prestart[139977673033536] [prestartUtils 102] command: ['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-ls-location', '--server-name', 'localhost']
2023-06-06 13:16:33,096 INFO updatemgr-vmon-prestart[139977673033536] [prestartUtils 105] rc: 0, stdout: https://vcenter02.domain.tld:443/lookupservice/sdk
, stderr:
2023-06-06 13:16:33,096 INFO updatemgr-vmon-prestart[139977673033536] [prestartUtils 111] lsUrl: https://vcenter02.domain.tld:443/lookupservice/sdk
2023-06-06 13:16:33,154 INFO updatemgr-vmon-prestart[139977673033536] [prestartUtils 218] Downloading trusted certs from http://localhost:7080/idm/tenant/vsphere.domain.tld/certificates?scope=TENANT&granularity=LEAF
2023-06-06 13:16:33,174 INFO updatemgr-vmon-prestart[139977673033536] [vumUtils 820] Executing cmd : /usr/bin/python /usr/lib/vmware/site-packages/vsr/db_tool/vpg_sync_registry.py --registry /etc/vmware/service-registry/updatemgr-sub-registry.yaml -U postgres --host /var/run/vpostgres --operation install --instance vpostgres --stage roles --stage grants --stage owners --database VCDB --debug
2023-06-06 13:16:33,352 INFO updatemgr-vmon-prestart[139977673033536] [vumUtils 823] rc: 0, stdout: , stderr: 2023-06-06 13:16:33,338 DEBUG : Initializing cache
2023-06-06 13:16:33,338 DEBUG : Cache found in /var/cache/vsr/dbms/vpg_sync_registry_cache.json, loading
2023-06-06 13:16:33,338 DEBUG : Checking in cache if work is needed.
Files:
['/etc/vmware/service-registry/updatemgr-sub-registry.yaml']
Stages:
['revoke_role', 'drop_role', 'create_role', 'alter_role', 'grant_role', 'grant_role', 'grant_obj', 'owner_obj']
2023-06-06 13:16:33,338 DEBUG : Computing hash for /etc/vmware/service-registry/updatemgr-sub-registry.yaml
2023-06-06 13:16:33,338 DEBUG : Hash for /etc/vmware/service-registry/updatemgr-sub-registry.yaml: 500b24f4316fd40cc668810a8c27e7c50538cad5
2023-06-06 13:16:33,338 DEBUG : All stages for all files cached

2023-06-06 13:16:33,352 INFO updatemgr-vmon-prestart[139977673033536] [vumUtils 1322] Successfully created VCDB vumuser role.

[mannharry]

Try restarting the service and once it failed , get the complete file .

File under 

/var/log/vmware/vmware-updatemgr/vum-server

 vmware-vum-server.log

-Harry

[Arnaud_L]

Either Starting/Restarting/SToping the service doesn't change the content of the log file, here attached.

I guess it's all related to the "failed" migration

[SureshKumarMuth]

Looks like something is wrong at db level.

2023-06-06T14:08:34.095Z error vmware-vum-server[07441] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 42703:1^M
2023-06-06T14:08:34.095Z error vmware-vum-server[07441] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (42703) - ERROR: column "bulletin_severity" does not exist;^M
--> Error while executing the query" is returned when executing SQL statement "SELECT id, type, title, meta_uid, generic, category, component, service_pack, security_fix, impact, hyperlink, patchconturl, signatureurl, locale, description, vendor, releasedate, downloadtime, path, vendor_uid, bugtraq_uid, cve_uid, deleted, run, exec_platform, release_build, bundle_type, installation_impacts, component_name, code_version, product_version, release, metadataFileId, hidden, recalled, hostupdate_category, bulletin_severity, bulletin_category, bulletin_urgency FROM VCI_UPDATES WHERE deleted ="^M
2023-06-06T14:08:34.095Z error vmware-vum-server[07441] [Originator@6876 sub=VcIntegrity] [vcIntegrity 764] Error in start VUM server^M
2023-06-06T14:08:34.095Z info vmware-vum-server[07441] [Originator@6876 sub=VcIntegrityPlugin] [plugin 270] VcIntegrityPlugin Start^M

a column named  "bulletin_severity" is missing.

Not sure if this is a bug. I could not find any KB article for this issue. May be you can raise a ticket with vmware and let them check. I doubt VUM DB reset will help here.

[Arnaud_L]

Thanks Suresh,

As I'm (currently at least) not able to open a support ticket, do you know if there is a way to re-do the Stage2 of the migration ? (if that may be of any help) ?

[SureshKumarMuth]

Could you please clarify about the upgrade task ?

I upgraded my vCenter Appliance 7 to 8 and since then, i'm not able to backup the configuration anymore (everything else seems to work nicely).

The import of the data did fail, but mainly because of the missing 2nd adapter on the new appliance in v8 (which I corrected manually once the upgrade process finished). 

-- if the import of the data failed how did the upgrade task completed successfully ? and what is 2nd adapter. you mean NIC ? VCSA by default contains only one NIC.  What did you do to fix it.

-- did the prevalidation before upgrade was clear?

--after upgrading you got the message as "upgrade completed successfully ". I suspect the upgrade was not fully done.

My issue raise when I want to backup my configuration, I got the following error :

Invalid vCenter Server Status: All required services are not up! Stopped services: 'updatemgr'.

How are you backing up configuration ? 

I dont think we can redo the stage 2 alone once the appliance is up on the desired version. You may have to restart the whole process if you have your old 7.x , you can power it on again after shutting down the new VC 8.x and retry the upgrade. Please note that if you try to revert, you may lose some data which was generated after the 8.x VC came into live (example; performance stats, tasks and events etc ...)

[Arnaud_L · Accepted solution]

Hi Suresh,

I'm trying again the migration.

Phase 1 was without issue, the pre-check of the phase 2 only show those warnings in attachement. See the latest warning regarding the 2nd NIC

The 2nd attachement, show the error I got at the first party of phase 2 (the one I was refering to)

the third attachement, show the warning at the end of phase 2.

Last EDIT: Finally, re-running the migration succeed and I don't face the issue anymore.

Thanks everyone for the help.

[SureshKumarMuth]

Hi @Arnaud_L 

Good to know that the issue is resolved. 

Since you are using second nic in VCSA. Could you please if you have any document for reference or steps to configure second nic in VCSA ? I saw someone asked for it in the recent past. I can see many use cases for this and I think still it is a manual process. It would be a great help for others if they have some reference. Currently, I dont have any lab to try that :). 

Regards,

Suresh

[Arnaud_L]

Hi Suresh,

Actually, if the setup of the 2nd NIC (called NIC1 on the screenshot attached) has to be set manually, on my case, it automatically create it, using the good (default) Port Group "VM Network" (as it's hosted on an ESXi).

so, not so much of a configuration to do at the end.

When the migration suceed (the last time I did it then), the temporary IP I selected for the new vCenter appliance, took automatically the same configuration as my previous one, so nothing to do in regards of the network.