VMware Cloud Community
DionRow
Enthusiast
Enthusiast
Jump to solution

vcsa update to 7.0u1 Returning from install(), data conversion failed post install hook failed

I just upgraded to 7.0 u1 from 7.0 and it appears to be fine except the applmgmt gui keeps displaying this:

pastedImage_0.png

And appears to be stuck.

Yet All logs appear normal with no errors and all services are running.  vcenter says 7.0.1.:

root@vc6 [ /var/log/vmware ]# vpxd -v

VMware VirtualCenter 7.0.1 build-16858589

Running:

applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp

Stopped:

vmcam vmware-imagebuilder vmware-netdumper vmware-vcha

I see in the log /var/log/vmware/applmgmt/update_microservice.log the following:

2020-10-20 01:18:45,451 - 59346 -        update_script::_executePatchRunnerAndSupplyUserData: 127 -    ERROR - Patch command patch failed

2020-10-20 01:18:45,460 - 59346 -        update_script::                _reportFailure: 159 -    DEBUG - Error status has been already reported

2020-10-20 01:18:45,464 - 59346 -           extensions::                      __call__: 107 -    DEBUG - The component script returned 'None'

2020-10-20 01:18:45,523 - 17819 -           update_b2b::                _runScriptHook: 344 -    DEBUG - update script result file '{\n    "end_time": "2020-10-20T01:18:45.232Z",\n    "error": {\n        "componentKey": "last_component:Patch",\n        "detail": [\n            {\n                "id": "ur.services.start.text",\n                "localized": "Failed to start all services after successful patching.",\n                "translatable": "Failed to start all services after successful patching."\n            }\n        ],\n        "problemId": null,\n        "resolution": null\n    },\n    "info": [],\n    "progress": 47,\n    "progress_message": {\n        "componentKey": "last_component:Patch",\n        "detail": [\n            {\n                "id": "ur.services.start.text",\n                "localized": "Failed to start all services after successful patching.",\n                "translatable": "Failed to start all services after successful patching."\n            }\n        ],\n        "problemId": null,\n        "resolution": null\n    },\n    "question": null,\n    "start_time": "2020-10-20T00:41:48.003Z",\n    "status": "error",\n    "warning": [\n        {\n            "msg": {\n                "id": "ur.services.retry.start.text",\n                "localized": "Failed to start all services, will retry operation.",\n                "translatable": "Failed to start all services, will retry operation."\n            },\n            "problemId": null,\n            "resolution": null\n        }\n    ]\n}'

2020-10-20 01:18:45,537 - 17819 -           update_b2b::                _runScriptHook: 362 -    DEBUG - Returning notifications {'errors': [{'id': 'com.vmware.apliance.noid', 'time': '2020-10-20T01:18:45.537Z', 'message': {'id': 'ur.services.start.text', 'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}], 'info': [], 'warnings': [{'id': 'com.vmware.appliance.noid', 'time': '2020-10-20T01:18:45.536Z', 'message': {'id': 'ur.services.retry.start.text', 'default_message': 'Failed to start all services, will retry operation.', 'args': []}, 'resolution': None}]} from _runScriptHook

2020-10-20 01:18:45,537 - 17819 -           update_b2b::                       install:2265 -    DEBUG - PostInstall errors: [{'id': 'com.vmware.apliance.noid', 'time': '2020-10-20T01:18:45.537Z', 'message': {'id': 'ur.services.start.text', 'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}]

2020-10-20 01:18:45,544 - 17819 -         task_manager::                        update:  79 -    DEBUG - UpdateTask: status=FAILED, progress=80, message={'id': 'com.vmware.appliance.postinstall_errors', 'default_message': 'Data conversion/Post install hook failed', 'args': []}

2020-10-20 01:18:45,664 - 17819 -          dbfunctions::                    executeDML:  54 -    DEBUG - Executing Query {INSERT INTO progress VALUES(NULL, ?,         DATETIME('now'), ?, ?, ?)} with parameters ('failed', 'Installation failed', 100, 95)

2020-10-20 01:18:45,684 - 17819 -           update_b2b::                       install:2272 -    DEBUG - Returning from install(), data conversion failed

Did the upgrade work?  It apear to have but the dialog will not go away.  If it is okay how can I reset this so the dialog goes away and the alliance management interface UI is usable again?

I have a snapshot and could rollback.

Regards,

dion

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon
1 Solution

Accepted Solutions
DionRow
Enthusiast
Enthusiast
Jump to solution

I was able to finally get it to succeed by giving it 8vcpu and 16gb ram.  (it only had 2vcpu and 12 before)

It has however still not told me its done in the 5480 UI.  It gives me the error: "Update installation is in progress" when trying to login in.

Logs show this:

2020-10-21T01:39:46.206 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create

2020-10-21T01:39:46.214 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

2020-10-21T01:39:46.215 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation

2020-10-21T01:39:46.216 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.

2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'

2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []

2020-10-21T01:39:46.231 [62589]INFO:vmware.appliance.vapi.auth:Reset pam tally for root

2020-10-21T01:39:46.232 [62589]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login           Failures Latest failure     From\nroot                0    \n'

2020-10-21T01:39:46.244 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.local_accounts, operation_id: get

2020-10-21T01:39:46.244 [62589]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']

2020-10-21T01:39:46.245 [62589]DEBUG:root:Validated user privileges in localstore or SSO

2020-10-21T01:39:46.246 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

2020-10-21T01:39:46.246 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation

2020-10-21T01:39:46.247 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.

2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'

2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:Only [] block []

and the file /etc/applmgmt/appliance/software_update_state.conf says :

{

    "state": "INSTALL_IN_PROGRESS",

    "version": "7.0.1.00000",

    "latest_query_time": "2020-10-20T00:08:43Z",

    "operation_id": "/storage/core/software-update/stage_install_operation"

}

I am assuming changing this state value to something else will allow log in again.

Any ideas what "state" should be when install is finished?

Rebooting it does not seem to reset this value.

dion

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon

View solution in original post

9 Replies
Lalegre
Virtuoso
Virtuoso
Jump to solution

Hey DionRow​,

I can see this:

'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}]

It seems that it failed to start some services but you show that all the services are up, what if you try to reboot the vCenter?

Reply
0 Kudos
DionRow
Enthusiast
Enthusiast
Jump to solution

Tried rebooting also and the result is the same

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon
Reply
0 Kudos
DionRow
Enthusiast
Enthusiast
Jump to solution

Rolled back snapshot and tried in ssh command line and get:

Command> software-packages install --iso --acceptEulas

[2020-10-20T17:08:18.294] : ISO mounted successfully

[2020-10-20T17:08:20.294] : Target VCSA version = 7.0.1.00000

[2020-10-20 17:08:20,438] : Running requirements script.....

[2020-10-20T17:08:47.294] : Evaluating packages to stage...

[2020-10-20T17:08:47.294] : Verifying staging area

[2020-10-20T17:08:47.294] : ISO unmounted successfully

[2020-10-20T17:08:47.294] : For the first instance of the identity domain, this is the password given to the Administrator account.  Otherwise, this is the password of the Administrator account of the replication partner.

Enter Single Sign-On administrator password:

[2020-10-20T17:09:03.294] : Validating software update payload

[2020-10-20 17:09:03,657] : Running validate script.....

[2020-10-20T17:09:12.294] : Validation successful

[2020-10-20 17:09:12,703] : Copying software packages  [2020-10-20T17:09:12.294] : ISO mounted successfully

207/207

[2020-10-20T17:12:31.294] : ISO unmounted successfully

[2020-10-20 17:12:32,007] : Running system-prepare script.....

[2020-10-20 17:12:45,273] : Running test transaction ....

[2020-10-20 17:12:50,324] : Running prepatch script.....

[2020-10-20 17:16:35,135] : Upgrading software packages ....

[2020-10-20T17:25:09.294] : Setting appliance version to 7.0.1.00000 build 16860138

[2020-10-20 17:25:09,660] : Running patch script.....

[2020-10-20T18:00:01.294] : Patch command patch failed

[2020-10-20T18:00:05.294] :

        Mismatch:

                summary: Failed to start all services after successful patching.

[2020-10-20T18:00:05.294] : The following warnings have been found:

['\tWarning: \n\t\tsummary: Failed to start all services, will retry operation.\n']

Command>

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon
DionRow
Enthusiast
Enthusiast
Jump to solution

I was able to finally get it to succeed by giving it 8vcpu and 16gb ram.  (it only had 2vcpu and 12 before)

It has however still not told me its done in the 5480 UI.  It gives me the error: "Update installation is in progress" when trying to login in.

Logs show this:

2020-10-21T01:39:46.206 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create

2020-10-21T01:39:46.214 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

2020-10-21T01:39:46.215 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation

2020-10-21T01:39:46.216 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.

2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'

2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []

2020-10-21T01:39:46.231 [62589]INFO:vmware.appliance.vapi.auth:Reset pam tally for root

2020-10-21T01:39:46.232 [62589]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login           Failures Latest failure     From\nroot                0    \n'

2020-10-21T01:39:46.244 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.local_accounts, operation_id: get

2020-10-21T01:39:46.244 [62589]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']

2020-10-21T01:39:46.245 [62589]DEBUG:root:Validated user privileges in localstore or SSO

2020-10-21T01:39:46.246 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

2020-10-21T01:39:46.246 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation

2020-10-21T01:39:46.247 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.

2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'

2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:Only [] block []

and the file /etc/applmgmt/appliance/software_update_state.conf says :

{

    "state": "INSTALL_IN_PROGRESS",

    "version": "7.0.1.00000",

    "latest_query_time": "2020-10-20T00:08:43Z",

    "operation_id": "/storage/core/software-update/stage_install_operation"

}

I am assuming changing this state value to something else will allow log in again.

Any ideas what "state" should be when install is finished?

Rebooting it does not seem to reset this value.

dion

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon
DionRow
Enthusiast
Enthusiast
Jump to solution

Never mind, found kb:

VMware Knowledge Base

solved issue.  remove file and restart applmgmt

dion

-- dion VMware Systems Admin vExpert VMUG Leader Saskatoon
shind0chaa
Contributor
Contributor
Jump to solution

I ran in to this too in my lab, my VCSA mem and CPU were right sized though

Tried below and was able to get rid of error and use VAMI interface.

pastedImage_0.png

Eventually rolled back the update as versions were showing different in vSphereUI and VAMI / vpxd and couldn't start few services at all (updatemanager, vstats and wcp)

😕

toprockRS
Enthusiast
Enthusiast
Jump to solution

Just want to add my two cent´s here, had the same issue for multiple updates but seem to have fixed it now.

https://www.mydailyitlessons.net/2021/01/vcenter-update-stuck-data.html

 

Reply
0 Kudos
poojadwivedi979
Contributor
Contributor
Jump to solution

This worked for me too.. I have increased vcpus to 4 and resumed and worked well.

Reply
0 Kudos
theexit
Contributor
Contributor
Jump to solution

This was working for me....

  • Connect to the vcsa shell
  • Check the update state

cat /etc/applmgmt/appliance/software_update_state.conf

{
"state": "INSTALL_FAILED",
"version": "7.0.2.00100",
"latest_query_time": "2021-05-14T06:26:30Z",
"operation_id": "/storage/core/software-update/install_operation"

  • Backup the update state config

cp /etc/applmgmt/appliance/software_update_state.conf /etc/applmgmt/appliance/software_update_state.conf.bak

  • Stop the Management Service

service-control --stop applmgmt

Operation not cancellable. Please wait for it to finish...
Performing stop operation on service applmgmt...
Successfully stopped service applmgmt

  • Delete the update state config file

rm /etc/applmgmt/appliance/software_update_state.conf

  • Start the Management Service

service-control --start applmgmt

Operation not cancellable. Please wait for it to finish...
Performing start operation on service applmgmt...
Successfully started service applmgmt

Thx to @shind0chaa