I just upgraded to 7.0 u1 from 7.0 and it appears to be fine except the applmgmt gui keeps displaying this:
And appears to be stuck.
Yet All logs appear normal with no errors and all services are running. vcenter says 7.0.1.:
root@vc6 [ /var/log/vmware ]# vpxd -v
VMware VirtualCenter 7.0.1 build-16858589
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
Stopped:
vmcam vmware-imagebuilder vmware-netdumper vmware-vcha
I see in the log /var/log/vmware/applmgmt/update_microservice.log the following:
2020-10-20 01:18:45,451 - 59346 - update_script::_executePatchRunnerAndSupplyUserData: 127 - ERROR - Patch command patch failed
2020-10-20 01:18:45,460 - 59346 - update_script:: _reportFailure: 159 - DEBUG - Error status has been already reported
2020-10-20 01:18:45,464 - 59346 - extensions:: __call__: 107 - DEBUG - The component script returned 'None'
2020-10-20 01:18:45,523 - 17819 - update_b2b:: _runScriptHook: 344 - DEBUG - update script result file '{\n "end_time": "2020-10-20T01:18:45.232Z",\n "error": {\n "componentKey": "last_component:Patch",\n "detail": [\n {\n "id": "ur.services.start.text",\n "localized": "Failed to start all services after successful patching.",\n "translatable": "Failed to start all services after successful patching."\n }\n ],\n "problemId": null,\n "resolution": null\n },\n "info": [],\n "progress": 47,\n "progress_message": {\n "componentKey": "last_component:Patch",\n "detail": [\n {\n "id": "ur.services.start.text",\n "localized": "Failed to start all services after successful patching.",\n "translatable": "Failed to start all services after successful patching."\n }\n ],\n "problemId": null,\n "resolution": null\n },\n "question": null,\n "start_time": "2020-10-20T00:41:48.003Z",\n "status": "error",\n "warning": [\n {\n "msg": {\n "id": "ur.services.retry.start.text",\n "localized": "Failed to start all services, will retry operation.",\n "translatable": "Failed to start all services, will retry operation."\n },\n "problemId": null,\n "resolution": null\n }\n ]\n}'
2020-10-20 01:18:45,537 - 17819 - update_b2b:: _runScriptHook: 362 - DEBUG - Returning notifications {'errors': [{'id': 'com.vmware.apliance.noid', 'time': '2020-10-20T01:18:45.537Z', 'message': {'id': 'ur.services.start.text', 'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}], 'info': [], 'warnings': [{'id': 'com.vmware.appliance.noid', 'time': '2020-10-20T01:18:45.536Z', 'message': {'id': 'ur.services.retry.start.text', 'default_message': 'Failed to start all services, will retry operation.', 'args': []}, 'resolution': None}]} from _runScriptHook
2020-10-20 01:18:45,537 - 17819 - update_b2b:: install:2265 - DEBUG - PostInstall errors: [{'id': 'com.vmware.apliance.noid', 'time': '2020-10-20T01:18:45.537Z', 'message': {'id': 'ur.services.start.text', 'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}]
2020-10-20 01:18:45,544 - 17819 - task_manager:: update: 79 - DEBUG - UpdateTask: status=FAILED, progress=80, message={'id': 'com.vmware.appliance.postinstall_errors', 'default_message': 'Data conversion/Post install hook failed', 'args': []}
2020-10-20 01:18:45,664 - 17819 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {INSERT INTO progress VALUES(NULL, ?, DATETIME('now'), ?, ?, ?)} with parameters ('failed', 'Installation failed', 100, 95)
2020-10-20 01:18:45,684 - 17819 - update_b2b:: install:2272 - DEBUG - Returning from install(), data conversion failed
Did the upgrade work? It apear to have but the dialog will not go away. If it is okay how can I reset this so the dialog goes away and the alliance management interface UI is usable again?
I have a snapshot and could rollback.
Regards,
dion
I was able to finally get it to succeed by giving it 8vcpu and 16gb ram. (it only had 2vcpu and 12 before)
It has however still not told me its done in the 5480 UI. It gives me the error: "Update installation is in progress" when trying to login in.
Logs show this:
2020-10-21T01:39:46.206 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create
2020-10-21T01:39:46.214 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2020-10-21T01:39:46.215 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation
2020-10-21T01:39:46.216 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.
2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []
2020-10-21T01:39:46.231 [62589]INFO:vmware.appliance.vapi.auth:Reset pam tally for root
2020-10-21T01:39:46.232 [62589]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login Failures Latest failure From\nroot 0 \n'
2020-10-21T01:39:46.244 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.local_accounts, operation_id: get
2020-10-21T01:39:46.244 [62589]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']
2020-10-21T01:39:46.245 [62589]DEBUG:root:Validated user privileges in localstore or SSO
2020-10-21T01:39:46.246 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2020-10-21T01:39:46.246 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation
2020-10-21T01:39:46.247 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.
2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:Only [] block []
and the file /etc/applmgmt/appliance/software_update_state.conf says :
{
"state": "INSTALL_IN_PROGRESS",
"version": "7.0.1.00000",
"latest_query_time": "2020-10-20T00:08:43Z",
"operation_id": "/storage/core/software-update/stage_install_operation"
}
I am assuming changing this state value to something else will allow log in again.
Any ideas what "state" should be when install is finished?
Rebooting it does not seem to reset this value.
dion
Hey DionRow,
I can see this:
'default_message': 'Failed to start all services after successful patching.', 'args': []}, 'resolution': None}]
It seems that it failed to start some services but you show that all the services are up, what if you try to reboot the vCenter?
Tried rebooting also and the result is the same
Rolled back snapshot and tried in ssh command line and get:
Command> software-packages install --iso --acceptEulas
[2020-10-20T17:08:18.294] : ISO mounted successfully
[2020-10-20T17:08:20.294] : Target VCSA version = 7.0.1.00000
[2020-10-20 17:08:20,438] : Running requirements script.....
[2020-10-20T17:08:47.294] : Evaluating packages to stage...
[2020-10-20T17:08:47.294] : Verifying staging area
[2020-10-20T17:08:47.294] : ISO unmounted successfully
[2020-10-20T17:08:47.294] : For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.
Enter Single Sign-On administrator password:
[2020-10-20T17:09:03.294] : Validating software update payload
[2020-10-20 17:09:03,657] : Running validate script.....
[2020-10-20T17:09:12.294] : Validation successful
[2020-10-20 17:09:12,703] : Copying software packages [2020-10-20T17:09:12.294] : ISO mounted successfully
207/207
[2020-10-20T17:12:31.294] : ISO unmounted successfully
[2020-10-20 17:12:32,007] : Running system-prepare script.....
[2020-10-20 17:12:45,273] : Running test transaction ....
[2020-10-20 17:12:50,324] : Running prepatch script.....
[2020-10-20 17:16:35,135] : Upgrading software packages ....
[2020-10-20T17:25:09.294] : Setting appliance version to 7.0.1.00000 build 16860138
[2020-10-20 17:25:09,660] : Running patch script.....
[2020-10-20T18:00:01.294] : Patch command patch failed
[2020-10-20T18:00:05.294] :
Mismatch:
summary: Failed to start all services after successful patching.
[2020-10-20T18:00:05.294] : The following warnings have been found:
['\tWarning: \n\t\tsummary: Failed to start all services, will retry operation.\n']
Command>
I was able to finally get it to succeed by giving it 8vcpu and 16gb ram. (it only had 2vcpu and 12 before)
It has however still not told me its done in the 5480 UI. It gives me the error: "Update installation is in progress" when trying to login in.
Logs show this:
2020-10-21T01:39:46.206 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create
2020-10-21T01:39:46.214 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2020-10-21T01:39:46.215 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation
2020-10-21T01:39:46.216 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.
2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
2020-10-21T01:39:46.218 [62589]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []
2020-10-21T01:39:46.231 [62589]INFO:vmware.appliance.vapi.auth:Reset pam tally for root
2020-10-21T01:39:46.232 [62589]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login Failures Latest failure From\nroot 0 \n'
2020-10-21T01:39:46.244 [62589]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.local_accounts, operation_id: get
2020-10-21T01:39:46.244 [62589]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']
2020-10-21T01:39:46.245 [62589]DEBUG:root:Validated user privileges in localstore or SSO
2020-10-21T01:39:46.246 [62589]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2020-10-21T01:39:46.246 [62589]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation
2020-10-21T01:39:46.247 [62589]DEBUG:vmware.appliance.update.update_state:Updating Task progress with B2B Framework progress.
2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
2020-10-21T01:39:46.248 [62589]DEBUG:vmware.vherd.transport.vapi:Only [] block []
and the file /etc/applmgmt/appliance/software_update_state.conf says :
{
"state": "INSTALL_IN_PROGRESS",
"version": "7.0.1.00000",
"latest_query_time": "2020-10-20T00:08:43Z",
"operation_id": "/storage/core/software-update/stage_install_operation"
}
I am assuming changing this state value to something else will allow log in again.
Any ideas what "state" should be when install is finished?
Rebooting it does not seem to reset this value.
dion
Never mind, found kb:
solved issue. remove file and restart applmgmt
dion
I ran in to this too in my lab, my VCSA mem and CPU were right sized though
Tried below and was able to get rid of error and use VAMI interface.
Eventually rolled back the update as versions were showing different in vSphereUI and VAMI / vpxd and couldn't start few services at all (updatemanager, vstats and wcp)
😕
Just want to add my two cent´s here, had the same issue for multiple updates but seem to have fixed it now.
https://www.mydailyitlessons.net/2021/01/vcenter-update-stuck-data.html
This worked for me too.. I have increased vcpus to 4 and resumed and worked well.
This was working for me....
cat /etc/applmgmt/appliance/software_update_state.conf
{
"state": "INSTALL_FAILED",
"version": "7.0.2.00100",
"latest_query_time": "2021-05-14T06:26:30Z",
"operation_id": "/storage/core/software-update/install_operation"
cp /etc/applmgmt/appliance/software_update_state.conf /etc/applmgmt/appliance/software_update_state.conf.bak
service-control --stop applmgmt
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service applmgmt...
Successfully stopped service applmgmt
rm /etc/applmgmt/appliance/software_update_state.conf
service-control --start applmgmt
Operation not cancellable. Please wait for it to finish...
Performing start operation on service applmgmt...
Successfully started service applmgmt
Thx to @shind0chaa