vCenter Server 7.0 update 2d failing to complete with "Exception occured in postInstallHook"

[eChuck]

Is there any advice for resolving the problem with the latest vCenter 7.0 update to 2d (build 18455184) where the update installer fails in the "Converting data as part of post install" section (80% complete) with this error message:

Exception occured [sic] in postInstallHook

The only option presented to the user is to "Resume," but this merely runs through the installation process again, winding up in the "Converting data..." section, where it hangs for considerable time before again throwing the error noted  above. 

vCenter can be accessed, and appears to be working. It reports its build # as 18455184, but reports "Updates available."

While vCenter is accessible, the VAMI web interface is not. I am able to ssh as root into the VCSA VM, but the login banner shows the version as 7.0.2.00400 instead of the expected 7.0.2.00500. 

Before taking any more drastic measures, I hope to get some advice from others who might have encountered this problem and found a resolution.

Thank you...

[erikgraa]

I am getting the same error upgrading to vCloud Foundation 4.3.1 for the workload domain. Management domain vCenter upgraded just fine to u2d.

[apsec-it]

have you found a solution for this problem?

[manadrain]

I do not know on your specific issue but I have had similar problems when trying to update VCenter Appliance 7.0X from the GUI in the past.  I have taken the approach of taking a snapshot of my vCenter (don't forget to do this step), downloading the ISO FP and mounting it on the VCenter Server.  Enable SSH on the VCenter Appliance and run the upgrade from the command line:

software-packages stage --iso

software-packages list --staged

software-packages install --staged

These instructions are listed in the details section when you download the FP ISO.  Also be sure to read the details in full because I know in the last patch it stated that the VCenter Server Appliance didn't need to be rebooted (7.0.2.00400). However they made a note that for the settings to apply you had to reboot the VCenter Appliance.

For my last three patches I have had this run successfully using this approach.  This morning I upgrade to 7.0.2.00500 and it appears that everything went as planned.

I don't know if this will help or not but to fix the issue with the Vcenter staying in a loop after a partial upgrade I have had to clear a upgrade state file directly on the VCenter appliance.  If I recall this I had to take similar steps to what is listed in the following KB

https://kb.vmware.com/s/article/67179

I have also had an issue with a failed upgrade of VCenter Appliance not able to start all services after the upgrade.  The Services that would not start I believe was related with VMware vSphere Update Manager.  This issue was able to be resolved by resetting the VCenter Update Manager Database.  If I recall the KB was what I have below.

https://kb.vmware.com/s/article/2147284

I have also had to restore a VCenter Server Appliance from backup with a failed upgrade.   I would highly recommend that you have this setup and working prior.  Also just from some experience from having to restore a VCenter appliance from a file backup if you have made any changes to the disk size you need to expand the disks to the same size on the restore.  The VCenter Appliance has a script that will expand the disks once this is done in the GUI.  If I recall I used the following KB to do this.

https://kb.vmware.com/s/article/2145603

Again I don't know your specific situation and each case in the past I have opened a case with VMware Support.  I am just providing my recommendations based on my past experiences and hope that this will help assist.

[ninjabrum]

I am having the same patch error upgrading from 7.0u1c to any U2 release (tried u2d today).  Today’s attempt was with VMware support on the line.  We changed service timeouts in some json files, and reset the UMDB.  No joy.  Sent a support bundle for them to pour over tomorrow. I have a linked VC in another DC that updated no problem.  My next patch window is two days from now.

[JWoodruff]

Yep just hit my environment... can't get past that message on the admin portal side but vcenter is still working

[AKTM]

Hit the same issue on Production environment. To operate other Management features on the impacted VCSA:5480 , 

For example : https://xx.xx.xx.xx:5480/#/ui/update/progress  change to  https://xx.xx.xx.xx:5480/#/ui/ . Then other function working well.

But still need a way to fixing the issue ASAP.

[eChuck]

Thank you for sharing these notes. I have used this approach frequently in the past, given prior problems with updating VCSA deployments. However, the last three updates prior to this most recent 2d patch have worked without problems, so it looked like VMware had stabilized the update process. 

This latest problem seems to be something new, and is different from prior update/upgrade failure scenarios. It feels like this is due to other changes in vCenter or the VCSA internals. As noted, one difference is that vCenter continues to work, and appears to have been at least partially updated. It is the VAMI interface that is hung up, though this might be something that could be worked around. 

My reason for posting this was to hopefully get some insights as to whether this new problem is an annoyance, or an indication of something more serious. At this point, I don't believe we know what the situation really is, or what consequences we might have to confront.

[eChuck]

Thanks for this hint. I can confirm that merely changing the URL as you suggested does restore access to the VAMI graphical user interface.

The situation I'm seeing my system is that vCenter's web interface displays on the vCenter VM Summary page that the current build is  "18455184," which corresponds to 7.0.2.00500 (2d). However, it also shows "Updates Available." Clicking on the update button takes the user to the Update Planner, where this error is reported:

"Unexpected error occurred while fetching the updates"

On VAMI, the "Summary" page shows the version of VCSA as 7.0.2.00400 Build 18356314. The "Update" page shows that both 7.0.2.00500 (Sep 15) and 7.0.3.00000 (Oct 4) updates are available. So I am left with uncertainty as to what the actual version/build is for this VCSA instance. 

This leads to further questions on how to proceed? I am inclined to take the approach recommended by manadrain to update to 7.0.3.00000, but remain concerned that this system is potentially in an unstable situation. In particular, what are the consequences of the exception thrown in "postInstallHook"?

I guess it's time for more research...

[ninjabrum]

I finally got through this with help from VMware support.  We increased the service timeouts, which got us through the error.  Used the patch ISO and installed through CLI.

[ninjabrum]

Part of the patch didn't complete.  You should roll back your snapshot and contact support for assistance.  I got the error due to service timeout during patching.  Support increased the timeouts and we were able to patch successfully.  

[apsec-it]

can you share how you managed to increase the timeouts?

[ninjabrum]

/etc/vmware/vmware-vmon/svcCfgfiles

imagebuilder.json

updatemgr.json

vsan-health.json

Make a backup copy of each file

chmod +rwx <file above> (to allow editing)

Using a text editor (vi, etc), locate the "StartTimeout" field and change to 1500 in each file above

[captbrando]

FYI, the 7.0.3.00000 update from 10/4 upgraded fine for me.