Update manager hanging at 55% on newly upgrade 7.0U3c hosts

[Cfakley]

We upgraded our hosts from 7.0U2d to 7.0U3c using the vCenter built in critical patch baseline. The hosts were installed using a HPE custom ISO 7.0 release and have been upgrade to several 7.0 updates overtime.

Logs seem to point to an issue with package compliance...

vmware-vum-server[09172] [Originator@6876 sub=HostUpgradeScanner] [scannerImpl 661] Precheck script test result: 'ERROR', test 'PACKAGE_COMPLIANCE', expected '', found 'HPE_bootbank_ilorest_700.3.3.0.3-15843807 HPE_bootbank_amsd_701.11.8.0.15-1OEM.701.0.0.16850804 MIS_bootbank_hpessacli_5.20.8.0-7.0.0.15525992.oem MIS_bootbank_smartpqi_70.4150.0.119-1OEM.700.1.0.15843807 HPE_bootbank_hponcfg_700.10.8.0.1-7.0.0.15843807 HPE_bootbank_conrep_700.10.8.0.21-7.0.0.15843807 INT_bootbank_icen_1.6.2.0-1OEM.700.1.0.15843807 HPE_bootbank_testevent_700.10.8.0.1-7.0.0.15843807 HPE_bootbank_fc-enablement_700.3.8.0.6-1OEM.700.1.0.15843807 HPE_bootbank_ilo_700.10.7.5.2-1OEM.700.1.0.15843807 QLC_bootbank_qfle3f_2.1.25.0-1OEM.700.1.0.15843807 HPE_bootbank_bootcfg_700.10.8.0.1-7.0.0.15843807 QLC_bootbank_qfle3i_2.1.8.0-1OEM.700.1.0.15843807 HPE_bootbank_amshelpr_701.11.8.0.12-1OEM.701.0.0.16850804 PEN_bootbank_ionic-en_18.0.0-1OEM.700.1.0.15843807 QLC_bootbank_qedf_2.2.62.0-1OEM.700.1.0.15843807 QLC_bootbank_qedentv_3.40.36.0-1OEM.700.1.0.15843807 QLC_bootbank_qedrntv_3.40.35.0-1OEM.700.1.0.15843807 HPE_bootbank_vnic-enablement_700.2.00.6-1OEM.700.1.0.15843807 QLC_bootbank_qfle3_1.4.17.0-1OEM.700.1.0.15843807 HPE_bootbank_sut_701.2.9.0.44-1OEM.701.0.0.16850804 MEL_bootbank_nmst_4.14.3.3-1OEM.700.1.0.15525992 QLC_bootbank_qcnic_2.0.59.0-1OEM.700.1.0.15843807 QLC_bootbank_qedi_2.19.59.0-1OEM.700.1.0.15843807' and errortype is 18  

Does anyone know the cause or how to fix this?

[Anil0210]

Can you clarify what your base HPE cuatom

image ? 

Have you synced online update of 70u3c or imported offline zip ? 

[Cfakley]

---

@Anil0210 wrote:

Can you clarify what your base HPE cuatom

image ? 

Have you synced online update of 70u3c or imported offline zip ? 

---

The original base install was done with the 7.0U1c ISO for HPE servers downloaded from the VMWare custom ISO repository.

Our vCenter is set to automatically download patches, so 7.0U3c was synced online.

[Cfakley · Accepted solution]

We were able to fix this by re-installing ESXi on all our hosts using the standard 7.0U3c iso.

[johnmcc22]

Nice, I was just going to ask if you used the custom ISO to upgrade with, that's what i'll be using.  Had logged in to see if there were any issues before going with it.

[Zahni]

Same Issue here. I hand opened a case.

Update: I had now made an Inplace-Upgrade using HPE latest ISO VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022.iso  and it will work so far.

[Zahni]

Ok, one possible source of this problem:

HPE had also recalled it U3 Vibdepots from http://vibsdepot.hpe.com/

This cause Update Manager database inconsistencies. HPE had  published VIBs for Jan2022 (http://vibsdepot.hpe.com/hpe/jan2022/) but forget update die XML files ( http://vibsdepot.hpe.com/hpe/hpe-index.xml f.e.) 

VmWare Update Manager (Lifecyle...) still don't have an option remove an already published update...

[imourza]

What VC version was this problem encountered with? (ie, was VC at 7.0U3c level or at lower version?)

[Zahni]

7.0U3c of course. I had included HPE VIBDEPOT (index.xml an index-drv.xml) in my Update Manager Setup.

http://vibsdepot.hpe.com/getting_started.html

[BerardD]

just looking to confirm, you did a total reinstall and reconfigure of each host not an in-place upgrade using the standard 7.0U3C iso?

[Zahni]

I had made an "Upgrade" installation with HPE's latest ISO file with U3c.

Reboot the host an don't make compliance search. Attach the Baselline with that ISO and remove all other baselines and go the "remediate". After finish, remove the upgrade baseline and attach the normal patch baselines and do the compliance check. 

[Cfakley]

Correct. I wanted to make sure nothing from the HPE custom image remained.

So far our HPE servers have been solid running the standard 7.0U3C image.

[Mordecai11803]

In case anyone is still having this issue; per support we fixed it by removing the following HP Vibs through SSH:

amscli, amshelpr, amshelprcli