Updating vCenter Server Appliance 5.1 to 5.1 U1 fails

[brucekconvergen]

Waited about an hour after it started installing... still not up.  Wen into the updatecli.log, and see that there's an error constantly repeating... any clues?

./etc/init.d/vami-sfcb: line 238:  7385 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
./etc/init.d/vami-sfcb: line 238:  7398 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
.failed, restarting vami-sfcbd.
Shutting down vami-sfcbd: done.
Starting vami-sfcbd: done.
Checking vami-sfcbd status:/etc/init.d/vami-sfcb: line 238:  7513 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
./etc/init.d/vami-sfcb: line 238:  7526 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
./etc/init.d/vami-sfcb: line 238:  7539 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
./etc/init.d/vami-sfcb: line 238:  7552 Aborted                 nohup /opt/vmware/share/vami/vami_sfcb_test > /dev/null 2>&1
"updatecli.log" 1450L, 98449C                                 1233,1        85%

[a_p_ · Accepted solution]

Not sure but maybe you experience the issue mentioned in the Release Notes.

Update of vCenter Server Appliance 5.1.x to vCenter Server  Appliance 5.1 Update 1 halts at web UI while showing update status as  installing updates*
When you attempt to upgrade vCenter Server Appliance 5.1.x to  vCenter Server Appliance 5.1 Update 1,   the update process halts for  nearly an hour and the update status at   Web UI shows as installing  updates. However, eventually, the update   completes successfully after  an hour.

Workaround: None.

André

[brucekconvergen]

Yeah, I saw that in the notes, that's why I gave it a long, long time to update... but those log entries seemed to indicate something was broken, the way they repeated almost the entire time after I initiated the update.  I did find that my VEEAM instance kicked on for some jobs, and though maybe that was causing the issue (as it uses vcenter to stage / manage the VMs during backups), so I shut it down, rolled back to 5.1 (snapshots are awesome), and am now trying again.

[brucekconvergen]

Nah... same errors... I'll just see if it just takes a very long time... all centers around vami-sfcb not "coming up" in this script.

[brucekconvergen]

And... it finally finished, whopping 1.5 hours...

[a_p_]

I ran the update and it took about 45 minutes. Quite annoying

André

[Tweety84]

At least, while the update is running, openssl seems to be broken!

I tried to access Port 5480 using openssl s_client. It dies with the message:

FATAL FIPS SELFTEST FAILURE

vami_sfcb is also unable to complete the SSL based communications with other services and therefore dies (take a look at the messages log, attaching strace to the init script is also pretty interesting).

Seems to be another bug...

[DanielOlofsson]

Same issue here but can confirm that it completed after ~45 minutes.

[Davi-jvs]

It seems this is an issue on the *ssl*.rpm packages being installer.

I did this and the update finished in about 5 minutes.

a) Inserted update ISO

b) SSH login

c) Mounted the ISO

mount /dev/cdrom /media/

cd /media/update/package-pool

rpm -Uvh openssl-*

rpm -Uvh libopenssl0_9_8-0.9.8j-0.44.1.x86_64.rpm

rpm -Uvh openssh-5.1p1-41.55.1.x86_64.rpm

umount /media/

d) Do the regular update from DVD.

e) Reboot.

Cheers,

Davi

[mknaup]

The issue is that the new openssl libraries do not match the (not updated?) hash files which are older:

-r-xr-xr-x 1 root root 1685176 Jul 10  2012 /usr/lib64/libcrypto.so.0.9.8
-r-xr-xr-x 1 root root  343040 Jul 10  2012 /usr/lib64/libssl.so.0.9.8

-rw-r--r-- 1 root root      65 Jan 11  2012 /usr/lib64/.libcrypto.so.0.9.8.hmac
-rw-r--r-- 1 root root      65 Jan 11  2012 /usr/lib64/.libssl.so.0.9.8.hmac

Once you delete (or move) the .hmac files OpenSSL will work fine again, just without the FIPS stuff (I doubt anyone really needs this anyway...). Most likely it is sufficient to delete the .hmac files before running the update from the GUI.

@VMware: I consider this to be a serious bug.

If you run into this AFTER trying to update the appliance, you'll have more trouble waiting even if SSL works again.

If the update did not finish or was interrupted during the long wait time, the postinstall script was not run. Most likely the vpxd will not start with the error message

     "Database version id '510' is incompatible with this release of VirtualCenter."

This will also break any attempt to re-install the update, which will result in a "Failed to install updates`(Error while running installation tests)".

Try the following:

- SSH to the appliance

- cd /usr/local/tcserver/vfabric-tc-server-standard/

- check if more than one subdirectory tomcat-6.x and tomcat-7.x exists.

  - if yes, de-install the older version of vfabric-tc-server:

    rpm -q vfabric-tc-server-standard

    rpm -e vfabric-tc-server-standard-2.6.4-1 (in my case)

  The reason is that otherwise /usr/lib/vmware-vpx/rpmpatches.sh will fail since some "ls .... | grep ..." statements do not properly check if ls returned more than one result... #bug

- cd /opt/vmware/var/lib/vami/update/data/job/

- cd into the latest subdirectory, which should have the higest number

- run the postinstall script:

  ./post_install '5.1.0.5300' '5.1.0.10000' 0 (the first one is the old build number, in my case for 5.1.0b)

  Ignore any errors.

- update the manifest

  ./manifest_update

- reboot the appliance

I hope I did not forget any steps I did during the troubleshooting, otherwise let me know in which problems you run, and I'll try to remember. 😉

This worked for me and saved the vCenter data(base) with the whole configuration.

Oh, and in case you're running the appliance in a home lab don't forget to change the config files to reduce the memory requirements. 😉 Some of them have been overwritten by the update.

http://www.vaspects.com/2013/04/24/minimizing-vcenter-memory-appliance/

Cheers,

Martin

[utw]

Martin, thanks for the advise, it helped getting Vcenter up running 🙂

Q1:

However I do worry what state my Vcenter is in now .... should I assume that I can apply the next patch for Vcenter after following your recipe ?

Q2:

Would you recommend following the advise from Davi-jvs in this thread about manually installing openssl from the iso after I have followed your recipe, and maybe get FIPS back (dont know if I need it, but I am sure it is there for a reason) ?

Q3:

And after the last reboot, I still have 2 tomcat folders in /usr/local/tcserver/vfabric-tc-server-standard/

tomcat-6.0.36.A.RELEASE

tomcat-7.0.32.B.RELEASE

Should I attempt to do something about this ?

Thanks in advance

Carsten

[mknaup]

Hi Carsten!

Thanks, good to know. 🙂

Q1: Well, after all this manual changes I would not consider the appliance to be in a stable state. If it was a production environment I would migrate the appliance to a fresh installation NOW (on the other hand: in production I would have had a snapshot or backup anyway...). In my test environment I plan to give the update a shot, just to see what happens. But afterwards I will migrate. Until that I just leave the appliance running, had no issues yet.

Q2: Depends. If the update finished (i.e. was interrupted during this ridiculous >1 hour wait time) the libraries are most likely already updated. Check with "rpm -qa | grep openss" in a SSH session on your vcenter appliance. On my system it looks like this:

openssl-0.9.8j-0.26.1

openssl-certs-0.9.8h-27.3.1

openssl-0.9.8j-0.44.1

openssl-certs-1.85-0.6.1

openssh-5.1p1-41.55.1

libopenssl0_9_8-0.9.8j-0.26.1

openssh-5.1p1-41.51.1

libopenssl0_9_8-0.9.8j-0.44.1

Which means everything has been updated to the versions in the updaterepo, and you cannot use Davi-jvs' advice - you'll only get "already installed" if trying.

@all: how's the list on an appliance after a successful update? Are the older versions of the packages removed?

I doubt FIPS is an issue, I would just leave it as it is. The hash files were included in libopenssl0_9_8-0.9.8j-0.26.1, they are no longer included in 0.44.1, so I suppose the FIPS support has been dropped anyway. I played around a bit and regenerate the hashes:

/usr/bin/fips_standalone_sha1 /usr/lib64/libcrypto.so.0.9.8 > /usr/lib64/.libcrypto.so.0.9.8.hmac

/usr/bin/fips_standalone_sha1 /usr/lib64/libssl.so.0.9.8 > /usr/lib64/.libssl.so.0.9.8.hmac

chmod 644 /usr/lib64/.libcrypto.so.0.9.8.hmac /usr/lib64/.libssl.so.0.9.8.hmac

The system boots up fine, otherwise no noticable change. Anyone from #VMware on this?

Q3: Nope, that's fine, don't change anything. Both versions are installed by the vfabric-tc-server-standard-2.8.1-1 package. The problem was caused by the older version still installed, so that there were TWO directories for 6.0.x and TWO for 7.0.x. That broke the script.

Cheers,

Martin

[Ebbebrich01]

Hmmm

I'm also seeing strange update results.

Like you - the update takes close to 45 min, and looking at the console for the appliance, it actually shows an init message/error.

The update seems to proceed and demands a reboot - no auto reboot ?

After the reboot - when I log onto the appliance  console - the vCenter tab is empty !!!

All info regarding services and general status of vCenter is GONE !!! just a blank page !!!

Logging in via the VSC works fine - and no errors are shown. It apears to be working..

How ever - I must admit that I do not trust the server anymore - what else is NOT working ??

I reverted to base image and is now trying to figure out what is going on.

/Jannik

[Nodnarb]

The workaround posted by Davi-jvs worked fine for me. I left my browser window alone after clicking the Update button and it eventually came back to the login screen after 5 minutes or so. I then logged in and rebooted it via the System tab.

There's a known bug with Windows session credentials logging into the vCenter appliance 5.1 u1. I found this form post very helpful (Re: vCenter Server 5.1.0, 947672 A general system error occurred: Cannot get user info). Here is a link to the KB article (VMware KB: Unable to log in to VMware vCenter Server Appliance using "Use Windows session crede...