VMware Cloud Community
asigurds
Enthusiast
Enthusiast

SSO fails to start after upgrading to vCenter 5.1 Update 2

After trying to update the vCenter server from V5.1 U1 to U2 the SSO service fails to start. The following information is registered in the event log with event id 7031:

The vCenter Single Sign On service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

No log files are being created / updated when the service is trying to start.

Anyone had had a similar experience with the SSO service or hints on what to look for?

29 Replies
atulkatiyar
VMware Employee
VMware Employee

Could you please upload the log bundled of SSO ?

Thanks,

Atul

0 Kudos
asigurds
Enthusiast
Enthusiast

Thanks for your reply, but I gave up hunting this problem and installed the vCenter Appliance V5.5.0b.

0 Kudos
pbrutsch
Contributor
Contributor

I had the same problem (the SSO service won't start after in-place upgrade). Uninstalling vCenter (version 5.1 Update 1c) and re-installing 5.1 Update 2 got me updated.

0 Kudos
Dave338
Contributor
Contributor

Also same problem here.

My system was at 5.1U1a, if I try to upgrade to U2, SSO fails to start. I tried to upgrade to U1c and has been succesful, but again if I try to update to U2, fails the same way.

Any suggestions??? I don't want to do a fresh install again, that environment was fresh installed on April-May 2013... If I uninstall and install again I lose my database and config???

Regards.

0 Kudos
Dave338
Contributor
Contributor

Hi again!!! Finally I managed to upgrade mi environment.

Yesterday I tried to stop all services, uninstall only SSO and new install SSO 5.1u2. I got another error that I found it was caused by bad/corrupted JRE folder (in \\program files\vmware\infraestructure), so I renamed that folder to "jre.old" and repeated the installation. It worked, but later vcenter server couldn't upgrade due to problems with SSO integration...

I recovered one more time the entire server and started again from ground.  FINALLY THE SOLUTION!

-Stop all vmware services

-Rename/delete/move "jre" folder

-Start upgrades from the UI in the typical order (SSO, Inventory, vcenter, client, web client)

Now working after three days of trying.

I expect that effort to help anyone!! xD

Thanks.

Dave338

0 Kudos
MentalTheo
Contributor
Contributor

Dave338 wrote:

Hi again!!! Finally I managed to upgrade mi environment.

Yesterday I tried to stop all services, uninstall only SSO and new install SSO 5.1u2. I got another error that I found it was caused by bad/corrupted JRE folder (in \\program files\vmware\infraestructure), so I renamed that folder to "jre.old" and repeated the installation. It worked, but later vcenter server couldn't upgrade due to problems with SSO integration...

I recovered one more time the entire server and started again from ground.  FINALLY THE SOLUTION!

-Stop all vmware services

-Rename/delete/move "jre" folder

-Start upgrades from the UI in the typical order (SSO, Inventory, vcenter, client, web client)

Now working after three days of trying.

I expect that effort to help anyone!! xD

Thanks.

Dave338

Hi Dave, thanks for you effort. I have been struggling this issue for days and performed the full system restore several times. I found the same issue with JRE.

Unfortunately, when i follow your steps and rename/delete/move JRE folder then i end up in SSO installer saying that password is wrong or blank. When i do not rename/delete/move JRE folder then SSO installer password check passes.

0 Kudos
Dave338
Contributor
Contributor

Wow, that a weird problem!!

Do you have any other java machine or program that uses java installed on the server??? I encountered something similar in one of my first attempts to upgrade.  From which versión are you upgrading?? Try to upgrade first to 5.1U1c if not in that one to see what happens. I was in 5.1U1a and upgrading to U1c helped me to surpass first issues...

And uninstall sun java if present!!!

Regards.

0 Kudos
MentalTheo
Contributor
Contributor

No sun Java. There is another java present in APC PCNS that i use to gracefully shutdown whole VI, but i tried to stop the service and rename jre folder before. No change.

Currently running on 5.1 U1b. I will give 5.1 U1c a shot, but i doubt it will help.

Thanks for help.

0 Kudos
Dave338
Contributor
Contributor

For me the 5.1u1c update was succesfull at first try, the issue was only in U2

0 Kudos
MentalTheo
Contributor
Contributor

Sucessfully updated to 5.1 U1c.

5.1 U2 SSO installer is still complaining about wrong or blank password after i renamed/moved the JRE folder. The password check itself and warning dialog box is displaying unusually fast which leads me to conclusion the installer actually uses JRE in "Program files\vmware\infraestructure" for some reason, because when i do not rename/move the folder then the password check takes a few seconds to complete.

So now i am stuck at the same point. I will try to upgrade from 5.1 U1c to 5.1 U2 later this week as i need vCenter up now for Veeam backups to work, but i fear the update will mess things up again and i will have to do full restore again.

0 Kudos
Dave338
Contributor
Contributor

Well at least you are in 5.1u1c. xD

What seems clear is that there is a problema with the U2 updater, so other updates work well, the only one that fails is the U2 one...

Do you have another vcenter available?? If not, you can try to do a fresh install on a vm of the U2 version and copy later the JRE folder to the problematic one.  Maybe that Works for you... IS strange... do you stop all vmware services before upgrading, right???

REgards.

0 Kudos
mahantesh_dinda
Contributor
Contributor

VC5.1 U1a & VC5.1 U1c use the same JRE and hence upgrade is smooth & not likely to hit JRE issues. VC5.1 U2 has a higher version of JRE update which I think requires right JRE folder. 

0 Kudos
MentalTheo
Contributor
Contributor

Today i tried upgrade again. This time from 5.1u1c to 5.1u2. Unlucky again, SSO wont start. Logs attached.

0 Kudos
MrBigFeathas
Enthusiast
Enthusiast

Just experienced this myself a couple weeks ago. Luckily in my dev environment. I had to rebuild my 5.1 U1b dev server again and was about to give it another shot when I happened upon this post. This was the only thing I could find from my Google search. Have that few people tried upgrading too update 2? Or are those of us in this thread just unlucky?

0 Kudos
mp5423
Contributor
Contributor

Same thing happened to me upgrading from 5.1 U1c to 5.1 U2.  Single sign on service failed to start during the install, it said you will have to start manually, then installer successfully finished.  I cannot get the single sign on service to start manually either.

0 Kudos
Dave338
Contributor
Contributor

Hello!!

You have to stop it manually, and delete JRE folder before upgrading. Then start upgrade SSO, and if successfull, continue with rest of the pack. For me, once I reached this point (SSO upgraded), the rest was easy Smiley Happy

SAlu2.

0 Kudos
knse
Contributor
Contributor

I wasted 10 hours chasing this round and round. I either forgot to sacrifice the goat to ESXi first, or failed to sprinkle salt on the Java logo. Whatever it was, the error logs aren't much help, and it always comes down to broken SSO in the installers. Fortunately, I made a snapshot of the vCenter before attempting everything I could think of.

Since moving to 5.1, I have dreaded every single vCenter update. Why did VMware utterly break the vCenter installers? And this dependence on buggy Java is not helping. This may be the time to go straight to 5.5 and see if it's any better. If VMware keeps releasing broken installers, I expect VCP's and others will start looking at alternatives -- it's just not worth the wasted hours trying to find workarounds!

knse
Contributor
Contributor

Method I used for getting this to work from any version of 5.1 Update 1:

  1. If you are running vCenter as a VM, make a pre-upgrade snapshot in case something goes sideways and you need to rollback quickly. If things work out well, you can delete the snapshot later.
  2. Open a command prompt with Administrator privileges.
  3. Stop the SSO service ("vCenter Single Sign On") in your GUI. Or,
    Command line: C:\>sc stop ssotomcat
  4. Open the folder containing the jre subfolder for SSO and vCenter. Typically, this is
    "C:\Program Files\VMware\Infrastructure\"
  5. Open Task Manager with admin privs, and sort by name to show java.exe processes. Add the PID column if you will use taskkill via command line.
  6. Kill all instances of java.exe in Task Manager or by PID using taskkill /pid #### /f. You can script this, or run a one-liner like:
    for /f "tokens=2" %j in ('tasklist ^| findstr java.exe') do taskkill /pid %j /f
  7. Immediately drag the jre folder to another folder location (as a backup) as soon as you kill the last java.exe in memory. You will have to be quick, as java.exe respawns quickly. You can also do this via command line using the MOVE command: C:\>move /y <your_jre_folder_path>\jre jre.old
  8. Kill any respawned java.exe, again -- I found there is usually one that survived the jre folder move.
  9. Launch the U2 upgrade and step through the component installers in order. the install still take s along time (up to 30 minutes for all components), so be patient for each component installer and let it stop/start each service on its own. Do NOT reboot between component installs -- there is no need, until the last component is installed.
  10. Reboot when ALL components are upgraded. Check running services to make sure they all started. Set any service recovery options or delayed start tweaks as needed for your environment.

Scripted, if you have a lot of these to do, or dread the next 5.x update, you can run this as a batch script, adjusting your jre path below. I didn't bother with error handling, logging, or pretty output -- this is just a brute force helper script to prep for upgrading.

======================================

@echo off

:: vCenter 5.x Upgrade Prep Script

cls

echo Make sure you have a pre-upgrade snapshot, or good backup,

echo of your vCenter server before upgrading. To abort, press CTRL-C now. Otherwise,

pause

sc stop ssotomcat

for /f "tokens=2" %%j in ('tasklist ^| findstr java.exe') do taskkill /pid %%j /f

:: Adjust the drive and path to your jre folder in these lines.

:: Note that if you have Update Manager installed, you will have another jre

:: installation -- usually in Program Files (x86) -- that can be ignored.

c:

cd "C:\Program Files\VMware\Infrastructure\"

:: Try to move the jre folder

set /a MoveCount=0

:MoveJRE

set /a MoveCount=%MoveCount%+1

move /y .\jre .\jre.old

for /f "tokens=2" %%j in ('tasklist ^| findstr java.exe') do taskkill /pid %%j /f

if %MoveCount%==3 goto AbortMove

if exist .\jre goto MoveJRE

:End

echo The jre folder has been moved to jre.old. This script has fulfilled its sole purpose in life

echo and is feeling gratified. Best of luck with your impending vCenter upgrade!

exit

:AbortMove

echo The jre folder cannot be moved, for some reason. Script aborted.

echo Time to delve into the mysteries of tomcat as a requirement of vCenter.

===========================================

Vibs1983
Contributor
Contributor

Hi,

Did you mange to get this sorted?

If you delete the jre folders it wont upgrade because the SSO service is stopped and the upgrade cant start with this services stopped.

Is it best to uninstall SSO and reinstall it from the update 2 media?

Thanks

0 Kudos