Upgrade to vCenter 5.1 update 1 SSO failing

[buckmaster]

Having problems upgrading SSO. It says it going to perform an upgrade, but login fails. I can login with admin@system-domain via the web client so I know the password. I moved system-domain ahead of my local ad ldap for authenication purposes but still can not login so no upgrade? I’ve been searching google with no solution.  I'm performing a custom upgrade verses simple as I'm running SSO in multisite and vCenter in Linked mode for SRM and vSphere replication. 

In the past I've had no issues upgrading from 5.1 to a to b.  Don't know what's different.  Running on w2k8r2 with sql express r2.

Thanks for the help.

[buckmaster]

Evidently someone was able to upgrade because I see this warning: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=205094...

Has anyone else had success UPGRADING to 5.1 Update 1 - not a clean install but upgradng?  I need to upgrade as I'm running a vDS and SSO in multisite mode and vCenter in linked mode.

I guess as an option I could install a clean VM, restore my databases, sso , vcenter, update mgr, then try to upgrade.  Seems like a lot of work when I should be able to perfrom an in-place upgrade.

[buckmaster]

Since I'm running 2 vCenters; one at Prod and one at DR I thought I'd jump over to the DR vCenter and start the upgrade there since the Prod vCenter SSO installer will not let me login - SAME ISSUE AT DR!

Error: "Provided password is wrong or empty".

That simply is wrong.  I can login into the web-client as admin@system-domain just fine.  Since the installer is not written very well it does not ask for a username ONLY a password.  So I removed my AD as a default domain and I am only using SYSTEM-DOMAIN as  the default domain to satify the poorly written installer requirement.  Still does not work.

[buckmaster]

OK I THINK this has something to do with SSO Multisite and vCenter in Linked Mode.

I have antoher vCenter that I use to demo SSO install to our team.  Difference being it is not in SSO Multisite nor is there another vCenter.  For training purposes only.  I can upgrade SSO to update 1.  Takes my password just fine?

[buckmaster]

The only error I see in "vim-sso-msi.log" is;

Error 3: -2147287038

and

Error while attempting to search in/ below folder: Documents and Settings

Have no clue what it is trying to tell me?

[buckmaster]

Here is the installer log if VMware wants to review.

[raog]

Have you changed the password for the admin user before upgrade? The master password(not the same as the login password) will not change if the password is changed via the sso UI. So try using the orignal password or change the master password via command line (rsautil reset-admin-password)

Please refer the KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203460...

Also,

If this doesnt solve the issue, also attach the viminst.log and the exact place where the upgrade fails.

Regards

Girish

[buckmaster]

I learned something.  There is a MASTER password and it's not the same password used for admin@system-domain.  Never knew that and I can't help wonder how many other folks don't know that either. Ouch.  Either way I never changed the MASTER password and it SHOULD be the same password I use for admin@system-domain and I've not changed that either.  However running "rsautil reset-admin-password" is failing with "Invalid password, failed to decrypt system key Root cause; javax.crypto.BadPaddingException: Given final block not properly padded". 

By the way there is no viminst.log file? 

[buckmaster]

I don't like where this is headed!!!!!!

Mr Google pointed me to this KB regarding the above error:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101757...

Corrupt certificates?  This is a little absurd.  In the past I've upgraded from 5.1 to 5.1a then again to 5.1b no issues.  AND I'm running vCenter in Linked mode and SSO in Multisite mode for SRM and vSphere Replication.  Currently both vCenters "Prod and DR" running vSphere 5.1b are functioning fine, at least ALL services are running.  

[raog]

thats definitely strange.. but by chance did you try that workaround mentioned in the KB?

Regards

Girish

[buckmaster]

Yes.  Same result.  I'm confident it has something to do with SSO multisite and vCenter Linked Mode and this version of vSphere code.  Like I said earlier in post I've upgraded a stand alone vCenter I use to train my team on SSO installs.  It upgraded fine and it was running the same version of vSphere as the two Linked vCenter's that are failing.  The SSO installer gets no place as it first ask for the master password which I'm confident I know but it says it's not right or blank.

[buckmaster]

Well - it seems like the MASTER password is not working.  Before this thread I did not know the MASTER password and the admin@system-domain password could potentially be different. So for previous upgrades thinking they were the same I don't understand how I could have ever upgraded from 5.1 to 5.1a to 5.1b if I didn't type in the correct password.  They should be the same but evidently somehow the master password all of a sudden is not what I know it to be?

[raog]

http://communities.vmware.com/message/2205417

The comment posted by grasshopper about the db hash may help.

Regards

Girish

[buckmaster]

Been there tried that already.  Here is a snippet of the imstrace.log

2013-04-29 20:19:53,045, [main], (ManageSecretsCmd.java:470), trace.com.rsa.security.keymanager.tools.ManageSecretsCmd, ERROR, "was server name  I blanked this out xxxxxx.xxxxxxxxx.xxx,,,,Fatal error

com.rsa.ims.security.keymanager.sys.InvalidPasswordException: Invalid password, failed to decrypt system key

    Root cause: javax.crypto.BadPaddingException: Given final block not properly padded

at com.rsa.ims.security.lockbox.crypto.d.a(d.java:2)

at com.rsa.ims.security.lockbox.crypto.c.c(c.java:1)

at com.rsa.ims.security.lockbox.crypto.c.a(c.java:104)

at com.rsa.ims.security.lockbox.crypto.d.c(d.java:48)

at com.rsa.ims.security.lockbox.b.loadFields(b.java:245)

at com.rsa.security.keymanager.tools.ManageSecretsCmd.list(ManageSecretsCmd.java:604)

at com.rsa.security.keymanager.tools.ManageSecretsCmd.execute(ManageSecretsCmd.java:439)

at com.rsa.security.keymanager.tools.ManageSecretsCmd.main(ManageSecretsCmd.java:468)

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded

at com.sun.crypto.provider.SunJCE_f.b(DashoA13*..)

at com.sun.crypto.provider.SunJCE_f.b(DashoA13*..)

at com.sun.crypto.provider.AESCipher.engineDoFinal(DashoA13*..)

at javax.crypto.Cipher.doFinal(DashoA13*..)

at com.rsa.ims.security.lockbox.crypto.b.b(b.java:13)

at com.rsa.ims.security.lockbox.crypto.d.a(d.java:36)

I have a question for my understanding.  I've installed SSO several times for my lab and for customers.  No where does it every ask to set a MASTER password unless it gets set the first time you assign a password for admin@system-domain.  If that is the case the password never changed.

[buckmaster]

Also starting to wonder if the process of exporting and importing the sso configs to each vCenter Prod and DR is messing up the accounts?  Followed this KB for the process of when exactly to export and import.    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203867...  and this is the actual steps: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=204284...

I've noticed if you export the config from prod then install sso at dr then immediately import the prod sso to dr, build out the rest of dr, then export dr sso and import to corp sso that any account you added to the corp sso are gone after import the sso file from dr?  Seems strange right?

[raog]

the first time password enters becomes the MASTER password. There is no UI to set it explicitly.

Also for multisite, the SSO local users/configuration will be lost if the proper replication steps are not followed:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203407...

Regards

Girish

[buckmaster]

Thanks for the input.  The KB you mention 2034074 is basically the same as the one I'm using http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=204284....  Difference being 2034074 explains in a little more detail what's occurring whereas 2042849 is just the steps to follow.

To get around the MASTER password issue I backed up my vCenter and Updatemgr DB's, built a new VM, installed SQL, restored DB's and installed a clean SSO and the vCenter installer upgraded my vCenter and UpdateMgr DB's.  Have to keep those as I'm running a vDS.  Prod is running SSO in Multisite and vCenter will be linked.  Prod is good for now.  Getting ready to do the same process for DR.  I've exported the PROD SSO and as soon as I install SSO at DR I will import the PROD SSO to DR.  When DR is completly built out I will export that SSO and import it to the PROD SSO.  Best I can tell from the KB's that's the process to follow.  I will keep screen shots of both SSO users before and after each export and import.

It feels like the import/export/ and import process is over writing data verses merging information?  Since each SSO config is it's own LDAP and provides no backup for each other how does SSO deferengiate between users at each location?  In other words how is admin@system-domain and admin2@system-domain different from each other at each location since the domain names are the same?

[raog]

Yes... you need to be careful during the replication process. The data will be overwritten. Recommendation is to not do any changes to the PROD while DR is being built otherwise these will get overwritten when you replicate the data back to PROD.

The way this will work is

Site1 has info.

Create Site 2. Import the data from site1. Create any additional data.

Import back to site 1. Site 1 now has data from both site1 and site2.

Regards

Girish

[buckmaster]

Girish,

Appreciate your responses.

Everything is up and running update 1.  Now to move on to update & re-sync SRM and vSphere Rep but SSO Multisite and vCenter Linked Mode is good.

Followed this process at both site's

PRODUCTION

1. Backup vCenter & Updatemgr DB's - shut down original vCenter VM
2. Create new vCenter VM - assume same name and IP as original VC
3. Install SQL and restore DB's - NOT SSO - I HAVE NO IDEA WHAT HAPPENED TO THE MASTER PASSWORD - CONCERNING - I'm confident I know what the password was AND I made system-domain the primary domain - Request - VMware needs to fix installer to ask username instead of assuming admin and only asking for a password.  In the field I will always make the company domain the default since 99% of the time this is how we will login.  Doing so you have to remember to flip your domains when doing upgrades?  Not really effecient.
4. Install SSO first node
5. Install vCenter and all the rest of the apps - resulting in DB's getting updated a saving my vDS info - important because upgrade was done with no downtime
6. Export SSO - needed for DR SSO import
7. ** After DR build out - Import the SSO from DR site

DR site

1. Backup vCenter & Updatemgr DB's - shut down original vCenter VM
2. Create new vCenter VM - assume same name and IP as original VC
3. Install SQL and restore DB's - NOT SSO
4. Install SSO and JOIN first SSO node
5. Import PROD SSO before proceeding
6. Install vCenter and all the rest of the apps
7. Export SSO - needed for PROD SSO import

Questions:

1. I always make the vCenter admin a SSO admin and the SSO admin a vCenter admin so those accts can back each other up.  Important because if you lose your AD directory your SSO admin can continue to manage vCenter while you fix your AD issues.  It's happened.
   1. So regarding the above comment - I've noticed when you import the PROD SSO to DR SSO your AD accounts are dropped or not added to the DR SSO and you have to add them back.  What's even stranger when your all done with the DR build and you export the DR SSO and import to the PROD SSO it removes the AD accts you had manually added to the PROD SSO?  That seems like a bug?  Why does this happen?  You have to remember after each import of SSO to manually add back in your AD accounts?  Not effecient in my opinion.
2. I've noticed after importing the PROD SSO to DR the password policy at the DR site has been modified to reflect the password policy at the PROD site SSO.  May be normal but unexpected considering question 1 above.  Seems like some aspects of SSO manaul replication is tightly integrated while others like preserving company domain accts are not?
3. After all this I do not gain the benefits of SSO in regards to SRM as SRM is not SSO compliant.  Any idea when we can look forward to SRM SSO compliance?

I would really appreciate a response to these questions.  I tried to document my upgrade process for the team I work with but maybe it will also help someone else in the community.

Thanks - Tom

[raog]

Will try to answer the first two questions, i have no idea on the 3rd one:

1. So regarding the above comment - I've noticed when you import the PROD SSO to DR SSO your AD accounts are dropped or not added to the DR SSO and you have to add them back.  What's even stranger when your all done with the DR build and you export the DR SSO and import to the PROD SSO it removes the AD accts you had manually added to the PROD SSO?  That seems like a bug?  Why does this happen?  You have to remember after each import of SSO to manually add back in your AD accounts?  Not effecient in my opinion.

This should not happen. Post the export of Prod SSO and import to DR SSO, once the DR SSO is setup and exported back to PROD, both SSO should have the same data.

"2.I've noticed after importing the PROD SSO to DR the password policy at the DR site has been modified to reflect the password policy at the PROD site SSO.  May be normal but unexpected considering question 1 above.  Seems like some aspects of SSO manaul replication is tightly integrated while others like preserving company domain accts are not?"

Password policy will be replicated along with users.