vCenter 5.1 upgrade gone horribly, horribly wrong leads to mystery password

[dgingeri]

OK, so, I work in a test lab.  We now have multiple 5.1 hosts that we are using for testing.  We have had a vCenter 5.0 server in place for a little while, but were not able to manage 5.1 hosts from it, so I was tasked with upgrading it to 5.1.  To make a long story short, the upgrade left the OS (Windows Server 2008 R2) so horribly corrupted through my attempts that we could no longer even reinstall vCenter.  (First complaints from SSO that the machine wasn't a member of a domain, then it didn't like that I made it a DC, so I removed AD DS, made a separate VM DC and joined it to a domain, then SQL would no longer work.)  I basically had to star over on a whole new VM.

First off, I have a separate VM (Hyper-V on our infrastructure storage server) as a stand alone DC for the lab domain.  I have two user accounts, nobody which is the renamed administrator account per MS recommended security practices and administrator which is a restricted domain user account.  The vCenter server is a VM on our main infractructure ESXi 5.0 host and has Windows Server 2008 R2 Standard installed and is a member of the lab domain.

I tried installing vCenter server, and everything seemed to go OK, but it won't let me sign in with any username/password combo.  It just says that the username or password is incorrect.  I've tried the nobody account, the domain 'administrator' account, and the local admininstrator account.  I also tried the credentials from the beginning of the SSO install (admin/secret password) only to get a different message of "you do not have permission to log in" type response. 

What the heck am I doing wrong?  I don't get it.  I've been fighting it for over two hours and getting nowhere fast.  People are already upset with the fact that I couldn't save the old 5.0 vCenter server.  (5.1 and its demands totally screwed it up, and the backup is corrupted.)  I need to get this stupid thing up as soon as possible.  the boss is a little upset that I haven't been able to get this going.  If I don't get this going soon, I'm going to have to step back to 5.0 and leave out the 5.1 hosts. 

I have over 20 years of experience with Windows (3.1 though Win8) and over 7 years of experience with Windows Server (2003 through 2008 R2.)  I took the vSphere 5.0 class and had no problems.  I've done several 5.0 vCeneter installs with no problems, but those were on stand alone machines.  I can't say for certain that it is the interaction of the domain with 5.1 or changes with 5.1 or bug in 5,1 that are causing this, or maybe I'm just not understanding what I'm typing into the installer. 

[Scruffy_Nerfher]

If you can obtain a copy of the Ultimate Boot CD, or perhaps ERD Commander 2008, run the Locksmith utility.  This will allow you to change the password on one, and only one, account.  I believe you, that you know what you're doing, but this will be a very quick and easy way to make absolutely certain you are typing the password you've assigned.  (It's also a fantastic tool to have in your aresnal.)

I've used both of these apps, religiously, in the past, and they have saved my bacon many times over.

[dgingeri]

Sorry, I wasn't clear on at least one thing: I can't log into vcenter.  I can log into the machine itself fine.  I just can't log into vcenter and manage any hosts. 

[Scruffy_Nerfher]

Ohhhhhhhhhhhhh.  Now that does make a difference.  I did a quick and nothing of any magnitude came back.

I wish I could help you on this one as I've been in your shoes many times, myself - people and boss breathing down your neck for "instant" results but you can't fix something when you don't know how\why it's broken.  You try your darndest but nada.  At this point, I'd recommend sticking with what's been working thus far - roll them back to 5.0 and keep plugging along.  I'm not saying to give up on the migration.  Just take a step back, gather your wits and your forces, then re-attack when you've regrouped.  You can always try the migration thing in smaller bites, and a little at a time, so you're not in this pickle, you find yourself in, again.

The thing is, most of these people who are breathing down your neck only know that they push a button and things work.  They don't understand that you build the button(s) that they push.  Without you, they're at a standstill.  But that's a concept far beyond their comprehension.  Sound familiar?

Hang in there, Brother!

[aaronwsmith]

This section of the vSphere 5.1 document describes how SSO can alter the security for vCenter 5.1 and who has access to login:

http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.upgrade.doc/GUID-3BDE41A9-32C2-40D8-A17E-...

What I would do is login to the Web Client using that id/password you mentioned "from the beginning of the install" (default ID I believe is admin@system-domain.)  Then select Administration -> Sign-On and Discovery -> Configuration and see what Identity Sources you have listed and let us know.  If you need to, try adding the "Local OS" identity source.  This is also where you can add your domain as an identity source if it's not listed.

Also under Administration check Access -> SSO Users and Groups and see if any users are locked/disabled that you would expect to be unlocked/enabled.

Finally, check Administration -> Access -> Role Manager and see if a vCenter Server is listed.  If so, you should be able to see who is mapped to the Admin role.

SSO FAQ:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203491...

Hope this helps.  If you make progress based on this, let us know and hopefully we can get you up & running again!

[dgingeri]

That was helpful.  I could log in using admin@System-Domain in the webclient.  The software client just told me I wasn't allowed to log in.  However, under the Administration panel, there is no "configuration" selection under "Sign-On and Discovery".  As a matter of fact, there is nothing under that heading, but I know there should be. 

[aaronwsmith · Accepted solution]

That's odd.  Can you take a screenprint of what you can see from the Administration page?

Also - Can you try any of your logins you previously mentioned via the Web Client and see if you're able to login?  Perhaps one of those IDs is now the SSO admin ID.

[dgingeri]

It finally came up, but just on the local machine.  any remote machine accessing the web client gets pages that won't completely load, like things are timing out.  However, bringing up the web client on the local mahine works fine now.  (I think it was due to massive disk activity.)  I was able to set the lab domain to the primary domain and get it working properly.  Thank you very much.

[aaronwsmith]

Glad you got it working!  Always nice to recover and be able to continue with your work invested in upgrading vSphere.  SSO centralizes authentication across all components (vCenter, Web Client, Inventory Service, Auto Deploy, etc.)  So if SSO is not working/down or is not configured as you'd expect, it will prevent you from accessing a lot of resources in your VI.

I would definitely recommend reading through the materials on SSO, there's a lot of good info you can find in reference to the vSphere doc link from my earlier post, even if the material is a little dry 🙂

The issue accessing the web client from remote devices is interesting.  Hopefully it's working now?  If not, might be worth investigating further.  The web client is written I believe in Adobe Flash/Flex, and it's going to be the future of access to vCenter and ESXi hosts.  The fat client is EOL after vSphere 5.1 according to VMware's release notes.

Take care!