Can't sign into VCSA with AD credentials

[7007VM7007]

I upgraded my vCenter Appliance from 6.5e to update 1 yesterday and upgraded the ESXi hosts from 6.5d to 6.5 update 1. There were no issues/errors during the upgrade.

Before performing the upgrade I have been logging into the Flash or HTML5 web client using my AD credentials. My VCSA is domain joined and only had an IPv4 address.

After the upgrade I added an IPv6 address to the VCSA and can no longer login to the VCSA using my AD credentials, I get the "Invalid credentials" error after clicking the Login button. This only started happening after adding a static IPv6 address to the VCSA. I did add a reverse DNS record for the VCSA FQDN. When I only had an IPv4 address SSO worked but now that I have dual IPv4/IPv6 addresses AD logins are failing and the only way for me to login is to use the administrator@domain.local account.

If I run the following on the VCSA while logged in as root and using the shell:

cat /var/log/messages | grep -i "error"

I get this:

2017-07-29T07:08:59.463241+00:00 localhost ntpd[24585]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

2017-07-29T07:26:38.000442+00:00 localhost lsassd[1129]: 0x7f898115c700:Failed to sync system time [error code: 40075]

2017-07-29T06:16:33.103697+00:00 localhost cli: root SSO initialization error: [Errno 111] Connection refused

This got me thinking about time...

So I checked my VCSA time and it was one hour off compared to the ESXi servers and domain but the VCSA does use an NTP server (which is my domain controller). I have set both IPv4/IPv6 addresses for the NTP server.

So I logged into https://vcsa.domain:5480/  and it was using the correct NTP settings and syncing but it was one hour off. The strange thing is, the time zone field is empty. I tried setting it a few times (and rebooting) but its still empty.

Not sure if the lack of a time zone is the cause of this issue but how can I get my SSO with AD working again?

I also tried disabling NTP and setting the time manually on the VCSA but it kept changing back to the incorrect time (even after a reboot).

All my SSO logins are currently broken and I can't do Citrix MCS as it keeps saying the username/password is incorrect so plenty has gone wrong.

Is this a time issue and if so how do I resolve it? Are there any other logs that would be helpful in troubleshooting this?

Added root SSO connection refused error from log

[jcfs1485]

Having the same issue except for the IPv6 portion; I'm not using IPv6 in anyway.

After the 6.5U1 upgrade I was no longer able to authenticate to my AD domain.  The VCSA appeared to still be attached to the domain, but authenticating to the identity source fails with 'invalid credentials'.  Logging into the console of the appliance I notice right away that the hostname had been reverted to localhost.localdom, I recall specifically having to manually change this prior to attaching it to the domain previously.  I also noticed the clock was off; both my DC and the VCSA are pointed to the same NTP server, and both are set to UTC but for some reason the VCSA was off by 7 hours and a few minutes.  I attempted the following steps with no such luck:

• Renamed the VCSA to it's previous assigned hostname and rebooted.
  • Same login failure.
• Removed the VCSA from the domain and re-attached (deleted AD object, waited 20 minutes, etc).
  • Same login failure.
• Removed and re-added the identity source.
  • Same login failure.
• Manually modify the time to match that of my DC
  • Same login failure and the clock reverted to the incorrect time after a reboot of the VCSA.

I don't have access to the logs at the moment but will post them later, I'm seeing the auth failures but am unable to determine the root cause.  I suspect it has something to do with the clock and/or the hostname being reverted back to localhost.localdom when the upgrade is applied.

[Camero]

This is already addressed in the post

Re: vCenter 6.5 VSCA unable to deploy OVF

[Jbir]

Thanks Camero, my OVF issue is now fixed.

In regards to the AD auth problem. My date and time are correct but my hostname has been set to localhost.localdom. I have always only used the FQDN during installs, migrations etc. I am hopefully having a dial in with VMware this morning so i will update later on their findings.

[JordonC]

Has everybody gotten their environment fixed?  I haven't seen any updates in several days of any official fixes.  I have the same behavior where if I ssh and run hostname is has the correct FQDN.  After update U1 it reverts to localhost.localdom and I can no longer login with AD credentials.  Want to get this fixed in our test environment first before moving forward with production. 

[JordonC · Accepted solution]

I have been able to fix my Test environment by doing the following.  My problem is the hostname was changed somehow after the update to U1.  Looked correct everywhere except when checking using ssh.  This is what I did to fix

1.     First I logged into the Web Console with administrator@vsphere.local

2.     Go to Administration and go to configuration and look at the certificates.  Check if they are they for the correct FQDN

3.     Next Go to hosts and cluster and right click on top VCenter Name and go to settings.  Expand Runtime Settings.  Check if the correct FQDN name is there also.

4.     After all those look good SSH into the VCenter server and run the command hostname.  If localhost.localdom comes up U1 probably defaulted to this hosts name.  It needs to be changed back to what your certificate says.

5.

1. Connect to the vCenter Server Applaince with an SSH session and log in as root.
2. Run the command shell.set --enabled=True to enable the shell interface
3. Run shell to activate the bash shell.
4. Remove the vCenter Server Appliance from the domain by running this command:
   
   /opt/likewise/bin/domainjoin-cli leave
   
   
5. Correct the hostname of the vCenter Server Applaince by running this command:  !!Make sure host name matches your existing certificates!!
   
   /opt/vmware/share/vami/vami_set_hostname vCenter-Appliance-FQDN
   
   

     Example:

     /opt/vmware/share/vami/vami_set_hostname vcenter01.company.com

1. After completing, join the Appliance to the domain using:
   
   /opt/likewise/bin/domainjoin-cli join domain user
   
   Example:
   
   /opt/likewise/bin/domainjoin-cli join company.net Administrator
   
   
2. When prompted, enter the password for the domain user
3. Reboot the vCenter Server Appliance

This worked for me.  You may have to remove the identity and re-add it but I did not.  Once my host name matched the certificate I was able to log in again with ad credentials.

[JordonC]

Just finishing up with support on the phone.  We may have foudn the issue of why it reverted to localhost.localdom.  During the boot process in 6.5 U1 the technician found the following. 

#

# reverse lookup the hostname from the IP address. If it can't be resolved,

# set the hostname to localhost. Wait for a max of 10 seconds if there is

# no reply then set the hostname to localhost

#

# prefer an IPv4 reverse lookup over an IPv6 reverse lookup

#

After I added a reverse lookup IP in DNS and rebooted the VCenter server the hostname reverted to the correct name and I could log in.

[7007VM7007]

Thank you very much JordonC, this worked perfectly for me! My VCSA now has the correct FQDN and I was able to join the domain and add the AD identity source using the machine account first time. Now I can login using SSO with the enhanced client plug-in.

Thank you!

[JordonC]

Great. 

[switchy]

Dear all

I have checked all the settings, discussed above. Our vCenter (6.5 U1) shows correct name on running "hostname" in shell. DNS Forward and PRT Records are present and correct. All settings in vCenter itself show the correct FQDN hostname.

Anyhow, I'm still unable to login vCenter using AD credentials. It's always telling me "Invalid credentials". We have tried with different credentials and the passwords are entered correctly.

I have tried to remove the vCenter from domain running "domainjoin-cli leave". The I get the following error message:

Error: ERROR_MEMBER_NOT_IN_GROUP [code 0x00000529]

Any idea, what I can do to get AD sign in back working? This problem just started today. I guess it's related to U1 of 6.5.

Thanks for any help.

[Camero]

Please use a standard NTP server to avoid such time issues going forward. And make sure to set every entity (ESXi hosts, vCenter  & AD etc)  in the environment to use same NTP server.

Once the time related issues are resolved, then proceed with the all of the steps as mentioned by Jordon in comment# Re: Can't sign into VCSA with AD credentials​ .

Once after joining the domain, vCenter server may needs to be restarted, please do so.

[stinuz]

Rejoining the appliance solved the problem over here. The identity source was still in place and all users can logon again. I followed aavvalitorontoca's procedure. Thanks

[GreenEnvy22]

We had the same issue after upgrading our 6.5 appliance to 6.5 update 1.

Could not login with AD credentials. I could still search for AD users in the permissions screen, but couldn't login to vsphere with AD and our Veeam backups failed due to login as well.

I tried similar steps to what Jordon posted and was able to get back in, but after reboot it broke again. I had corrected the name in a different method than in step 5 here, so maybe it wasn't permanent.

I did it now like jordon posted. I'm running backups now and will reboot the VCSA once they are done to see if it is fixed permanently.

[GreenEnvy22]

After backups finished, I rebooted the VCSA again and login is broken again for AD credentials.

Logged in as local admin, and when I go back to the network settings under admin>system config>select node, I see hostname as localhost again.

So something keeps overwriting the hostname back to localhost.

Oddly, we have two other VCSA's in alternate sites, almost identically setup (same AD domain, similar number of hosts), and they upgraded perfectly fine, no issues at all.

[GreenEnvy22]

Not sure why I always figure out the issue 5 minutes after i submit a support request, but I did.

We had a typo in reverse DNS for this vcenter. I don't know why it didn't affect the VCSA in it's previous state (also 6.5, GA), but after fixing that and rebooting the VCSA the hostname now shows up properly and I can login with AD fine.

[Madhuin]

GreenEnvy,

Please have look into KB : vCenter HA cluster fails when upgraded/patched to vCenter Server Appliance 6.5 U1 (2151548) | VMware... 

Above KB article says on VCHA environment after upgrading to 6.5U1 etc hosts changing to localhost.localdom and I observed  in case normal VCSA scenario if reverse lookup  is not properly set/DNS resolution not working,  then you see the same issue,

workaround to fix the issue if you have already hit:

1. Modify /opt/vmware/etc/vami/vami_ovf_info.xml file the content
   
   <network ovf="unset"/>
   
   to
   
   <network ovf="unset" ovfeth0="false"/>
   
   
   
2. Modify /etc/hosts and /etc/hostname with correct hostname entries and reboot.

workaround to avoid issue in case of if you plan to upgrade to U1:

1) After the upgrade U1, before reboot

Modify /opt/vmware/etc/vami/vami_ovf_info.xml file the content

<network ovf="unset"/>

to

<network ovf="unset" ovfeth0="false"/>

and reboot the machine then you wont see the issue.

I tested in my lab environment it works perfectly fine. 

If it is useful, plz mark answer as correct or helpful.

----------------------------------------------------------------

Thanks & Regards

Madhukumar J, VCP50

-----------------------------------------------------------------

Disclaimer: Any views or opinions expressed here are strictly my own. I am solely responsible for all content published here. Content published here is not read, reviewed or approved in advance

[jayf628]

I have a very similar problem, after upgrading from 6.0 to 6.5 U1...  I created a lab environment to test our upgrade in a non-prod environment.

1. Installed fresh environment, added a couple of hosts and VMs, created a dvSwitch.  Version 6.0, using external PSC model
2. Upgraded secondary/backup PSC to 6.5 U1, then installed latest patch; currently at 6.5.0.12000 Build Number 7119157
3. Repointed to the upgraded PSC, then upgraded the other one
4. Upgraded VCSA in same manner

After upgrading the first PSC and then repointing the VCSA to it, noticed I could only log in with my domain credentials if I used the enhanced plugin.  Entering credentials manually will not work.  Same if I repoint to the second (upgraded) PSC.

I saw this thread, and tried to remove one of the PSCs from the AD domain, then re-add.  When I re-add, it adds with the wrong computername and creates an AD computer account with this same (wrong) name.  If I repeat the process (leave, then join domain again), it will use that same (wrong) computername, so the name is not randomly generated each time.  Command output below shows the issue.  I will note that the initial upgrade of the PSC showed the proper distinguished name when running the domainjoin-cli query command.

Note that when adding, it knows the correct hostname to add

root@psc01 [ ~ ]# /opt/likewise/bin/domainjoin-cli join region.prod.myrealdomain.com my_username

Joining to AD Domain:   region.prod.myrealdomain.com

With Computer DNS Name: psc01.mydomain.com

SUCCESS

But when it's been added, I query to confirm the proper setting, and the name is wrong

root@psc01 [ ~ ]# /opt/likewise/bin/domainjoin-cli query

Name = psc01

Domain = region.prod.myrealdomain.com

Distinguished Name = CN=PSC01-EM22TUB,CN=Computers,DC=region,DC=prod,DC=myrealdomain,DC=com

And when I look at the computer account that was created in AD, it too has the wrong name - PSC01-EM22TUB

Over the last 6-12 months, moving to 6.0, I have deployed probably 10-12 PSCs, and have never had any problem like this.  This is completely new and unique to 6.5 U1.  I don't know if it would've happened under 6.5 vanilla, because I upgraded right to 6.5 U1.

Where could it be getting this incorrect name from, when adding to the domain?  Hostname command returns the proper name, /etc/hosts and /etc/hostname have the proper name.  It seems this defect is coming from the domainjoin-cli join command itself.  I have opened a case w/ VMware Support and sent them the PSC and VCSA logs.

[jayf628]

I was able to work around my (similar) problem, where the hostname of the (upgraded to 6.5) appliance does not match the name that gets added to the AD domain.  What I did was to replace the domainjoin-cli binary with one from a pre-upgrade appliance (which happened to be on 6.0 U2).

1. System working normally
2. Upgraded PSC to 6.5 U1 and noticed I could not log in with AD credentials unless I was using enhanced plugin
3. Attempted to remove the PSC from the domain and re-add, but when appliance was added back to the AD domain it was being added with incorrect hostname, when hostname on the appliance was already correct
4. Removed appliance from AD domain and deleted its computer account from the domain
5. Replaced (6.5 U1) domainjoin-cli binary with one from a pre-upgrade (6.0 U2) appliance
6. Re-added appliance to the AD domain, confirmed it was added with correct name
7. Now able to log into vCenter using manually-entered AD credentials

Once the system was added using the old version of domainjoin-cli, I wanted to see if the 6.5 U1 version would work, so I removed from AD domain once again.  On re-adding, this time with the 6.5 U1 binary, it worked normally again.  I repeated this process on another appliance and it worked the same way, so I think this will be how it goes for the rest of the appliances that I need to upgrade.  Adding the appliance to the AD domain once, using the old binary, seems to be the workaround, because after that the new version of the binary works fine.  I asked VMware Support to send this information to their developers as a bug to be fixed.

[AWo]

I had the same. The PSC is in another domain than the AD domain but after the update to 6.5.0.23000 Build 10964411 the PSC was joined to the domain with the wrong FQDN (th eone from teh AD domain).

I just changed the DNS entry of the AD PSC server object via ADSI.

That did it, as well.

Cheers

AWo