I upgraded my vCenter Appliance from 6.5e to update 1 yesterday and upgraded the ESXi hosts from 6.5d to 6.5 update 1. There were no issues/errors during the upgrade.
Before performing the upgrade I have been logging into the Flash or HTML5 web client using my AD credentials. My VCSA is domain joined and only had an IPv4 address.
After the upgrade I added an IPv6 address to the VCSA and can no longer login to the VCSA using my AD credentials, I get the "Invalid credentials" error after clicking the Login button. This only started happening after adding a static IPv6 address to the VCSA. I did add a reverse DNS record for the VCSA FQDN. When I only had an IPv4 address SSO worked but now that I have dual IPv4/IPv6 addresses AD logins are failing and the only way for me to login is to use the administrator@domain.local account.
If I run the following on the VCSA while logged in as root and using the shell:
cat /var/log/messages | grep -i "error"
I get this:
2017-07-29T07:08:59.463241+00:00 localhost ntpd[24585]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
2017-07-29T07:26:38.000442+00:00 localhost lsassd[1129]: 0x7f898115c700:Failed to sync system time [error code: 40075]
2017-07-29T06:16:33.103697+00:00 localhost cli: root SSO initialization error: [Errno 111] Connection refused
This got me thinking about time...
So I checked my VCSA time and it was one hour off compared to the ESXi servers and domain but the VCSA does use an NTP server (which is my domain controller). I have set both IPv4/IPv6 addresses for the NTP server.
So I logged into https://vcsa.domain:5480/ and it was using the correct NTP settings and syncing but it was one hour off. The strange thing is, the time zone field is empty. I tried setting it a few times (and rebooting) but its still empty.
Not sure if the lack of a time zone is the cause of this issue but how can I get my SSO with AD working again?
I also tried disabling NTP and setting the time manually on the VCSA but it kept changing back to the incorrect time (even after a reboot).
All my SSO logins are currently broken and I can't do Citrix MCS as it keeps saying the username/password is incorrect so plenty has gone wrong.
Is this a time issue and if so how do I resolve it? Are there any other logs that would be helpful in troubleshooting this?
Added root SSO connection refused error from log
Having the same issue except for the IPv6 portion; I'm not using IPv6 in anyway.
After the 6.5U1 upgrade I was no longer able to authenticate to my AD domain. The VCSA appeared to still be attached to the domain, but authenticating to the identity source fails with 'invalid credentials'. Logging into the console of the appliance I notice right away that the hostname had been reverted to localhost.localdom, I recall specifically having to manually change this prior to attaching it to the domain previously. I also noticed the clock was off; both my DC and the VCSA are pointed to the same NTP server, and both are set to UTC but for some reason the VCSA was off by 7 hours and a few minutes. I attempted the following steps with no such luck:
I don't have access to the logs at the moment but will post them later, I'm seeing the auth failures but am unable to determine the root cause. I suspect it has something to do with the clock and/or the hostname being reverted back to localhost.localdom when the upgrade is applied.
This is already addressed in the post
Thanks Camero, my OVF issue is now fixed.
In regards to the AD auth problem. My date and time are correct but my hostname has been set to localhost.localdom. I have always only used the FQDN during installs, migrations etc. I am hopefully having a dial in with VMware this morning so i will update later on their findings.
Has everybody gotten their environment fixed? I haven't seen any updates in several days of any official fixes. I have the same behavior where if I ssh and run hostname is has the correct FQDN. After update U1 it reverts to localhost.localdom and I can no longer login with AD credentials. Want to get this fixed in our test environment first before moving forward with production.
I have been able to fix my Test environment by doing the following. My problem is the hostname was changed somehow after the update to U1. Looked correct everywhere except when checking using ssh. This is what I did to fix
1. First I logged into the Web Console with administrator@vsphere.local
2. Go to Administration and go to configuration and look at the certificates. Check if they are they for the correct FQDN
3. Next Go to hosts and cluster and right click on top VCenter Name and go to settings. Expand Runtime Settings. Check if the correct FQDN name is there also.
4. After all those look good SSH into the VCenter server and run the command hostname. If localhost.localdom comes up U1 probably defaulted to this hosts name. It needs to be changed back to what your certificate says.
5.
Example:
/opt/vmware/share/vami/vami_set_hostname vcenter01.company.com
This worked for me. You may have to remove the identity and re-add it but I did not. Once my host name matched the certificate I was able to log in again with ad credentials.
Just finishing up with support on the phone. We may have foudn the issue of why it reverted to localhost.localdom. During the boot process in 6.5 U1 the technician found the following.
#
# reverse lookup the hostname from the IP address. If it can't be resolved,
# set the hostname to localhost. Wait for a max of 10 seconds if there is
# no reply then set the hostname to localhost
#
# prefer an IPv4 reverse lookup over an IPv6 reverse lookup
#
After I added a reverse lookup IP in DNS and rebooted the VCenter server the hostname reverted to the correct name and I could log in.
Thank you very much JordonC, this worked perfectly for me! My VCSA now has the correct FQDN and I was able to join the domain and add the AD identity source using the machine account first time. Now I can login using SSO with the enhanced client plug-in.
Thank you!
Great.
Dear all
I have checked all the settings, discussed above. Our vCenter (6.5 U1) shows correct name on running "hostname" in shell. DNS Forward and PRT Records are present and correct. All settings in vCenter itself show the correct FQDN hostname.
Anyhow, I'm still unable to login vCenter using AD credentials. It's always telling me "Invalid credentials". We have tried with different credentials and the passwords are entered correctly.
I have tried to remove the vCenter from domain running "domainjoin-cli leave". The I get the following error message:
Error: ERROR_MEMBER_NOT_IN_GROUP [code 0x00000529]
Any idea, what I can do to get AD sign in back working? This problem just started today. I guess it's related to U1 of 6.5.
Thanks for any help.
Please use a standard NTP server to avoid such time issues going forward. And make sure to set every entity (ESXi hosts, vCenter & AD etc) in the environment to use same NTP server.
Once the time related issues are resolved, then proceed with the all of the steps as mentioned by Jordon in comment# Re: Can't sign into VCSA with AD credentials .
Once after joining the domain, vCenter server may needs to be restarted, please do so.
Rejoining the appliance solved the problem over here. The identity source was still in place and all users can logon again. I followed aavvalitorontoca's procedure. Thanks
We had the same issue after upgrading our 6.5 appliance to 6.5 update 1.
Could not login with AD credentials. I could still search for AD users in the permissions screen, but couldn't login to vsphere with AD and our Veeam backups failed due to login as well.
I tried similar steps to what Jordon posted and was able to get back in, but after reboot it broke again. I had corrected the name in a different method than in step 5 here, so maybe it wasn't permanent.
I did it now like jordon posted. I'm running backups now and will reboot the VCSA once they are done to see if it is fixed permanently.
After backups finished, I rebooted the VCSA again and login is broken again for AD credentials.
Logged in as local admin, and when I go back to the network settings under admin>system config>select node, I see hostname as localhost again.
So something keeps overwriting the hostname back to localhost.
Oddly, we have two other VCSA's in alternate sites, almost identically setup (same AD domain, similar number of hosts), and they upgraded perfectly fine, no issues at all.
Not sure why I always figure out the issue 5 minutes after i submit a support request, but I did.
We had a typo in reverse DNS for this vcenter. I don't know why it didn't affect the VCSA in it's previous state (also 6.5, GA), but after fixing that and rebooting the VCSA the hostname now shows up properly and I can login with AD fine.
GreenEnvy,
Please have look into KB : vCenter HA cluster fails when upgraded/patched to vCenter Server Appliance 6.5 U1 (2151548) | VMware...
Above KB article says on VCHA environment after upgrading to 6.5U1 etc hosts changing to localhost.localdom and I observed in case normal VCSA scenario if reverse lookup is not properly set/DNS resolution not working, then you see the same issue,
workaround to fix the issue if you have already hit:
workaround to avoid issue in case of if you plan to upgrade to U1:
1) After the upgrade U1, before reboot
Modify /opt/vmware/etc/vami/vami_ovf_info.xml file the content
<network ovf="unset"/>
to
<network ovf="unset" ovfeth0="false"/>
and reboot the machine then you wont see the issue.
I tested in my lab environment it works perfectly fine.
If it is useful, plz mark answer as correct or helpful.
----------------------------------------------------------------
Thanks & Regards
Madhukumar J, VCP50
-----------------------------------------------------------------
Disclaimer: Any views or opinions expressed here are strictly my own. I am solely responsible for all content published here. Content published here is not read, reviewed or approved in advance
I have a very similar problem, after upgrading from 6.0 to 6.5 U1... I created a lab environment to test our upgrade in a non-prod environment.
After upgrading the first PSC and then repointing the VCSA to it, noticed I could only log in with my domain credentials if I used the enhanced plugin. Entering credentials manually will not work. Same if I repoint to the second (upgraded) PSC.
I saw this thread, and tried to remove one of the PSCs from the AD domain, then re-add. When I re-add, it adds with the wrong computername and creates an AD computer account with this same (wrong) name. If I repeat the process (leave, then join domain again), it will use that same (wrong) computername, so the name is not randomly generated each time. Command output below shows the issue. I will note that the initial upgrade of the PSC showed the proper distinguished name when running the domainjoin-cli query command.
Note that when adding, it knows the correct hostname to add
root@psc01 [ ~ ]# /opt/likewise/bin/domainjoin-cli join region.prod.myrealdomain.com my_username
Joining to AD Domain: region.prod.myrealdomain.com
With Computer DNS Name: psc01.mydomain.com
SUCCESS
But when it's been added, I query to confirm the proper setting, and the name is wrong
root@psc01 [ ~ ]# /opt/likewise/bin/domainjoin-cli query
Name = psc01
Domain = region.prod.myrealdomain.com
Distinguished Name = CN=PSC01-EM22TUB,CN=Computers,DC=region,DC=prod,DC=myrealdomain,DC=com
And when I look at the computer account that was created in AD, it too has the wrong name - PSC01-EM22TUB
Over the last 6-12 months, moving to 6.0, I have deployed probably 10-12 PSCs, and have never had any problem like this. This is completely new and unique to 6.5 U1. I don't know if it would've happened under 6.5 vanilla, because I upgraded right to 6.5 U1.
Where could it be getting this incorrect name from, when adding to the domain? Hostname command returns the proper name, /etc/hosts and /etc/hostname have the proper name. It seems this defect is coming from the domainjoin-cli join command itself. I have opened a case w/ VMware Support and sent them the PSC and VCSA logs.
I was able to work around my (similar) problem, where the hostname of the (upgraded to 6.5) appliance does not match the name that gets added to the AD domain. What I did was to replace the domainjoin-cli binary with one from a pre-upgrade appliance (which happened to be on 6.0 U2).
Once the system was added using the old version of domainjoin-cli, I wanted to see if the 6.5 U1 version would work, so I removed from AD domain once again. On re-adding, this time with the 6.5 U1 binary, it worked normally again. I repeated this process on another appliance and it worked the same way, so I think this will be how it goes for the rest of the appliances that I need to upgrade. Adding the appliance to the AD domain once, using the old binary, seems to be the workaround, because after that the new version of the binary works fine. I asked VMware Support to send this information to their developers as a bug to be fixed.
I had the same. The PSC is in another domain than the AD domain but after the update to 6.5.0.23000 Build 10964411 the PSC was joined to the domain with the wrong FQDN (th eone from teh AD domain).
I just changed the DNS entry of the AD PSC server object via ADSI.
That did it, as well.
Cheers
AWo