VMware Cloud Community
emtunc
Contributor
Contributor

vCenter 5.1 SSO and failure to successfully authenticate users

I have experienced this problem with the 5.1 upgrade including the latest 5.1.0b. In fact the latest patch makes it impossible to add users explicitly via the vSphere client whereas previously I could do this.

The problem description is as follows:

  • Users do not authenticate via the web client or via vSphere client
  • Users receive the error "Cannot complete login due to an incorrect user name or password" on the vSphere client
  • Users receive the error "The authentication server returned an unexpected error: ns0:RequestFailed: Internal Error while creating SAML 2.0 Token. The error may be caused by a malfunctioning identity source.

Things I have tried:

  • I have followed this article but the solution re: RPC server not available does not apply as this does not appear in the logs @ http://kb.vmware.com/kb/2034798
  • Added new domain users
  • Explicitly added users in the vSphere client giving them full administrative privileges and propogating to all child objects
  • Domain groups which the users are part of (IT Operations) are added under SSO administrators group (SSO Users and Groups --> Groups --> __Administrators__ have the principals 'ITOperations' and 'Domain Admins' - they are both Active Directory groups)

Observations:

  • Putting the user in the 'Domain Admins' group allows the user to successfully log-in to both vSphere and web clients - obviously this is not a practical solution to the problem but unsure as to why it works - members of the IT Operations group can successfully log-in to the vCenter server so also unsure as to what permissions would be required for this to work.
  • Granting users explicit access via the vSphere client used to work in previous version of 5.1.0 - with 5.1.0b the users get the "Cannot complete login" error

Our set-up:

  • vCenter sits on Server 2008 R2 Enterprise
  • Active Directory runs in our environment and handles all log-ins - SSO should be set-up to intergrate itself with AD but not sure why it's not working
  • 1 ESXi 5.0 host

imsTrace log:

2013-01-03 13:48:51,781, [castle-exec-1], (LocalisAccessHelper.java:567), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG, vCenter.Company.local,,,,Invoking GetAllLocalOSDomains() Local OS call
2013-01-03 13:48:51,842, [castle-exec-1], (LocalisAccessHelper.java:575), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG, vCenter.Company.local,,,,GetAllLocalOSDomains Local OS call status 0
2013-01-03 13:48:51,855, [castle-exec-1], (LocalisAccessHelper.java:523), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG, vCenter.Company.local,,,,Invoking GetUserGroupsByName(COMPANY\vspheretest) Local OS call
2013-01-03 13:48:51,958, [castle-exec-1], (LocalisAccessHelper.java:531), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG, vCenter.Company.local,,,,GetUserGroupsByName Local OS call status 6
2013-01-03 13:48:51,964, [castle-exec-1], (GroupAccessLocalIS.java:313), trace.com.rsa.ims.admin.dal.localis.PrincipalAccessLocalIS, DEBUG, vCenter.Company.local,,,,Lookup failure: [GroupInfo.c:254] NetUserGetLocalGroups failed: Access is denied.

2013-01-03 13:48:51,969, [castle-exec-1], (SecurityTokenServiceImpl.java:117), trace.com.rsa.riat.sts.impl.SecurityTokenServiceImpl, ERROR, vCenter.Company.local,,,,Error while trying to generate RequestSecurityTokenResponse
com.rsa.common.UnexpectedDataStoreException: Unexpected Local OS exception
    Caused by: com.rsa.ims.localis.LocalisAccessError: Local O/S Identity Source Error: LOCALIS_STATUS_INTERNAL, extended error: 5 : [GroupInfo.c:254] NetUserGetLocalGroups failed: Access is denied.

imsRuntimeAudit log:

2013-01-03 13:48:51,580, <longstring1>,<longstring2>,,192.168.0.110,AUTHN_LOGIN_EVENT,13002,SUCCESS,AUTHN_METHOD_SUCCESS,<longstring3>,<longstring4>,<longstring5>,<longstring6>,vspheretest,vspheretest,SYSTEM,,,,,,000000000000000000001000f0022001,LDAP_Password,,,,,,,,,,,,,

vpxd log:

2013-01-03T13:48:50.565Z [00620 info '[SSO]' opID=C001001B-00000004-3b] [UserDirectorySso] Authenticate(vspheretest, "not shown")
2013-01-03T13:48:52.049Z [00620 error '[SSO]' opID=C001001B-00000004-3b] [UserDirectorySso] AcquireToken SsoException: Unexpected SOAP fault: ns0:RequestFailed; request failed.
2013-01-03T13:48:52.049Z [00620 error 'authvpxdUser' opID=C001001B-00000004-3b] Failed to authenticate user <vspheretest>
2013-01-03T13:48:56.051Z [00620 info 'commonvpxLro' opID=C001001B-00000004-3b] [VpxLRO] -- FINISH task-internal-574 --  -- vim.SessionManager.login --
2013-01-03T13:48:56.051Z [00620 info 'Default' opID=C001001B-00000004-3b] [VpxLRO] -- ERROR task-internal-574 --  -- vim.SessionManager.login: vim.fault.InvalidLogin:
--> Result:
--> (vim.fault.InvalidLogin) {
-->    dynamicType = <unset>,
-->    faultCause = (vmodl.MethodFault) null,
-->    msg = "",
--> }
--> Args:
-->

Tags (4)
18 Replies
yxcvyxcv
Contributor
Contributor

hi,

do you have a solution for this problem? i have exactly the same issue

thanks

markus

Reply
0 Kudos
yxcvyxcv
Contributor
Contributor

for me this solved the problem:

Regardless of the cause of NetUserGetLocalGroups failure, removing the local identity source will allow domain users to  log in. Before doing this, you must ensure that at least one domain user  has full Administrator privileges for the vCenter Server. By default,  only the local Administrators group has these privileges.  Removing the  local identity source causes local users to be unable to log in to  vCenter Server. All permissions associated with local users and groups  will be deleted when vCenter Server is next restarted.

Reply
0 Kudos
PJVanMeerbeeck
Contributor
Contributor

Hi,
we had the same problem and we think we solved it by assigning the service account the necesary rights in AD DS.
the service account needs read acces on the Account Restrictions Propertyset , tokengroups attribuut and memberof attribute of all accounts that
need to login.
In powershell execute:
(dsacls "$ou"  /I:S /G "${n}:RP;Account Restrictions;user")[-1]
(dsacls "$ou"  /I:S /G "${n}:RP;tokenGroups;user")[-1]
(dsacls "$ou"  /I:S /G "${n}:RP;memberOf;user")[-1]
where $ou is the ou containing the accounts that need access to vCenter
$n is the name of the service account.
Reply
0 Kudos
Schaedle
Enthusiast
Enthusiast

We had the same problem here and it really helped to remove the local identy. The problem only occurs when the vcenter server is a member of a domain.

I got it solved by vmware support. The solution shall be published via KB: 2043070 but thsi KB is not yet online, but shall be soon.

Regards Wolfgang

Reply
0 Kudos
emtunc
Contributor
Contributor

Thanks Schaedle and other posters. My preference is to have an official fix for this problem. I don't really want to start messing around with access controls unless I really have to.

@Schaedle can you briefly describe the fix support have offered you? I checked the KB for article '2043070' but nothing up there as of yet Smiley Sad

Reply
0 Kudos
Schaedle
Enthusiast
Enthusiast

Our vShpere server had three local identity sources.

* Active Directory (please set correct credentials)

* system-domain (this one was newley created by the installtion)

* local users of the vCenter server

The problem was the local one of the vCenter server. After deleting this one all went fine. This one onlay appears if the vCenter server is a domain member ( I was told by vmware support).

Have luck !

Regards Wolfgang

Reply
0 Kudos
PJVanMeerbeeck
Contributor
Contributor

The problem on our side was that the service account did not had enough rights for querying the users in AD. In the audit log on AD I noticed that the service Account was missing read permissions on tokenGroups attribute and Account Restrictions propertys set.

Afther granting these permissions on AD we had still the same problem. but no extra aduit failures were logged in AD. If i made the service account member of the "pre-windows 2000 compatible access" group the problem was solved, users coula login to vCenter. This is not a desired configuration cause, this way, the service account got a lot of acces rights in AD and according the principle of least privilege I only give service accounts the rights needed/used.

Anyway to make a long story short. The service account was missing read acces on the "member of" attribute of the users who wanted to login to vCenter. granting this read rights solved the problem. probaly because vCenter uses the ms-smar function "openuser" with a desired acces of

USER_READ_GROUP_INFORMATION or USER_LIST_GROUPS, cf. http://msdn.microsoft.com/en-us/library/cc245476.aspx,

The lack of this right in AD was not logged in the audit logs on AD.

I don't see how a local identity sources interferes with this problem.
.

We are probaly not talking about the same problem. I thought the problem was related because of the phrase

"Putting the user in the 'Domain Admins' group allows the user to successfully log-in to both vSphere and web clients"

What kind of rights has your service account in your AD environment?

My reply was mainly a reaction on http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203754...
wich resulted in the same error log

com.rsa.common.UnexpectedDataStoreException: Unexpected Local OS exception
  Caused by: com.rsa.ims.localis.LocalisAccessError: Local O/S Identity Source Error: LOCALIS_STATUS_INTERNAL, extended error: 5 : [GroupInfo.c:254] NetUserGetLocalGroups failed: Access is denied.

As in the first post.

Regards,

Reply
0 Kudos
Schaedle
Enthusiast
Enthusiast

Our account which queries the AD has more right than just read some information, because it is used for other purposes. Since it was not clear what the problem is I used a domain admin for testing - just to be sure it has nothing to do with access rights.

"NetUserGetLocalGroups failed: Access is denied." was exactly our problem.

Regards Wolfgang

Reply
0 Kudos
kgraw21
Contributor
Contributor

I did the following and was able to fix the problem immediatley.

Check the following logs:

C:\Program Files\VMware\Infrastructure\SSOServer\logs\ssoAdminServer.log

2013-01-23 11:19:37,309 ERROR opID=DF1C47AC-00000005-1e pool-31-thread-8  com.vmware.vim.sso.groupcheck.vlsi.GroupCheckServiceImpl] Unexpected Local OS exception

com.rsa.common.UnexpectedDataStoreException: Unexpected Local OS exception

Caused by: com.rsa.ims.localis.LocalisAccessError: Local O/S Identity Source Error: LOCALIS_STATUS_INTERNAL, extended error: 1722 : [GroupInfo.c:254]

NetUserGetLocalGroups failed: The RPC server is unavailable.

C:\Program Files\VMware\Infrastructure\SSOServer\logs\imstrace.log

2013-01-23 15:30:10,303, [pool-31-thread-4], (LocalisAccessHelper.java:523), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG, 0.0.0.0,,,,Invoking GetUserGroupsByName(Domain\username) Local OS call

2013-01-23 15:30:27,254, [pool-31-thread-4], (LocalisAccessHelper.java:531), trace.com.rsa.ims.localis.LocalisAccessHelper, DEBUG,  0.0.0.0,,,,GetUserGroupsByName Local OS call status 6

2013-01-23 15:30:27,254, [pool-31-thread-4], (GroupAccessLocalIS.java:313), trace.com.rsa.ims.admin.dal.localis.PrincipalAccessLocalIS, DEBUG,  0.0.0.0,,,,Lookup failure: [GroupInfo.c:254] NetUserGetLocalGroups failed: The RPC server is unavailable

Note: 0.0.0.0 is IP of VC

Then do the following:

  • Ensure Port 135 is open on your DC using the netstat -a -t cmd.
  • Ensure Netbios over TCP/IP v4 is enabled on the VC Windows Server.
  • Add forest/domain to DNS Suffixes on the VC Windows Server. (All domains must be added to DNS suffix which are added as a identity source in the SSO configuration).

See: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203479... for more info

Message was edited by: kgraw21

fletch00
Enthusiast
Enthusiast

KB 2043070 is up now and works perfectly for this issue in my case (eventhough I did not see the same error messages in the logs)

Now my users can be delegated vCenter rights and login again!

thanks for that reference

VCP5 VSP5 VTSP5 vExpert http://vmadmin.info
Reply
0 Kudos
Raman_Shcharbak
Contributor
Contributor

Just registered to say thanks to PJVanMeerbeeck for the solition related to user permissions.

Removing  local identity source fixes one problem but it will allow domain users to  log in ONLY IF they have permissions granted in vCenter directly to their accounts.

But there is another problem:

If the user is granted access in vCenter via group and the user is not a Domain Admin, then this user won't be able to log in.

As PJVanMeerbeeck stated, this is related to the permissions that service account has regarding reading other user accounts' properties.

For some reason, domain admin accounts have permissions configured on them that allow all authenticated users to read more properties than can be read from a regular account. Thus SSO service account can read required properties like Restrictions and Group Membership from domain admins' accounts. But it cannot get this info for non-Domain Admins users. (this results in "NetUserGetLocalGroups failed: Access is denied." error)

If you give permissions to some "VMUsers" group in vcenter and this group has a domain admin  and non-admin account in it, then domain admin would be able to successfully login while non-admin would not.

In my deployment SSO is configured to run under "network service" which means that it will access AD using its computer account credentials.

What I did to fix the problem:

1. Created a group called "Vmware SSO Servers", added vcenter computer to it (SSO and Vcenter run on the same computer) and rebooted the computer so that new group membership applied.

2. Created additional delegation to this group and allowed reading all user properties (on the OU that has in it all user accounts that need access  to vCenter).

You can skip step 1 and just delegate these permissions directly to the service account (or VC Computer) that you are using.

Or you can add your service account to the "pre-windows 2000 compatible access" group as it already has access to the required properties. If it is "Network Service', then don't forget to reboot computer for the membership to apply. (as PJVanMeerBeeck has noted, this not good according to  the principle of least privilege.

Another thing I'd like to notice, that it is confusing what accounts services run under.

"vCenter Single Sign On" Service runs under local system.

It is VMwareVCMSDS that runs under the account that you configure during installation.

I spent some time trying to run "vCenter Single Sign On" service under Network Service account as I thought that it was an installation bug.

Hope this helps.

Reply
0 Kudos
Gene_H
Enthusiast
Enthusiast

We had this issue.  Our solution was to change the "vCenter single sign on windows service" to logon using a domain user account rather than the local system account.

This was documented by vmware in their vSphere 5 documentation.

"Add a vCenter Single Sign On Source"

http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-B23B1360...

"This type of authentication is supported only if the identity source is an Active Directory server and the Single Sign On service runs as a user that has been authenticated against the same Windows domain to which the Active Directory server belongs"

After making this change and rebooting the vCenter server, the vSphere Client windows authentication "checkbox" feature started working.

Reply
0 Kudos
FSvcoe
Enthusiast
Enthusiast

What an annoying problem. We have a VC housed in one domain, with the user accounts located in another. Even after creating the corresponding Identity Sources within the Webclient SSO, users could not login, but a user account from the VC account could.This started after a change in the AD DC topology and it required removing and recreating the Identity Source.

After removing the Local Identity Source as outlined in the KB (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=204307...), this fixed it for us also. I didn't try setting it to a domain account for the VMware vSphere Web Client Service, but sounds like that's another workaround.

Reply
0 Kudos
firestartah
Virtuoso
Virtuoso

Hi,

I was seeing kinda the same problem on a client site recently but ours was linked to CommVault Auto-Discovery not being able to discover the hosts and datastores managed by a newly upgraded vCenter 5.1 server and getting "incorrect username or password" errors.

I wrote a blog about it here for someone experiencing the saem problems as we were:

http://thesaffageek.co.uk/2013/04/18/vcenter-5-1-sso-default-domain-clashes-if-domain-is-added-to-us...

Gregg

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful". Gregg http://thesaffageek.co.uk
Reply
0 Kudos
nareshunik
Enthusiast
Enthusiast

I am also getting the same error message, service account not able to access the vCenter....

But able to access vCenter application ,when they directly RDP to vCenter and access from vSphere client it works fine. but not remotly.

Pls help.

Reply
0 Kudos
Virtual_Martin
Contributor
Contributor

Thanks!

For me, adding of dns suffixes solved the problem immediately. Nothing more was neccessary!

Add forest/domain to DNS Suffixes on the VC Windows Server. (All domains must be added to DNS suffix which are added as a identity source in the SSO configuration).

Reply
0 Kudos
ADBrennick
Contributor
Contributor

I tried following KB2048177, but there are no options under Sign-On and Discovery. I can expand and collapes the folder, but there is no option to click on for Configuration. Has anyone else seen this?

Reply
0 Kudos
dalexiev
Enthusiast
Enthusiast

For me, it worked user@domain.com but not domain\user

5.1 u2c - upgraded linked mode from 5.0

Reply
0 Kudos