No Permissions to Virtual Center

[benschaefer]

Ok, over the weekend we had a SAN fabric failure, that's been resolved, but in the mean time the Virtual Center has some how removed permissions for all my Administrators Roles!

From vpx log (usernames masked with *s):

2008-02-24 18:23:04.312 'BaseLibs' 3400 infoADS Failed to lookup account ***\Domain Admins (err: 1789, 16, 256)+

2008-02-24 18:23:04.312 'App' 3400 error Removing invalid permission 13: user ***\Domain Admins not found+

2008-02-24 18:23:04.343 'BaseLibs' 3400 info ADS Failed to lookup account ***\Enterprise Admins (err: 1789, 16, 256])+

2008-02-24 18:23:04.343 'App' 3400 error Removing invalid permission 22: user ***\Enterprise Admins not found+

2008-02-24 18:23:04.343 'BaseLibs' 3400 info ADS Failed to lookup account **\**(err: 1789, 16, 256)+

2008-02-24 18:23:04.343 'App' 3400 error Removing invalid permission 28: user **\**not found+

2008-02-24 18:23:04.359 'BaseLibs' 3400 info ADS Failed to lookup account **\**(err: 1789, 16, 256)+

2008-02-24 18:23:04.359 'App' 3400 error Removing invalid permission 27: user **\**not found+

2008-02-24 18:23:04.359 'BaseLibs' 3400 info ADS Failed to lookup account **\**(err: 1789, 16, 256)+

2008-02-24 18:23:04.359 'App' 3400 error- Removing invalid permission 29: user **\**not found+

2008-02-24 18:23:04.375 'BaseLibs' 3400 info ADS Failed to lookup account **\**(err: 1789, 16, 256)+

2008-02-24 18:23:04.375 'App' 3400 error Removing invalid permission 30: user **\**not found+

2008-02-24 18:23:04.375 'BaseLibs' 3400 info ADS] Failed to lookup account **\**(err: 1789, 16, 256)+

2008-02-24 18:23:04.375 'App' 3400 error Removing invalid permission 33: user **\**not found+

I now can authenticate when I login to Virtual Center, but it tells me I have no permission and returns me to the login screen. VC is not a DC but is a member of the top level domain. VC is not virtualized, but all DCs are. I think I've gotten my self into one of those chicken / egg situations here haven't I?

Currently I can see from the Hosts that VC functions are working, including vMotion and resource allocations. But I have now way of geting in.

Does anyone know if it's possible to readd a user through SQL or regedit or something? So far it looks like I'll have to do a reinstall!

Thanks!

Ben Schaefer

ILSC

[HyperSprite]

Have you tried logging into VC with the local windows adminstrator account?

[khughes]

Have you tried logging into VC with the local windows adminstrator account?

Logging in with the local admin to the VirtualCenter will give you access. From there you should be able to repair any permission issues you are having

[jhanekom]

Are your domain controllers running a non-english version of Windows? (i.e. is your "Domain Admins" group called something other than "Domain Admins"?)

VirtualCenter looks up Active Directory groups by name, not SID. So if the groups are renamed (or do not have the expected name), you're likely to run into this issue.

The solution is usually to log on with a local administrator account (as per previous post's suggestion) and re-add the groups you want to grant access to VirtualCenter.

[benschaefer]

I didn't give the full background, sorry.

VC was once a DC, but was demoted later to conserve resources... VC installation was done while it was a DC so it never created permissions for the Local Admin, demoting the box of course didn't add this or through any flags... Lucky for me we use esxRanger and when I installed that (after DC demote) I created a dedicated admin for backups! I was able to login to VC and recreate the previous permissions. Whew! I also added the Local Admin back into list...

So the problem is solved for me, but I had a lucky break, (had to dig for an hour for my esxRanger install notes for the password...) But what if I hadn't created this extra account? Would I still be reinstalling my infrastructure right now? There must be some kind of backdoor (hate that term) or something to get permissions back in. And why did VC delete my permissions in the first place, makes me very nervous to boot the VC before booting my DCs now...Can't justify a support ticket for it though.

Thanks to everyone that took the time to respond!

Ben Schaefer

ILSC.ca

[jhanekom]

The permissions are stored in the SQL database, specifically the VPX_ACCESS table. The group names are stored in plain text, and it doesn't take a rocket scientist to figure out what the rest of the columns are for

I'm not sure that VC "deleted" your permissions. I may be wrong, but I suspect it's amore a case of "OK, I've loaded this list of permissions from the database into memory, but I cannot find this user group, so I'm ignoring it." I've not been in a position to confirm whether that's the case or not, though.

What I find puzzling is that VC should still have found your domain's Domain Admin group, unless it was taken out of the domain completely and made a workgroup member.

[benschaefer]

Yeah I tried that, and it didn't stick, as soon as I restarted the VC service any lines I added would disappear, trying to login before restarting failed as well. This is where I discovered that 1 user remained, the one I created for the backup service. If this table was empty then I probably wouldn't have even tried, it's obvious to know the columns if they're populated Principal: User, Role: LU 'Roles', Entity_ID: 1= Entire ESX Structure, FLAG: 1 for User 3 for Group? But I have some data to go by now.

As I mentioned earlier, the scenario is pretty rare, but I don't have a test system (or time, or courage!) to try and repeat it. SAN fabric failed from over heating (thermal sensor has been ordered and is now in transit), crashing all the VMs. Remote trouble shooting indicated some kind of SCSI reservation conflicts; I could see the LUNs but could not connect to the VMFS'. Wasn't making too much sense without knowing the environmental factors. A few reboots, and then a visit to the DataCenter and some late hours and the system is up and running again, I can run VI client directly on ESX, but have no permissions to login to VC, I can see that VC is running as machines vMotion and balance across the Hosts as designed. Now, in the panic of the failed fabric, VC was restarted a few times while it's Domain was unavailable (all DCs are VMs), I had to log on to the VC with the Local Admin and not the usual account.

This scenario I now know how to prevent; make sure your VC is a DC (dcpromo will be run today), and/o rmake sure that you have a Local Machine Admin assigned to VC. During installation VC automatically adds the Domain Admin or the Local Admin to permissions list depending if it's a DC or not, ours was and later was demoted...

It's an odd set of events and makes sense to me now, including the vpx.log I posted in the beginning. Does VC modify permissions after X restarts? That makes sense, in an auto cleanup, self healing kind of way (too much Novell history there). It would have been nice to get a confirmation of some kind though and not have to dig for a few hours in the logs....