VCenter start fails after demoting Domain Controller

[goppi]

Hi.

Instead of hijacking other one's thread having the same problem I decided to start a new thread, so sorry for double posting.

After promoting a new DC and demoting the old one we have the problem that VCenter 5.1a no longer starts.

Taking a look at the imsTrace.log file of SSOServer we see the following line:

Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection olddc.ourdomain.com:3268

As we can see SSO still tries to contact the old dc now beeing a simple member server instead of the new one - the question is why?

We checked health of AD by running DCDIAG and BPA both not seeing any problems. Additionally we checked DNS by hand to ensure that no SRV or any other records are present which reference the old DC. Also all other AD dependent services like Exchange, SharePoint, DFS run without issue and do not indicate any problems.

We also found the krb5.conf file of SSO which still references the old DC by the following entry:

[realms]
OURDOMAIN.COM = {
    kdc = olddc.ourdomain.com
    default_domain = OURDOMAIN.COM
}

However changing that entry does not solve the problem because after a short period this entry is replaced automagically by ??? - that's the question.

Mybe someon else has some ideas what to check next.

Cheers.

[goppi]

Hi.

Here is what I have found additionally:

Taking a look at the SSO database there is a table named IMS_CONFIG_VALUES.

There I found two entries (ims.ldap-slots.0.primary-url and ims.ldap-slots.0-global.primary-url) still referencing the old

DC. So next I will try to change those values and see what happens next.

Stay tuned.

[goppi]

Hi again.

So I'm back on track now.

After changing those two values in the SSO database and restarting SSO service the krb5.conf file gets changed automagically to the correct values and VCenter starts and operates correctly again.

What still is misterious to me.

Why on earth are those values getting hardocded duing installation.

Kerberos also supports DNS-based configuration via SRV records which is in place for MS Active Directory by default.

Cheers.

[BeenGD]

I was having similar issues with vcenter after upgrading AD to server 2008 r2 and demoting our old server 2003 r2 DC's. Error's in the Application event logs indiciated:

Event Type:    Error
Event Source:    VMware VirtualCenter Server
Event Category:    None
Event ID:    1000
Date:        3/6/2013
Time:        9:54:10 AM
User:        N/A
Computer:    VCENTER SERVER
Description:
The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: The group account "DOMAIN\GROUP" could not be successfully resolved.  Check network connectivity to domain controllers and domain membership.  Users may not be able to log in until connectivity is restored..

And in System event logs:

Event Type:    Error
Event Source:    Service Control Manager
Event Category:    None
Event ID:    7024
Date:        3/6/2013
Time:        9:58:37 AM
User:        N/A
Computer:    VCENTER SERVER
Description:
The VMware VirtualCenter Server service terminated with service-specific error 2 (0x2).

I had already tried creating DNS aliases for the old DC, which did not seem to work. Changing the entries mentioned here in the SSO database corrected this issue. Thank you!!

[goppi]

Glad that it also worked for you.

Maybe you could mark the right answer, so for others it would be easier

to find it if they were facing the same problem.

Thanks.

Cheers.

[BeenGD]

As far as I can tell the question has already been marked as answered! I don't see any other options than replying, hopefully anyone else having the same problem will see the fix you posted above. Thanks again!

[dbutch1976]

Hello,

I'm not a DBA, could you go into detail as to how to searched for and then modified these values?  Also, were you using MS SQL as your back-end?

Thanks.

[BeenGD]

MS SQL is the backend database, yes. If you have access to the DB server, the easiest way to view/modify the table is via MS SQL Server Management Studio.

- Open MS SQL Server Management Studio and point it to the server/instance that houses your vcenter SSO database.

- Expand the Databases object, then expand the vcenter SSO database (this could go by any name, if you're unsure you may have to check within multiple DBs)

- Expand the tables object and look for the table with "IMS_CONFIG_VALUE" in the name

- Right click on that table and choose edit top 200 rows

- When those are displayed you can find and edit the fields referenced, which will change where the vmware SSO service looks to bind to AD. In my case I altered:

ims.ldap-slots.0.primary-url

ims.ldap-slots.0-global.primary-url

ims.ldap-slots.0-global.secondary-url

Probably a good idea to note the current values and double check the new values you enter. Changes made are effective immediately.

[dbutch1976]

This is crazy!  After decommissioning DC2.dc.local I'm having the exact same problem that I had all those months ago as shown by the logs:

pi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
Caused by: java.net.SocketException: Connection reset
com.rsa.common.ConnectionException: Error connecting to the identity source
Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269]
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
Caused by: java.net.SocketException: Connection reset

The solution is simple, change the static references to the new domain controller, P1.dc.local, but believe it or not I simply can't FIND what my SSO database is using.  I believe I'm using my Windows 2008 R2 SQL server, but when I Management Studio and browse to the RSA database --> tables it's blank.  Perhaps I used a local database instead, but I'm having trouble confiirming that also.  Is there a place in the logs which will tell me definititivaly what server is hosting the SSO database back-end?

Many thanks for your help.

[BeenGD]

I'm sure there are other ways, but the first thing that came to mind was to check the .log file from installation. Find the "vim-sso-msi.log" file and do a search for "JDBC_URL property", this will contain a string that will tell you the server name, port and db name that SSO was setup to use. That should point you in the right direction.

Ben

[dbutch1976]

Thanks for the tip, I found it by using a slightly different method, I checked the database configuration information file located here:

C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties

bc Url
db.url=jdbc:jtds:sqlserver://;serverName=;portNumber=63105;databaseName=RSA
## DB Username
db.user=RSA_USER
## DB password
db.pass=i.yl1hBdKI
## DB type
db.type=mssql
## DB host
db.host=VC51.DC.local

More info on where to find these files can be found here:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203343...

So now I know (I believe) my SSO database runs off MYSQL on the vCenter itself.  I'm assuming in order to modify it I'm going to need to install the management studio on my vCenter.  What version of MySQL does the SSO installer install?  I normally run SQL 2008 R2 in my environment, will I be able to manage the database using that?  Also, does anyone know the credentials that are assigned by default?  Do I log in with SA?

[goppi]

Not sure why you are talking about mysql?

From what I can see you are using mssql so install Management Studio Express and open the database.

You can use sa with MSSQL authentication or Windows Administrator when connecting via Windows authentication.

Cheers.

[krismcewan]

Also worth checking if you DC has an SSL cert that the DC cert has not expired.

Particularly prudent if you do not have auto enroll switched on.

you usually have an issue about binding but worth checking

[felakkad]

For vCenter 5.5

Hi to inform you that i get those in error when i move to another domain a VC and change the IP in a test environement, SSO failed to connect to the vcenter because it keep trying to connect to the the old nameof the vcenter

Error : ConnectComplete] Connect failed to <cs p:000000000a044270, TCP:MYVC.OLDdomaine:7444>; cnx: (null), error: class Vmacore::SystemException(Hôte inconnu)

To solve that i put in the host file the old DNS mapped to the new IP

Then vCenter restart

And in the webClient, in administration delete the old domain and add the new one.

[ewanat]

Thanks. Great help! I didn't demote a DC but had an issue with secure LDAP on one of the DC's.

So temporary changed the entries in the DB until I figure out why it is not working.

[picklewithanO]

Awesome article. I had the exact issue and this article resolved it. Thanks,

[mark_chuman]

*bump on this on*

Anyone aware of where the targeted domain controller is registered in SSO 5.5?

Trying to search through all files in Windows, but running into the dreaded, Powershell memory exceeded conundrum.