vCenter

 View Only
Expand all | Collapse all

VCenter start fails after demoting Domain Controller

  • 1.  VCenter start fails after demoting Domain Controller

    Posted Jan 12, 2013 05:54 PM

    Hi.

    Instead of hijacking other one's thread having the same problem I decided to start a new thread, so sorry for double posting.

    After promoting a new DC and demoting the old one we have the problem that VCenter 5.1a no longer starts.

    Taking a look at the imsTrace.log file of SSOServer we see the following line:

    Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection olddc.ourdomain.com:3268

    As we can see SSO still tries to contact the old dc now beeing a simple member server instead of the new one - the question is why?

    We checked health of AD by running DCDIAG and BPA both not seeing any problems. Additionally we checked DNS by hand to ensure that no SRV or any other records are present which reference the old DC. Also all other AD dependent services like Exchange, SharePoint, DFS run without issue and do not indicate any problems.

    We also found the krb5.conf file of SSO which still references the old DC by the following entry:

    [realms]
    OURDOMAIN.COM = {
        kdc = olddc.ourdomain.com
        default_domain = OURDOMAIN.COM
    }

    However changing that entry does not solve the problem because after a short period this entry is replaced automagically by ??? - that's the question.

    Mybe someon else has some ideas what to check next.

    Cheers.



  • 2.  RE: VCenter start fails after demoting Domain Controller

    Posted Jan 12, 2013 06:08 PM

    Hi.

    Here is what I have found additionally:

    Taking a look at the SSO database there is a table named IMS_CONFIG_VALUES.

    There I found two entries (ims.ldap-slots.0.primary-url and ims.ldap-slots.0-global.primary-url) still referencing the old

    DC. So next I will try to change those values and see what happens next.

    Stay tuned.



  • 3.  RE: VCenter start fails after demoting Domain Controller

    Posted Jan 12, 2013 06:19 PM

    Hi again.

    So I'm back on track now.

    After changing those two values in the SSO database and restarting SSO service the krb5.conf file gets changed automagically to the correct values and VCenter starts and operates correctly again.

    What still is misterious to me.

    Why on earth are those values getting hardocded duing installation.

    Kerberos also supports DNS-based configuration via SRV records which is in place for MS Active Directory by default.

    Cheers.



  • 4.  RE: VCenter start fails after demoting Domain Controller

    Posted Mar 06, 2013 03:15 PM

    I was having similar issues with vcenter after upgrading AD to server 2008 r2 and demoting our old server 2003 r2 DC's. Error's in the Application event logs indiciated:

    Event Type:    Error
    Event Source:    VMware VirtualCenter Server
    Event Category:    None
    Event ID:    1000
    Date:        3/6/2013
    Time:        9:54:10 AM
    User:        N/A
    Computer:    VCENTER SERVER
    Description:
    The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: The group account "DOMAIN\GROUP" could not be successfully resolved.  Check network connectivity to domain controllers and domain membership.  Users may not be able to log in until connectivity is restored..

    And in System event logs:

    Event Type:    Error
    Event Source:    Service Control Manager
    Event Category:    None
    Event ID:    7024
    Date:        3/6/2013
    Time:        9:58:37 AM
    User:        N/A
    Computer:    VCENTER SERVER
    Description:
    The VMware VirtualCenter Server service terminated with service-specific error 2 (0x2).

    I had already tried creating DNS aliases for the old DC, which did not seem to work. Changing the entries mentioned here in the SSO database corrected this issue. Thank you!!



  • 5.  RE: VCenter start fails after demoting Domain Controller

    Posted Mar 07, 2013 07:30 PM

    Glad that it also worked for you.

    Maybe you could mark the right answer, so for others it would be easier

    to find it if they were facing the same problem.

    Thanks.

    Cheers.



  • 6.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 12, 2013 08:12 PM

    Hello,

    I'm not a DBA, could you go into detail as to how to searched for and then modified these values?  Also, were you using MS SQL as your back-end?

    Thanks.



  • 7.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 12, 2013 08:32 PM

    MS SQL is the backend database, yes. If you have access to the DB server, the easiest way to view/modify the table is via MS SQL Server Management Studio.

    - Open MS SQL Server Management Studio and point it to the server/instance that houses your vcenter SSO database.

    - Expand the Databases object, then expand the vcenter SSO database (this could go by any name, if you're unsure you may have to check within multiple DBs)

    - Expand the tables object and look for the table with "IMS_CONFIG_VALUE" in the name

    - Right click on that table and choose edit top 200 rows

    - When those are displayed you can find and edit the fields referenced, which will change where the vmware SSO service looks to bind to AD. In my case I altered:

    ims.ldap-slots.0.primary-url

    ims.ldap-slots.0-global.primary-url

    ims.ldap-slots.0-global.secondary-url

    Probably a good idea to note the current values and double check the new values you enter. Changes made are effective immediately.



  • 8.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 14, 2013 02:09 PM

    This is crazy!  After decommissioning DC2.dc.local I'm having the exact same problem that I had all those months ago as shown by the logs:

    pi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
    Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
    Caused by: java.net.SocketException: Connection reset
    com.rsa.common.ConnectionException: Error connecting to the identity source
    Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269]
    Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
    Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
    Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
    Caused by: java.net.SocketException: Connection reset

    The solution is simple, change the static references to the new domain controller, P1.dc.local, but believe it or not I simply can't FIND what my SSO database is using.  I believe I'm using my Windows 2008 R2 SQL server, but when I Management Studio and browse to the RSA database --> tables it's blank.  Perhaps I used a local database instead, but I'm having trouble confiirming that also.  Is there a place in the logs which will tell me definititivaly what server is hosting the SSO database back-end?

    Many thanks for your help.



  • 9.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 15, 2013 01:42 PM

    I'm sure there are other ways, but the first thing that came to mind was to check the .log file from installation. Find the "vim-sso-msi.log" file and do a search for "JDBC_URL property", this will contain a string that will tell you the server name, port and db name that SSO was setup to use. That should point you in the right direction.

    Ben



  • 10.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 15, 2013 04:09 PM

    Thanks for the tip, I found it by using a slightly different method, I checked the database configuration information file located here:

    C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties

    bc Url
    db.url=jdbc:jtds:sqlserver://;serverName=;portNumber=63105;databaseName=RSA
    ## DB Username
    db.user=RSA_USER
    ## DB password
    db.pass=i.yl1hBdKI
    ## DB type
    db.type=mssql
    ## DB host
    db.host=VC51.DC.local

    More info on where to find these files can be found here:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033430

    So now I know (I believe) my SSO database runs off MYSQL on the vCenter itself.  I'm assuming in order to modify it I'm going to need to install the management studio on my vCenter.  What version of MySQL does the SSO installer install?  I normally run SQL 2008 R2 in my environment, will I be able to manage the database using that?  Also, does anyone know the credentials that are assigned by default?  Do I log in with SA?



  • 11.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 17, 2013 07:44 PM

    Not sure why you are talking about mysql?

    From what I can see you are using mssql so install Management Studio Express and open the database.

    You can use sa with MSSQL authentication or Windows Administrator when connecting via Windows authentication.

    Cheers.



  • 12.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 11, 2014 01:54 AM

    Thanks. Great help! I didn't demote a DC but had an issue with secure LDAP on one of the DC's.

    So temporary changed the entries in the DB until I figure out why it is not working.



  • 13.  RE: VCenter start fails after demoting Domain Controller

    Posted Mar 08, 2013 02:18 PM

    As far as I can tell the question has already been marked as answered! I don't see any other options than replying, hopefully anyone else having the same problem will see the fix you posted above. Thanks again!



  • 14.  RE: VCenter start fails after demoting Domain Controller

    Posted May 27, 2013 09:40 AM

    Also worth checking if you DC has an SSL cert that the DC cert has not expired.

    Particularly prudent if you do not have auto enroll switched on.

    you usually have an issue about binding but worth checking



  • 15.  RE: VCenter start fails after demoting Domain Controller

    Posted Mar 04, 2014 11:26 AM

    For vCenter 5.5

    Hi to inform you that i get those in error when i move to another domain a VC and change the IP in a test environement, SSO failed to connect to the vcenter because it keep trying to connect to the the old nameof the vcenter

    Error : ConnectComplete] Connect failed to <cs p:000000000a044270, TCP:MYVC.OLDdomaine:7444>; cnx: (null), error: class Vmacore::SystemException(Hôte inconnu)

    To solve that i put in the host file the old DNS mapped to the new IP

    Then vCenter restart

    And in the webClient, in administration delete the old domain and add the new one.



  • 16.  RE: VCenter start fails after demoting Domain Controller

    Posted Apr 11, 2014 06:02 PM

    Awesome article. I had the exact issue and this article resolved it. Thanks,



  • 17.  RE: VCenter start fails after demoting Domain Controller

    Posted Aug 04, 2014 04:04 PM

    *bump on this on*

    Anyone aware of where the targeted domain controller is registered in SSO 5.5?

    Trying to search through all files in Windows, but running into the dreaded, Powershell memory exceeded conundrum.