1 2 Previous Next 16 Replies Latest reply on Aug 4, 2014 9:03 AM by mark.chuman

    VCenter start fails after demoting Domain Controller

    goppi Enthusiast

      Hi.

       

      Instead of hijacking other one's thread having the same problem I decided to start a new thread, so sorry for double posting.

       

      After promoting a new DC and demoting the old one we have the problem that VCenter 5.1a no longer starts.

       

      Taking a look at the imsTrace.log file of SSOServer we see the following line:

       

      Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection olddc.ourdomain.com:3268

       

      As we can see SSO still tries to contact the old dc now beeing a simple member server instead of the new one - the question is why?

       

      We checked health of AD by running DCDIAG and BPA both not seeing any problems. Additionally we checked DNS by hand to ensure that no SRV or any other records are present which reference the old DC. Also all other AD dependent services like Exchange, SharePoint, DFS run without issue and do not indicate any problems.

       

      We also found the krb5.conf file of SSO which still references the old DC by the following entry:

       

      [realms]
      OURDOMAIN.COM = {
          kdc = olddc.ourdomain.com
          default_domain = OURDOMAIN.COM
      }

       

      However changing that entry does not solve the problem because after a short period this entry is replaced automagically by ??? - that's the question.

       

      Mybe someon else has some ideas what to check next.

       

      Cheers.

        • 1. Re: VCenter start fails after demoting Domain Controller
          goppi Enthusiast

          Hi.

           

          Here is what I have found additionally:

           

          Taking a look at the SSO database there is a table named IMS_CONFIG_VALUES.

          There I found two entries (ims.ldap-slots.0.primary-url and ims.ldap-slots.0-global.primary-url) still referencing the old

          DC. So next I will try to change those values and see what happens next.

           

          Stay tuned.

          • 2. Re: VCenter start fails after demoting Domain Controller
            goppi Enthusiast

            Hi again.

             

            So I'm back on track now.

            After changing those two values in the SSO database and restarting SSO service the krb5.conf file gets changed automagically to the correct values and VCenter starts and operates correctly again.

             

            What still is misterious to me.

            Why on earth are those values getting hardocded duing installation.

            Kerberos also supports DNS-based configuration via SRV records which is in place for MS Active Directory by default.

             

            Cheers.

            • 3. Re: VCenter start fails after demoting Domain Controller
              BeenGD Novice

              I was having similar issues with vcenter after upgrading AD to server 2008 r2 and demoting our old server 2003 r2 DC's. Error's in the Application event logs indiciated:

              Event Type:    Error
              Event Source:    VMware VirtualCenter Server
              Event Category:    None
              Event ID:    1000
              Date:        3/6/2013
              Time:        9:54:10 AM
              User:        N/A
              Computer:    VCENTER SERVER
              Description:
              The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: The group account "DOMAIN\GROUP" could not be successfully resolved.  Check network connectivity to domain controllers and domain membership.  Users may not be able to log in until connectivity is restored..

               

              And in System event logs:

              Event Type:    Error
              Event Source:    Service Control Manager
              Event Category:    None
              Event ID:    7024
              Date:        3/6/2013
              Time:        9:58:37 AM
              User:        N/A
              Computer:    VCENTER SERVER
              Description:
              The VMware VirtualCenter Server service terminated with service-specific error 2 (0x2).

               

              I had already tried creating DNS aliases for the old DC, which did not seem to work. Changing the entries mentioned here in the SSO database corrected this issue. Thank you!!

              • 4. Re: VCenter start fails after demoting Domain Controller
                goppi Enthusiast

                Glad that it also worked for you.

                 

                Maybe you could mark the right answer, so for others it would be easier

                to find it if they were facing the same problem.

                 

                Thanks.

                 

                Cheers.

                • 5. Re: VCenter start fails after demoting Domain Controller
                  BeenGD Novice

                  As far as I can tell the question has already been marked as answered! I don't see any other options than replying, hopefully anyone else having the same problem will see the fix you posted above. Thanks again!

                  • 6. Re: VCenter start fails after demoting Domain Controller
                    dbutch1976 Hot Shot

                    Hello,

                     

                    I'm not a DBA, could you go into detail as to how to searched for and then modified these values?  Also, were you using MS SQL as your back-end?

                     

                    Thanks.

                    • 7. Re: VCenter start fails after demoting Domain Controller
                      BeenGD Novice

                      MS SQL is the backend database, yes. If you have access to the DB server, the easiest way to view/modify the table is via MS SQL Server Management Studio.

                      - Open MS SQL Server Management Studio and point it to the server/instance that houses your vcenter SSO database.

                      - Expand the Databases object, then expand the vcenter SSO database (this could go by any name, if you're unsure you may have to check within multiple DBs)

                      - Expand the tables object and look for the table with "IMS_CONFIG_VALUE" in the name

                      - Right click on that table and choose edit top 200 rows

                      - When those are displayed you can find and edit the fields referenced, which will change where the vmware SSO service looks to bind to AD. In my case I altered:

                      ims.ldap-slots.0.primary-url

                      ims.ldap-slots.0-global.primary-url

                      ims.ldap-slots.0-global.secondary-url

                       

                      Probably a good idea to note the current values and double check the new values you enter. Changes made are effective immediately.

                      • 8. Re: VCenter start fails after demoting Domain Controller
                        dbutch1976 Hot Shot

                        This is crazy!  After decommissioning DC2.dc.local I'm having the exact same problem that I had all those months ago as shown by the logs:

                         

                        pi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
                        Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
                        Caused by: java.net.SocketException: Connection reset
                        com.rsa.common.ConnectionException: Error connecting to the identity source
                        Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269]
                        Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldaps://DC2.DC.local:3269' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
                        Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection SASL bind failed: DC2.DC.local:3269
                        Caused by: javax.naming.CommunicationException: SASL bind failed: DC2.DC.local:3269 [Root exception is java.net.SocketException: Connection reset]
                        Caused by: java.net.SocketException: Connection reset

                         

                        The solution is simple, change the static references to the new domain controller, P1.dc.local, but believe it or not I simply can't FIND what my SSO database is using.  I believe I'm using my Windows 2008 R2 SQL server, but when I Management Studio and browse to the RSA database --> tables it's blank.  Perhaps I used a local database instead, but I'm having trouble confiirming that also.  Is there a place in the logs which will tell me definititivaly what server is hosting the SSO database back-end?

                         

                        Many thanks for your help.

                        • 9. Re: VCenter start fails after demoting Domain Controller
                          BeenGD Novice

                          I'm sure there are other ways, but the first thing that came to mind was to check the .log file from installation. Find the "vim-sso-msi.log" file and do a search for "JDBC_URL property", this will contain a string that will tell you the server name, port and db name that SSO was setup to use. That should point you in the right direction.

                           

                          Ben

                          • 10. Re: VCenter start fails after demoting Domain Controller
                            dbutch1976 Hot Shot

                            Thanks for the tip, I found it by using a slightly different method, I checked the database configuration information file located here:

                             

                             

                            C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties

                             

                            bc Url
                            db.url=jdbc:jtds:sqlserver://;serverName=;portNumber=63105;databaseName=RSA
                            ## DB Username
                            db.user=RSA_USER
                            ## DB password
                            db.pass=i.yl1hBdKI
                            ## DB type
                            db.type=mssql
                            ## DB host
                            db.host=VC51.DC.local

                             

                            More info on where to find these files can be found here:
                            http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033430

                             

                            So now I know (I believe) my SSO database runs off MYSQL on the vCenter itself.  I'm assuming in order to modify it I'm going to need to install the management studio on my vCenter.  What version of MySQL does the SSO installer install?  I normally run SQL 2008 R2 in my environment, will I be able to manage the database using that?  Also, does anyone know the credentials that are assigned by default?  Do I log in with SA?

                            • 11. Re: VCenter start fails after demoting Domain Controller
                              goppi Enthusiast

                              Not sure why you are talking about mysql?

                              From what I can see you are using mssql so install Management Studio Express and open the database.

                              You can use sa with MSSQL authentication or Windows Administrator when connecting via Windows authentication.

                               

                              Cheers.

                              • 12. Re: VCenter start fails after demoting Domain Controller
                                krismcewan Hot Shot

                                Also worth checking if you DC has an SSL cert that the DC cert has not expired.

                                 

                                Particularly prudent if you do not have auto enroll switched on.

                                 

                                you usually have an issue about binding but worth checking

                                • 13. Re: VCenter start fails after demoting Domain Controller
                                  felakkad Novice

                                  For vCenter 5.5

                                   

                                  Hi to inform you that i get those in error when i move to another domain a VC and change the IP in a test environement, SSO failed to connect to the vcenter because it keep trying to connect to the the old nameof the vcenter

                                   

                                  Error : ConnectComplete] Connect failed to <cs p:000000000a044270, TCP:MYVC.OLDdomaine:7444>; cnx: (null), error: class Vmacore::SystemException(Hôte inconnu)

                                   

                                  To solve that i put in the host file the old DNS mapped to the new IP

                                   

                                  Then vCenter restart

                                   

                                  And in the webClient, in administration delete the old domain and add the new one.

                                  • 14. Re: VCenter start fails after demoting Domain Controller
                                    ewanat Novice

                                    Thanks. Great help! I didn't demote a DC but had an issue with secure LDAP on one of the DC's.

                                    So temporary changed the entries in the DB until I figure out why it is not working.

                                    1 2 Previous Next