1 2 Previous Next 17 Replies Latest reply on Jun 4, 2018 10:40 PM by ncekic

    vco auth provider registration fails with db error

    ncekic Novice

      Today i tried to start my vRO 7.3.1. No login was possible because the node is not active. I have resetted the authentication provider settings:

      /etc/init.d/vco-configurator stop

      /etc/init.d/vco-server stop

      /usr/lib/vco/tools/configuration-cli/bin/vro-configuration

      /etc/init.d/vco-configurator start

      /etc/init.d/vco-server start

       

      Then i have been waiting for some minutes and logged into the vco-controlcenter -> Authentication Provider.

      I have unregistered and registered again, applied the cert import and when i was prompted to type the vsphjere.local password i got the error:Exception occurred. Details: com.vmware.o11n.licensing.SerialNumException: Database error: impossible to add this license.

       

      I retried reset the auth provider again, tried to login to the vco-contolcenter, but now i get:

      HTTP Status 503 – Service Unavailable


      Type Status Report

      Message Current node is not ACTIVE! Server is initializing...

      Description The server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.

       

      In the server.log i have this output:

      2018-05-22 08:58:04.575+0000 [pool-1-thread-1] INFO  {} [DatabaseVerifier] Could not establish a connection to the database.

      2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN  {} [InfrastructureContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bea

      n with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.

      2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN  {} [ClassPathXmlApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error cre

      ating bean with name 'vcoApplicationContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to

      bean 'bootContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/

      WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework

      .beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLo

      cator-context.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; ne

      sted exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish

      a connection to the database.

      2018-05-22 08:58:04.575+0000 [pool-1-thread-1] ERROR {} [Bootstrap$InitVcoAppContextCommand] vRealize Orchestrator applicaiton context initialization failed! Retrying in 30s: Unable to initialize group definition. Group resource name [cl

      asspath*:/META-INF/spring/singletonBeanFactoryLocator-context.xml], factory key [vcoApplicationContext]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcoApplicationContext' d

      efined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'bootContext' while setting constructor

      argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!

      /META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error cr

      eating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Bean instantiation via con

      structor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; nested exception is org.springframework.beans.fa

      ctory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.

       

       

      Can anyone help to solve this issue????

        • 1. Re: vco auth provider registration fails with db error
          ncekic Novice

          I have an internal postegre mysql and i´m confused with the error connection refused:

           

          2018-05-22 11:06:57.409+0000 [pool-1-thread-1] ERROR {} [Driver] Connection error:

          org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

                  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)

                  at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)

                  at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)

                  at org.postgresql.Driver.makeConnection(Driver.java:431)

                  at org.postgresql.Driver.connect(Driver.java:247)

                  at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319)

                  at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212)

                  at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:736)

                  at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:668)

                  at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483)

                  at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)

                  at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)

                  at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)

                  at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)

                  at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy.afterPropertiesSet(LazyConnectionDataSourceProxy.java:162)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)

                  at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)

                  at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)

                  at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)

                  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585)

                  at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)

                  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

                  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)

                  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)

                  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)

                  at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

                  at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)

                  at com.vmware.o11n.infra.InfrastructureContext.<init>(InfrastructureContext.java:16)

                  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

                  at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)

                  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)

                  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

                  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

                  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

                  at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)

                  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

                  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

                  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

                  at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)

                  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

                  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)

                  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)

                  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)

                  at org.springframework.context.access.ContextSingletonBeanFactoryLocator.initializeDefinition(ContextSingletonBeanFactoryLocator.java:143)

                  at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:383)

                  at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:35)

                  at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:26)

                  at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                  at java.lang.Thread.run(Thread.java:748)

          Caused by: java.net.ConnectException: Connection refused (Connection refused)

                  at java.net.PlainSocketImpl.socketConnect(Native Method)

                  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

                  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

                  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

                  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

                  at java.net.Socket.connect(Socket.java:589)

                  at org.postgresql.core.PGStream.<init>(PGStream.java:62)

                  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)

                  ... 91 more

          • 2. Re: vco auth provider registration fails with db error
            daphnissov Champion
            Community WarriorsvExpert

            Is the database service started?

            • 3. Re: vco auth provider registration fails with db error
              ncekic Novice

              ohhhh /storage/db is on 100% :-(

               

              Is it only possible to shutdown the appliance add a new disk ???

              • 4. Re: vco auth provider registration fails with db error
                daphnissov Champion
                Community WarriorsvExpert

                First figure out where that space is coming from and why.

                • 5. Re: vco auth provider registration fails with db error
                  ncekic Novice

                  STATEMENT:  update vmo_workflowtokencontent set compressedXmlcontent=$1, workflowtokenid=$2, xmlcontent=$3 where id=$4

                  ERROR:  could not extend file "base/16385/16459": No space left on device

                  HINT:  Check free disk space.

                  STATEMENT:  insert into vmo_changelog (eLast login: Tue May 22 06:50:12 UTC 2018 on console

                  FATAL:  could not write lock file "postmaster.pid": No space left on device

                  Last login: Tue May 22 12:40:11 UTC 2018 on console

                  FATAL:  could not write lock file "postmaster.pid": No space left on device

                  • 6. Re: vco auth provider registration fails with db error
                    daphnissov Champion
                    vExpertCommunity Warriors

                    Ok, so same problem. What is consuming the space on that file system?

                    • 7. Re: vco auth provider registration fails with db error
                      ncekic Novice

                      serverlog: 10Gb

                      Disk has max. 10Gb in vcenter.

                      • 8. Re: vco auth provider registration fails with db error
                        daphnissov Champion
                        Community WarriorsvExpert

                        Still not answering the question...

                        • 9. Re: vco auth provider registration fails with db error
                          ncekic Novice

                          the vro is running out of space. we use a postegre sql for that. i do not know what you mean exactly???

                          Usage is for vRO.

                          • 10. Re: vco auth provider registration fails with db error
                            daphnissov Champion
                            Community WarriorsvExpert

                            What is consuming the space on that file system? I understand this is vRO and I understand you're using the internal vPostgres. The question is why has this file system filled up, and without knowing where the usage is coming from you don't know how to begin.

                            • 11. Re: vco auth provider registration fails with db error
                              ncekic Novice

                              that´s exactly what i want to figure out:-)

                              • 12. Re: vco auth provider registration fails with db error
                                daphnissov Champion
                                Community WarriorsvExpert

                                Show output of the following commands from an SSH session on the vRO appliance:

                                 

                                du -sh /storage/log/*

                                 

                                du -sh /storage/log/vmware/vco/app-server/*

                                • 13. Re: vco auth provider registration fails with db error
                                  ncekic Novice
                                  726M/storage/log/core
                                  16K /storage/log/lost+found
                                  3.7G/storage/log/vmware

                                   

                                   

                                  the other command is showing the logs like server.log, warning.log etc. about 30-40 logs not bigger than 5MB/each.

                                   

                                  /storage/db/pgdata:

                                  4.0KPG_VERSION
                                  1.2Gbase
                                  472Kglobal
                                  16M pg_clog
                                  4.0Kpg_commit_ts
                                  4.0Kpg_dynshmem
                                  8.0Kpg_hba.conf
                                  4.0Kpg_ident.conf
                                  12K pg_logical
                                  28K pg_multixact
                                  4.0Kpg_notify
                                  4.0Kpg_replslot
                                  4.0Kpg_serial
                                  4.0Kpg_snapshots
                                  4.0Kpg_stat
                                  4.0Kpg_stat_tmp
                                  144Kpg_subtrans
                                  4.0Kpg_tblspc
                                  4.0Kpg_twophase
                                  705Mpg_xlog
                                  4.0Kpostgresql.auto.conf
                                  24K postgresql.conf
                                  4.0Kpostmaster.opts
                                  8.0Kserver.crt
                                  4.0Kserver.key
                                  11M serverlog
                                  • 14. Re: vco auth provider registration fails with db error
                                    daphnissov Champion
                                    Community WarriorsvExpert

                                    Please show output of du -sh /storage/log/vmware/vco/app-server/* anyway.

                                    1 2 Previous Next