VMware Cloud Community
ncekic
Contributor
Contributor

vco auth provider registration fails with db error

Today i tried to start my vRO 7.3.1. No login was possible because the node is not active. I have resetted the authentication provider settings:

/etc/init.d/vco-configurator stop

/etc/init.d/vco-server stop

/usr/lib/vco/tools/configuration-cli/bin/vro-configuration

/etc/init.d/vco-configurator start

/etc/init.d/vco-server start

Then i have been waiting for some minutes and logged into the vco-controlcenter -> Authentication Provider.

I have unregistered and registered again, applied the cert import and when i was prompted to type the vsphjere.local password i got the error:Exception occurred. Details: com.vmware.o11n.licensing.SerialNumException: Database error: impossible to add this license.

I retried reset the auth provider again, tried to login to the vco-contolcenter, but now i get:

HTTP Status 503 – Service Unavailable


Type Status Report

Message Current node is not ACTIVE! Server is initializing...

Description The server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.

In the server.log i have this output:

2018-05-22 08:58:04.575+0000 [pool-1-thread-1] INFO  {} [DatabaseVerifier] Could not establish a connection to the database.

2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN  {} [InfrastructureContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bea

n with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.

2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN  {} [ClassPathXmlApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error cre

ating bean with name 'vcoApplicationContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to

bean 'bootContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/

WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework

.beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLo

cator-context.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; ne

sted exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish

a connection to the database.

2018-05-22 08:58:04.575+0000 [pool-1-thread-1] ERROR {} [Bootstrap$InitVcoAppContextCommand] vRealize Orchestrator applicaiton context initialization failed! Retrying in 30s: Unable to initialize group definition. Group resource name [cl

asspath*:/META-INF/spring/singletonBeanFactoryLocator-context.xml], factory key [vcoApplicationContext]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcoApplicationContext' d

efined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'bootContext' while setting constructor

argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!

/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error cr

eating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Bean instantiation via con

structor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; nested exception is org.springframework.beans.fa

ctory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.

Can anyone help to solve this issue????

Reply
0 Kudos
17 Replies
ncekic
Contributor
Contributor

I have an internal postegre mysql and i´m confused with the error connection refused:

2018-05-22 11:06:57.409+0000 [pool-1-thread-1] ERROR {} [Driver] Connection error:

org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)

        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)

        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)

        at org.postgresql.Driver.makeConnection(Driver.java:431)

        at org.postgresql.Driver.connect(Driver.java:247)

        at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319)

        at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212)

        at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:736)

        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:668)

        at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483)

        at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)

        at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)

        at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)

        at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)

        at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy.afterPropertiesSet(LazyConnectionDataSourceProxy.java:162)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)

        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)

        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585)

        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)

        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)

        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)

        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)

        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)

        at com.vmware.o11n.infra.InfrastructureContext.<init>(InfrastructureContext.java:16)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)

        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)

        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)

        at org.springframework.context.access.ContextSingletonBeanFactoryLocator.initializeDefinition(ContextSingletonBeanFactoryLocator.java:143)

        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:383)

        at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:35)

        at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:26)

        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused (Connection refused)

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.postgresql.core.PGStream.<init>(PGStream.java:62)

        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)

        ... 91 more

Reply
0 Kudos
daphnissov
Immortal
Immortal

Is the database service started?

Reply
0 Kudos
ncekic
Contributor
Contributor

ohhhh /storage/db is on 100% 😞

Is it only possible to shutdown the appliance add a new disk ???

Reply
0 Kudos
daphnissov
Immortal
Immortal

First figure out where that space is coming from and why.

Reply
0 Kudos
ncekic
Contributor
Contributor

STATEMENT:  update vmo_workflowtokencontent set compressedXmlcontent=$1, workflowtokenid=$2, xmlcontent=$3 where id=$4

ERROR:  could not extend file "base/16385/16459": No space left on device

HINT:  Check free disk space.

STATEMENT:  insert into vmo_changelog (eLast login: Tue May 22 06:50:12 UTC 2018 on console

FATAL:  could not write lock file "postmaster.pid": No space left on device

Last login: Tue May 22 12:40:11 UTC 2018 on console

FATAL:  could not write lock file "postmaster.pid": No space left on device

Reply
0 Kudos
daphnissov
Immortal
Immortal

Ok, so same problem. What is consuming the space on that file system?

Reply
0 Kudos
ncekic
Contributor
Contributor

serverlog: 10Gb

Disk has max. 10Gb in vcenter.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Still not answering the question...

Reply
0 Kudos
ncekic
Contributor
Contributor

the vro is running out of space. we use a postegre sql for that. i do not know what you mean exactly???

Usage is for vRO.

Reply
0 Kudos
daphnissov
Immortal
Immortal

What is consuming the space on that file system? I understand this is vRO and I understand you're using the internal vPostgres. The question is why has this file system filled up, and without knowing where the usage is coming from you don't know how to begin.

Reply
0 Kudos
ncekic
Contributor
Contributor

that´s exactly what i want to figure out:-)

Reply
0 Kudos
daphnissov
Immortal
Immortal

Show output of the following commands from an SSH session on the vRO appliance:

du -sh /storage/log/*

du -sh /storage/log/vmware/vco/app-server/*

Reply
0 Kudos
ncekic
Contributor
Contributor

726M/storage/log/core
16K /storage/log/lost+found
3.7G/storage/log/vmware

the other command is showing the logs like server.log, warning.log etc. about 30-40 logs not bigger than 5MB/each.

/storage/db/pgdata:

4.0KPG_VERSION
1.2Gbase
472Kglobal
16M pg_clog
4.0Kpg_commit_ts
4.0Kpg_dynshmem
8.0Kpg_hba.conf
4.0Kpg_ident.conf
12K pg_logical
28K pg_multixact
4.0Kpg_notify
4.0Kpg_replslot
4.0Kpg_serial
4.0Kpg_snapshots
4.0Kpg_stat
4.0Kpg_stat_tmp
144Kpg_subtrans
4.0Kpg_tblspc
4.0Kpg_twophase
705Mpg_xlog
4.0Kpostgresql.auto.conf
24K postgresql.conf
4.0Kpostmaster.opts
8.0Kserver.crt
4.0Kserver.key
11M serverlog
Reply
0 Kudos
daphnissov
Immortal
Immortal

Please show output of du -sh /storage/log/vmware/vco/app-server/* anyway.

Reply
0 Kudos
daphnissov
Immortal
Immortal

3.0M    /storage/log/vmware/vco/app-server/vc-metrics.log

5.1M    /storage/log/vmware/vco/app-server/vc-metrics.log.1

5.1M    /storage/log/vmware/vco/app-server/vc-metrics.log.2

5.1M    /storage/log/vmware/vco/app-server/vc-metrics.log.3

5.1M    /storage/log/vmware/vco/app-server/vc-metrics.log.4

3.3G    /storage/log/vmware/vco/app-server/vco.hprof

4.7M    /storage/log/vmware/vco/app-server/warning.log

5.1M    /storage/log/vmware/vco/app-server/warning.log.1

5.1M    /storage/log/vmware/vco/app-server/warning.log.2

5.1M    /storage/log/vmware/vco/app-server/warning.log.3

5.1M    /storage/log/vmware/vco/app-server/warning.log.4

You have a 3.3 GB Java dump file sitting in this directory which can be deleted to reclaim all that space. This is why I asked for that output in the first place.

Reply
0 Kudos
ncekic
Contributor
Contributor

sorry for late reply and your help:-)

problem has been solved

Reply
0 Kudos