Today i tried to start my vRO 7.3.1. No login was possible because the node is not active. I have resetted the authentication provider settings:
/etc/init.d/vco-configurator stop
/etc/init.d/vco-server stop
/usr/lib/vco/tools/configuration-cli/bin/vro-configuration
/etc/init.d/vco-configurator start
/etc/init.d/vco-server start
Then i have been waiting for some minutes and logged into the vco-controlcenter -> Authentication Provider.
I have unregistered and registered again, applied the cert import and when i was prompted to type the vsphjere.local password i got the error:Exception occurred. Details: com.vmware.o11n.licensing.SerialNumException: Database error: impossible to add this license.
I retried reset the auth provider again, tried to login to the vco-contolcenter, but now i get:
Type Status Report
Message Current node is not ACTIVE! Server is initializing...
Description The server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.
In the server.log i have this output:
2018-05-22 08:58:04.575+0000 [pool-1-thread-1] INFO {} [DatabaseVerifier] Could not establish a connection to the database.
2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN {} [InfrastructureContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bea
n with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.
2018-05-22 08:58:04.575+0000 [pool-1-thread-1] WARN {} [ClassPathXmlApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error cre
ating bean with name 'vcoApplicationContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to
bean 'bootContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/
WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework
.beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLo
cator-context.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; ne
sted exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish
a connection to the database.
2018-05-22 08:58:04.575+0000 [pool-1-thread-1] ERROR {} [Bootstrap$InitVcoAppContextCommand] vRealize Orchestrator applicaiton context initialization failed! Retrying in 30s: Unable to initialize group definition. Group resource name [cl
asspath*:/META-INF/spring/singletonBeanFactoryLocator-context.xml], factory key [vcoApplicationContext]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcoApplicationContext' d
efined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'bootContext' while setting constructor
argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!
/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Cannot resolve reference to bean 'infrastructureContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error cr
eating bean with name 'infrastructureContext' defined in URL [jar:file:/usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/o11n-services-bootstrap-7.3.1.jar!/META-INF/spring/singletonBeanFactoryLocator-context.xml]: Bean instantiation via con
structor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.o11n.infra.InfrastructureContext]: Constructor threw exception; nested exception is org.springframework.beans.fa
ctory.BeanCreationException: Error creating bean with name 'infrastructureContextConfig': Invocation of init method failed; nested exception is java.lang.RuntimeException: Could not establish a connection to the database.
Can anyone help to solve this issue????
I have an internal postegre mysql and i´m confused with the error connection refused:
2018-05-22 11:06:57.409+0000 [pool-1-thread-1] ERROR {} [Driver] Connection error:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:736)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:668)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy.afterPropertiesSet(LazyConnectionDataSourceProxy.java:162)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
at com.vmware.o11n.infra.InfrastructureContext.<init>(InfrastructureContext.java:16)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.springframework.context.access.ContextSingletonBeanFactoryLocator.initializeDefinition(ContextSingletonBeanFactoryLocator.java:143)
at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:383)
at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:35)
at com.vmware.o11n.service.spring.bootstrap.Bootstrap$InitVcoAppContextCommand.get(Bootstrap.java:26)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:62)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 91 more
Is the database service started?
ohhhh /storage/db is on 100% 😞
Is it only possible to shutdown the appliance add a new disk ???
First figure out where that space is coming from and why.
STATEMENT: update vmo_workflowtokencontent set compressedXmlcontent=$1, workflowtokenid=$2, xmlcontent=$3 where id=$4
ERROR: could not extend file "base/16385/16459": No space left on device
HINT: Check free disk space.
STATEMENT: insert into vmo_changelog (eLast login: Tue May 22 06:50:12 UTC 2018 on console
FATAL: could not write lock file "postmaster.pid": No space left on device
Last login: Tue May 22 12:40:11 UTC 2018 on console
FATAL: could not write lock file "postmaster.pid": No space left on device
Ok, so same problem. What is consuming the space on that file system?
serverlog: 10Gb
Disk has max. 10Gb in vcenter.
Still not answering the question...
the vro is running out of space. we use a postegre sql for that. i do not know what you mean exactly???
Usage is for vRO.
What is consuming the space on that file system? I understand this is vRO and I understand you're using the internal vPostgres. The question is why has this file system filled up, and without knowing where the usage is coming from you don't know how to begin.
that´s exactly what i want to figure out:-)
Show output of the following commands from an SSH session on the vRO appliance:
du -sh /storage/log/*
du -sh /storage/log/vmware/vco/app-server/*
726M | /storage/log/core |
16K | /storage/log/lost+found |
3.7G | /storage/log/vmware |
the other command is showing the logs like server.log, warning.log etc. about 30-40 logs not bigger than 5MB/each.
/storage/db/pgdata:
4.0K | PG_VERSION |
1.2G | base |
472K | global |
16M | pg_clog |
4.0K | pg_commit_ts |
4.0K | pg_dynshmem |
8.0K | pg_hba.conf |
4.0K | pg_ident.conf |
12K | pg_logical |
28K | pg_multixact |
4.0K | pg_notify |
4.0K | pg_replslot |
4.0K | pg_serial |
4.0K | pg_snapshots |
4.0K | pg_stat |
4.0K | pg_stat_tmp |
144K | pg_subtrans |
4.0K | pg_tblspc |
4.0K | pg_twophase |
705M | pg_xlog |
4.0K | postgresql.auto.conf |
24K | postgresql.conf |
4.0K | postmaster.opts |
8.0K | server.crt |
4.0K | server.key |
11M | serverlog |
Please show output of du -sh /storage/log/vmware/vco/app-server/* anyway.
3.0M /storage/log/vmware/vco/app-server/vc-metrics.log
5.1M /storage/log/vmware/vco/app-server/vc-metrics.log.1
5.1M /storage/log/vmware/vco/app-server/vc-metrics.log.2
5.1M /storage/log/vmware/vco/app-server/vc-metrics.log.3
5.1M /storage/log/vmware/vco/app-server/vc-metrics.log.4
3.3G /storage/log/vmware/vco/app-server/vco.hprof
4.7M /storage/log/vmware/vco/app-server/warning.log
5.1M /storage/log/vmware/vco/app-server/warning.log.1
5.1M /storage/log/vmware/vco/app-server/warning.log.2
5.1M /storage/log/vmware/vco/app-server/warning.log.3
5.1M /storage/log/vmware/vco/app-server/warning.log.4
You have a 3.3 GB Java dump file sitting in this directory which can be deleted to reclaim all that space. This is why I asked for that output in the first place.
sorry for late reply and your help:-)
problem has been solved