Upgraded from 9.1.0.1 to 9.5 on three out of four cell servers. The fourth cell post upgrade errors on service vmware-vcd start with the following errors.
Cell.log
- Bootstrap application: start Fatal exception caught during application bootstrap. Halting application. Message: '/opt/vmware/vcloud-director/etc/global.properties'
Cell-runtime.log
- 2019-01-10 17:05:25,937 | WARN | Bootstrap Application | CommonsConfigurationFactory | Server configuration file not found: /opt/vmware/vcloud-director/etc/global.properties |
org.apache.commons.configuration.ConfigurationException: Unable to load the configuration from the URL file:/opt/vmware/vcloud-director/etc/global.properties
However i can "cat /opt/vmware/vcloud-director/etc/global.properties" and comparing it to other cells seems correct. See output below.
# Database connection settings
database.jdbcUrl = jdbc:jtds:sqlserver://redacted:1433/vcloud;socketTimeout=90
database.username = redacted
database.password = redacted
# Product display name
product.display_name = VMware vCloud Director
# Maximum number blocking tasks (per cell) to be resumed when expired.
# This property is taken into account only when blocking task's default timeout action is "Resume".
# Uncomment this line and set the desired value. Otherwise, the default value (1000) will be used.
# blockingTasks.timeoutResumeRate = 1000
product.version = 9.5.0.11038232
product.build_date = 2018-11-27T21:25:46-0800
vcloud.cell.ip.primary = redacted
consoleproxy.host.https = redacted
vcloud.ssl.password = redacted
vcloud.ssl.key.password = redacted
vcloud.ssl.truststore.password = redacted
consoleproxy.keystore.password = redacted
consoleproxy.keystore.path = /opt/vmware/vcloud-director/etc/proxycertificates
audit.syslog.host =
audit.syslog.port = 514
system.info = aDYYIDRwlRaU1fLfnqgvgTMzp9YgEesQDALS53T73EM=
system.version = 1
vcloud.cell.uuid = redacted
vcloud.cell.ips = redacted
user.keystore.path = /opt/vmware/vcloud-director/bin/../jre/bin/certificates.ks
user.keystore.password redacted
Could be permissions on the file - did you do a comparison of the chmod permissions between one of the working cells and the inoperable cell? Your vcloud user should have rw permissions along with vcloud:vcloud as the user/group
Could be permissions on the file - did you do a comparison of the chmod permissions between one of the working cells and the inoperable cell? Your vcloud user should have rw permissions along with vcloud:vcloud as the user/group
Thank you for the quick response. The permissions were sit to root:root. I resolved this and the startup of the service did not immediately stop, instead it progressed until it just sits at the following in the cloud.log
Application startup begins: Subsystem 'com.vmware.vcloud.common.core' at 1/11/19 12:54 PM
Application Initialization: 'com.vmware.vcloud.common.core' 4% complete. Subsystem 'com.vmware.vcloud.vcell-runtime-impl' started
Application Initialization: 'com.vmware.vcloud.cloud-proxy-server' 14% complete. Subsystem 'com.vmware.vcloud.vcell-runtime-impl' started
Application startup event: Subsystem 'com.vmware.vcloud.consoleproxy' startup initiated.
Successfully connected to database: jdbc:jtds:sqlserver://redacted:1433/vcloud;socketTimeout=90
Successfully initialized system cryptography
Successfully configured Console Proxy SSL Connector from certificate store: /opt/vmware/vcloud-director/etc/proxycertificates
Application Initialization: 'com.vmware.vcloud.common.core' 8% complete. Subsystem 'com.vmware.vcloud.common-cell-impl' started
Application Initialization: 'com.vmware.vcloud.cloud-proxy-server' 28% complete. Subsystem 'com.vmware.vcloud.common-cell-impl' started
Application Initialization: 'com.vmware.vcloud.networking-server' 20% complete. Subsystem 'com.vmware.vcloud.common-cell-impl' started
Application Initialization: 'com.vmware.vcloud.common.core' 12% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application Initialization: 'com.vmware.vcloud.cloud-proxy-server' 42% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application Initialization: 'com.vmware.vcloud.networking-server' 40% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application shutdown event: Beginning
^C
Was that the last message in the cell log?
Yes it was it kind of just stopped mid statement.
It also appears that the cell is appearing healthy and operational inside VCD > Manage & Monitor > Cloud Resources > Cloud Cells. But the /cloud website for that site doesn't work and the vmware-vcd service is definitely not happy.
Might it be best to just rebuild this cell server and use the response file to join it back with the healthy cell?
Some possible interesting entries from other logs.
> /opt/vmware/vcloud-director/logs/vmware-vcd-watchdog.log.last <==
19-01-11 13:55:12 | INFO | Started vmware-vcd-cell (pid=15241)
2019-01-11 13:55:12 | WARN | wget failed, return code: 4, will retry in 30 seconds
....
2019-01-11 13:57:42 | ALERT | vmware-vcd-cell is dead but /var/run/vmware-vcd-cell.pid exist s, attempting to restart it
> /opt/vmware/vcloud-director/logs/cell-runtime.log <==
19-01-11 14:33:25,640 | FATAL | Console Proxy Application | AbstractCellApplication | Shutting down cell due to exception thrown while starting 'com.vmware.consoleproxy.async.server.ConsoleProxyCellApplication' |
201
> /opt/vmware/vcloud-director/logs/cell-runtime.log <==
19-01-11 14:33:25,639 | ERROR | Console Proxy Application | AbstractCellApplication | Application startup event: Unable to start service: 'com.vmware.vcloud.consoleproxy' |
java.lang.IllegalStateException: Unable to bind to 10.42.41.68:443. Permission denied. Is this address available on this machine?
###############I have no idea what that IP is to as its not configured on any of the Cells, doesnt ping and is not in DNS##################
> /opt/vmware/vcloud-director/logs/cell-runtime.log <==
2019-01-11 14:33:26,700 | WARN | Spring Context: com.vmware.vcloud.cloud-proxy-services | DirectlyConfigurableXmlApplicationContext | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistenceConfigurationData' defined in URL [jar:file:/opt/vmware/vcloud-director/system/com/vmware/vcloud/cloud-proxy-services/1.0.0/cloud-proxy-services-1.0.0.jar!/META-INF/spring/cloud-proxy-services-persistence.xml]: Cannot resolve reference to bean 'configurationService' while setting bean property 'configurationService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationService': Invocation of init method failed; nested exception is java.lang.NullPointerException |
2019-01-11 14:33:26,700 | FATAL | Spring Context: com.vmware.vcloud.cloud-proxy-services | OsgiBundleSpringContextFactory | Error in module: "com.vmware.vcloud.cloud-proxy-services"; Bean: persistenceConfigurationData, Message: Error creating bean with name 'persistenceConfigurationData' defined in URL [jar:file:/opt/vmware/vcloud-director/system/com/vmware/vcloud/cloud-proxy-services/1.0.0/cloud-proxy-services-1.0.0.jar!/META-INF/spring/cloud-proxy-services-persistence.xml]: Cannot resolve reference to bean 'configurationService' while setting bean property 'configurationService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationService': Invocation of init method failed; nested exception is java.lang.NullPointerException |
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistenceConfigurationData' defined in URL [jar:file:/opt/vmware/vcloud-director/system/com/vmware/vcloud/cloud-proxy-services/1.0.0/cloud-proxy-services-1.0.0.jar!/META-INF/spring/cloud-proxy-services-persistence.xml]: Cannot resolve reference to bean 'configurationService' while setting bean property 'configurationService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationService': Invocation of init method failed; nested exception is java.lang.NullPointerException
java.lang.IllegalStateException: Unable to bind to 10.42.41.68:443. Permission denied. Is this address available on this machine?
This error tells me you have something awry with the cell and console IP address. Do you know how it was configured? Check your ifconfig and find out how you guys are configuring console/proxy access - your healthy cells might be good guiding lights.