Enabling centralized logging of vCloud Director cell logs

Enabling centralized logging of vCloud Director cell logs

All product logs produced by VMware vCloud Director are stored on a  cell's local disk [1].  The vCloud Director Security Hardening Guide  mentions centrally logging everything for additional securitiy, but  doesn't go into any detail on how to achieve this.

[1] The exception are audit event logs, which can be transmitted to a central syslog server.

The easiest way to do it is to modify the Log4j configuration that the  product uses and add an additional appender to the loggers.  You'll need  to know the IP address or FQDN of the syslog server as well as what UDP  port it is listening on.  You'll also want to think about what level of  logging information you want transmitted -- only errors & warnings,  information, or debug level.  The more fine-grained you go, the more  data will be sent.


1. You'll need a version of syslog that supports listening for remote  connections.  Modern versions of syslog  as well as syslog-ng support  this.  There are other 3rd party syslog-compatible products as well.  A  quick Web search will tell you how to enable this for your particular  log server and OS.  Tip: for syslog the -r argument will enable it.
2. You'll need to ensure that the appropriate firewall ports are open on  both the VCD host for outbound UDP traffic as well as on the syslog  host for inbound UDP access (or you'll need need to set up the syslog  server in a passive fashion).

Steps to enabling centralized logging

The steps below demonstrate how to add a new syslog appender that will transmit log messages that are INFO-level or above.

1. Log into your VCD cell using the console or SSH
2. Change directory (cd) to /opt/vmware/vcloud-director/etc
3. Make a backup of the logging configuration: cp log4j.properties log4j.properties.default
4. Edit log4j.properties in your preferred text editor and add the following lines:

#Logs got to port 514 unless you specify a port, as in the disable example below.
log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n

You'll also want to modify line 2 of the file and append the name of the new syslog appender, as follows:

log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog

5. Save the file and restart the cell: service vmware-vcd restart

If you consult your central syslog server you should now see logs from  the cell appearing.  You'll need to repeat the procedure for each cell  in your VCD cluster.


  • Log4j's SyslogAppender only supports UDP.  In order to use TCP or  other more advanced syslog features you would need to modify the cell's  runtime configuration, which is not supported by VMware (so I will not  cover it here).

  • As noted in the hardening guide, the Jetty request log is configured  by Jetty programmatically so there is no way to centrally log the  access and error information using Log4j short of using a separate  log-collection utility.



IHAC that requires Multiple syslog destinations, One for trouble shooting and the second for security audits, I'm not sure how to add multiple syslog appenders.

This allows for 1 destination:


How can I add another??  for example


I performed these exact steps, and while I do have logging now appearing on my central syslog-ng host, I am also getting a lot of garbage now being logged direclty into /opt/vmware/cloud-director/etc

- there are some "flat" .log files....

1317693160771.log  1317693523151.log  1317693885433.log  1317694247567.log

(these 4 were created in about 5 minutes - they continue to be created)

There are directories as follows:



|   `-- bundles
|       |-- 12
|       |   `-- data
|       |       `-- config
|       |           `-- org
|       |               |-- apache
|       |               |   `-- servicemix
|       |               |       |-- features.config
|       |               |       |-- kernel
|       |               |       |   `-- filemonitor
|       |               |       |       `-- FileMonitor
|       |               |       |           `-- factory.config
|       |               |       |-- management.config
|       |               |       |-- shell.config
|       |               |       `-- transaction.config
|       |               `-- ops4j
|       |                   `-- pax
|       |                       `-- url
|       |                           `-- mvn.config
|       |-- 136
|       |   `-- 1
|       |-- 14
|       |   `-- data
|       |       `-- P36_FeaturesServiceState.ser
|       |-- 146
|       |   `-- 1
|       |-- 147
|       |   `-- 1
|       |-- 148
|       |   `-- 1
|       |-- 149
|       |   `-- 1
|       |-- 152
|       |   `-- 1
|       |-- 81
|       |   `-- 1
|       `-- 82
|           `-- 1

What's causing this?

The edits aren't that eath-shaking - all I had to do was sub in the IP of my syslog host and restart the service......

diff log4j.properties.default log4j.properties
< log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info
> log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog
> # Modifications for remote Syslog
> log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
> log4j.appender.vcloud.system.syslog.syslogHost=
> # Logs got to port 514 unless you specify a port, as in the disable example below.
> # log4j.appender.vcloud.system.syslog.syslogHost=
> log4j.appender.vcloud.system.syslog.facility=LOCAL1
> log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.CustomPatternLayout
> log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
> log4j.appender.vcloud.system.syslog.threshold=INFO

Welp, this seems to have permanently broken my cell.  Restoring the original configuration file and restarting the service does NOT stop this new logging to the local directory.

153 new log files generated overnight, deleting the created directories is futile as they return almost immediately.

If I could make a suggestion to the author (and to the VMware KB folks that plagiarized this page into a KB article) consider being explicit about the required edits when you suggest a group of changes.  I'd like to believe that I made the (only) required change - but clearly something has been borked on my box.....

The org.eclipse.* directories are always generated in the etc directory, regardless of whether or not you've modified your logging configuration. The org.eclipse.osgi directory contains a cache of the modules that are loaded in to the vCD runtime. You should not alter or remove those directories while the product is running.

The *.log files that are generated there usually indicate a cell lifecycle error (i.e. a NullPointerException while attempting to shut down the cell). Their location is unrelated to your logging configuration changes (the lowest level of the runtime will write to that directory because it's independent of the product's logging), but if your edits had any errors a log file may be produced there.

I'd suggest shutting down the cell (service vmware-vcd stop), verify it's terminated, and then restarting it while monitoring the $VCLOUD_HOME/logs/vcloud-container-info.log file to see if there are errors.

You can add additional syslog appenders, but note that it's going to add even more overhead to every logging call the product makes. I'd recommend using a single syslog host and splitting out logs there (e.g. one with more details for troubleshooting and one with less details for audits).

If you really want you can in fact add another appender, but you'll want to familiarize yourself with Log4j first because the example you posted is incorrect: http://logging.apache.org/log4j/1.2/manual.html

Thanks Kyle.  But I am puzzled why I didn't see any logging here prior.

I will look into what's generating the errors, because clearly there's lots....

This is one of 2 cells in the cluster, and I have verified that the other does not show anything (in ~/etc) at present.  Just a little gun-shy about editing another host and having it start to behave in the same fashion

Version history
Revision #:
1 of 1
Last update:
‎11-23-2010 11:36 AM
Updated by: