All product logs produced by VMware vCloud Director are stored on a cell's local disk [1]. The vCloud Director Security Hardening Guide mentions centrally logging everything for additional securitiy, but doesn't go into any detail on how to achieve this.
[1] The exception are audit event logs, which can be transmitted to a central syslog server.
The easiest way to do it is to modify the Log4j configuration that the product uses and add an additional appender to the loggers. You'll need to know the IP address or FQDN of the syslog server as well as what UDP port it is listening on. You'll also want to think about what level of logging information you want transmitted -- only errors & warnings, information, or debug level. The more fine-grained you go, the more data will be sent.
1. You'll need a version of syslog that supports listening for remote connections. Modern versions of syslog as well as syslog-ng support this. There are other 3rd party syslog-compatible products as well. A quick Web search will tell you how to enable this for your particular log server and OS. Tip: for syslog the -r argument will enable it.
2. You'll need to ensure that the appropriate firewall ports are open on both the VCD host for outbound UDP traffic as well as on the syslog host for inbound UDP access (or you'll need need to set up the syslog server in a passive fashion).
The steps below demonstrate how to add a new syslog appender that will transmit log messages that are INFO-level or above.
1. Log into your VCD cell using the console or SSH
2. Change directory (cd) to /opt/vmware/vcloud-director/etc
3. Make a backup of the logging configuration: cp log4j.properties log4j.properties.default
4. Edit log4j.properties in your preferred text editor and add the following lines:
You'll also want to modify line 2 of the file and append the name of the new syslog appender, as follows:
5. Save the file and restart the cell: service vmware-vcd restart
If you consult your central syslog server you should now see logs from the cell appearing. You'll need to repeat the procedure for each cell in your VCD cluster.
IHAC that requires Multiple syslog destinations, One for trouble shooting and the second for security audits, I'm not sure how to add multiple syslog appenders.
This allows for 1 destination:
log4j.appender.vcloud.system.syslog.syslogHost=remoteSyslogHost.example.com
How can I add another?? for example
log4j.appender.vcloud.system.syslog.syslogHost2=secondRemoteSyslogHost.example.com
I performed these exact steps, and while I do have logging now appearing on my central syslog-ng host, I am also getting a lot of garbage now being logged direclty into /opt/vmware/cloud-director/etc
- there are some "flat" .log files....
1317693160771.log 1317693523151.log 1317693885433.log 1317694247567.log
(these 4 were created in about 5 minutes - they continue to be created)
There are directories as follows:
org.eclipse.core.runtime
org.eclipse.equinox.app
org.eclipse.osgi
| `-- bundles
| |-- 12
| | `-- data
| | `-- config
| | `-- org
| | |-- apache
| | | `-- servicemix
| | | |-- features.config
| | | |-- kernel
| | | | `-- filemonitor
| | | | `-- FileMonitor
| | | | `-- factory.config
| | | |-- management.config
| | | |-- shell.config
| | | `-- transaction.config
| | `-- ops4j
| | `-- pax
| | `-- url
| | `-- mvn.config
| |-- 136
| | `-- 1
| |-- 14
| | `-- data
| | `-- P36_FeaturesServiceState.ser
| |-- 146
| | `-- 1
| |-- 147
| | `-- 1
| |-- 148
| | `-- 1
| |-- 149
| | `-- 1
| |-- 152
| | `-- 1
| |-- 81
| | `-- 1
| `-- 82
| `-- 1
What's causing this?
The edits aren't that eath-shaking - all I had to do was sub in the IP of my syslog host and restart the service......
diff log4j.properties.default log4j.properties
2c2
< log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info
---
> log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog
48a49,58
>
> # Modifications for remote Syslog
> log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
> log4j.appender.vcloud.system.syslog.syslogHost=172.22.22.41
> # Logs got to port 514 unless you specify a port, as in the disable example below.
> # log4j.appender.vcloud.system.syslog.syslogHost=172.22.22.41:5555
> log4j.appender.vcloud.system.syslog.facility=LOCAL1
> log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.CustomPatternLayout
> log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
> log4j.appender.vcloud.system.syslog.threshold=INFO
Welp, this seems to have permanently broken my cell. Restoring the original configuration file and restarting the service does NOT stop this new logging to the local directory.
153 new log files generated overnight, deleting the created directories is futile as they return almost immediately.
If I could make a suggestion to the author (and to the VMware KB folks that plagiarized this page into a KB article) consider being explicit about the required edits when you suggest a group of changes. I'd like to believe that I made the (only) required change - but clearly something has been borked on my box.....
The org.eclipse.* directories are always generated in the etc directory, regardless of whether or not you've modified your logging configuration. The org.eclipse.osgi directory contains a cache of the modules that are loaded in to the vCD runtime. You should not alter or remove those directories while the product is running.
The *.log files that are generated there usually indicate a cell lifecycle error (i.e. a NullPointerException while attempting to shut down the cell). Their location is unrelated to your logging configuration changes (the lowest level of the runtime will write to that directory because it's independent of the product's logging), but if your edits had any errors a log file may be produced there.
I'd suggest shutting down the cell (service vmware-vcd stop), verify it's terminated, and then restarting it while monitoring the $VCLOUD_HOME/logs/vcloud-container-info.log file to see if there are errors.
You can add additional syslog appenders, but note that it's going to add even more overhead to every logging call the product makes. I'd recommend using a single syslog host and splitting out logs there (e.g. one with more details for troubleshooting and one with less details for audits).
If you really want you can in fact add another appender, but you'll want to familiarize yourself with Log4j first because the example you posted is incorrect: http://logging.apache.org/log4j/1.2/manual.html
Thanks Kyle. But I am puzzled why I didn't see any logging here prior.
I will look into what's generating the errors, because clearly there's lots....
This is one of 2 cells in the cluster, and I have verified that the other does not show anything (in ~/etc) at present. Just a little gun-shy about editing another host and having it start to behave in the same fashion