I have a cluster of 8 ESXi machines where I wished to retain their log messages back for a month. The only way to do this is to write them to a remote syslog server. I chose to use rsyslog instead because it allows me to filter the incoming messages and write them to separate files for each ESXi host. The only downside to using rsyslog is that I found the documentation to be less than clear or complete. Anyone who simply gives you a link to the doc page at the rsyslog site isn't doing you any favors...
I'm no rsyslog expert. I just got something to work well enough that I figured others might want to see it and use it themselves. Between using google to see what other folks have done and tweaking on my own, I've got something that works. I don't understand all of the rsyslog syntax I'm using, but it works for me.
I built a VM using our company standard, RHEL 5, x86_64 bit. What I'm doing here should work OK for 32 bit RHEL, Cent O/S and the Debian/Ubuntu Linux varients with minor changes.
When you install RHEL, it gives you good ol' standard syslog. You need to shut it off and render it mute with chkconfig, and then using yum (so I'm lazy, mark of a good sysadmin...), install rsyslog and mark it active.
service syslog stop
chkconfig --level 2345 syslog off
yum -y install rsyslog.x86_64
chkconfig --level 2345 rsyslog on
You'll need to add a ModLoad statement to /etc/rsyslog.conf so it will be listening on UDP port 514, as well as some filtering statements to sort the incoming log messages on a per machine basis. I wanted something that would filter them based upon hostname, but I never figured that one out. Perhaps someone smarter than me can post a note on how to do that. I did get it to work by IP address. My rsyslog.conf file follows:
# cat /etc/rsyslog.conf
# Begin . Allow remote logging
$ModLoad imudp.so
$UDPServerRun 514
# End . Allow remote logging
# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
:fromhost-ip, isequal, "100.100.100.2" /var/log/vmware/esx81
& ~
:fromhost-ip, isequal, "100.100.100.4" /var/log/vmware/esx82
& ~
:fromhost-ip, isequal, "100.100.100.6" /var/log/vmware/esx83
& ~
:fromhost-ip, isequal, "100.100.100.8" /var/log/vmware/esx84
& ~
:fromhost-ip, isequal, "100.100.100.10" /var/log/vmware/esx85
& ~
:fromhost-ip, isequal, "100.100.100.12" /var/log/vmware/esx86
& ~
:fromhost-ip, isequal, "100.100.100.14" /var/log/vmware/esx87
& ~
:fromhost-ip, isequal, "100.100.100.16" /var/log/vmware/esx88
& ~
You then get this running with:
mkdir /var/log/vmware
service rsyslog start
You need to configure your ESXi machines to send logs to your rsyslog server. You do this by first selecting an ESXi machine, then going to:
Configuration Tab -> Software Advanced Settings -> Syslog -> Remote
and setting the Syslog.Remote.Hostname field.
This will start your ESXi machine sending log messages to your rsyslog server.
The next thing you will want to deal with is keeping /var/log/messages clean on your rsyslog server. The rsyslog server will write your ESXi log messages to both the file you specify in /etc/rsyslog.conf and to /var/log/messages. This will take up twice the disk space. What I've done to deal with this is to add a lengthy sed statement to the daily logrotate script:
#cat /etc/cron.daily/logrotate
#!/bin/sh
sed -i -e "/esx8/d" \
-e "/VMware/d" \
-e "/vmware/d" \
-e "/scripts/d" \
-e "/print_args/d" \
-e "/issue_cmd/d" \
-e "/hostCompatList/d" \
-e "/100\.100\.100/d" \
-e "/Vpxa/d" /var/log/messages
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
You manage your log rotations in /etc/logrotate.conf
