kern messages and syslogd

[rhchodur]

We are trying to utilize a 3rd party monitor system with our ESX 3.0.1 servers. The monitor system requires syslogs to be forwarded to a central server which are then parsed for watches and alerts.

We recently completely patched some of our 3.0.1 servers with all the security and critical updates. After patching when we perform a reboot we begin to receive all the kern messages we expect. But once the system is rebooted and running we do not. Doing testing and looking at the other non-patched systems it appears that something is intercepting the kern messages and they never get forwarded by syslogd to the remote system on any of the ESX boxes, only entered into the /var/log/messages file.

When testing I setup a syslogd on my XP workstation and used logger to send test kern messages. Anything .err is passed and any kern message less then that does not. I have verified my syslog.conf file and even tried variations. I do not believe it is klogd, but something vmware as like mentioned on reboots it suddenly works for a brief moment.

What piece to the puzzle am I missing or forgetting? Thanks for any help.

[Texiwill]

Hello,

Your /etc/syslog.conf needs to be updated to send everything to the remote host. Add to the top or bottom of the file:

\*.* @hostname

The documentation is in 'man syslog.conf'

That should solve your problem. It is possible the patches changed the settings in syslog.conf.

Best regards,

Edward

[rhchodur]

Thanks for the info, I did some further testing and when I add that to syslog.conf you do get all logger test messages to pass through. If I only have:

kern.* @hostname

and do

logger -p kern.info TEST

I do not get the test log.

What I think I am seeing is that these are passed as user messages and not kernel messages, even though I am specifying kern. I send this in my syslogd running on my PC. Weird.

So does logger not allow you to actually send kernel log messages? Maybe not really a ESX question unless there is something different in ESX related to all this over standard Unix/Linux.

[MikeAvery]

Check your firewall settings to be sure that you are allowing syslog traffic through.

[Texiwill]

Hello,

Syslog uses UDP port 514 <== Standard

Reliable Syslog Service uses UDP/TCP ports 601.

You need to enable this for outgoing. You would use:

esxcfg-firewall -o 514,udp,out,SYSLOG

Best regards,

Edward

[rreynol]

We have setup a separate loghost running linux and then parse the logs it receives from all the ESX servers with an open source log checker. The log checker sends email every hour that contains log entries that we have chosen not to ignore. Here is what our /etc/syslog.conf looks like on a VI 3.0.1 server. The 123.45.67.89 address is where you would put your loghost address. The cut and paste may jam the lines together in the example below but you will still be able to get an idea.

\# Log all kernel messages to the console.

\# Logging much else clutters up the screen.

#kern.* /dev/console

\# Log anything (except mail) of level info or higher.

\# Don't log private authentication, cron, or vmkernel messages!

*.info;mail.none;authpriv.none;cron.none;local6.none;local5.none /var/log/messages

*.info;mail.none;authpriv.none;cron.none;local6.none;local5.none @123.45.67.89

\# The authpriv file has restricted access.

authpriv.* /var/log/secure

authpriv.* @123.45.67.89

\# Log all the mail messages in one place.

mail.* /var/log/maillog

mail.* @123.45.67.89

\# Log cron stuff

cron.* /var/log/cron

cron.* @123.45.67.89

\# Everybody gets emergency messages

*.emerg *

*.emerg *

\# Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

uucp,news.crit @123.45.67.89

\# Save boot messages also to boot.log

local7.* /var/log/boot.log

local7.* @123.45.67.89

#send all local6.info messages to special summary log only.

local6.info;local6.!notice /var/log/vmksummary

local6.info;local6.!notice @123.45.67.89

#send all local6.warning messages to warnings logs.

local6.warning /var/log/vmkwarning

local6.warning @123.45.67.89

#send all local6.notice and higher messages to vmkernel log.

local6.notice /var/log/vmkernel

local6.notice @123.45.67.89

#send all userworld proxy messages to proxy log

local5.* /var/log/vmkproxy

local5.* @123.45.67.89

#send all storage monitor related messages to storageMonitor log

local4.* /var/log/storageMonitor

local4.* @123.45.67.89

[rhchodur]

The logs are going through and firewall set, just not kernel logs when sent using:

logger -p kern.info TEST

If I do the command:

logger -p daemon.info TEST

I do see this.

Syslogd shows the first as a User log and the second as a daemon on the remote system.

So in a sense they are all going through, but why does the first get translated to a user log?

[rreynol]

So I assume your syslog.conf file looks similar to ours. Sorry to ask the obvious but did you look through the man page on syslogd for clues on how loghosting works? Wish I could be more help.

[Texiwill]

Hello,

I imagine it is more an issue with 'logger' than syslog....

Do something that forces a normal kernel log message.... For example mount an iso using the loop back device.

mount -o loop Name.iso directoryName

Then umount the device. It will add a kernel log message without using logger.

Best regards,

Edward

[MikeAvery]

If logger does not submit to the facility that you choose, I suspect that VMware have modified the source of either syslogd or the logger utility to force the user facility to be used. This is likely to make try to ensure that support don't spin their wheels with potentially tainted syslog facilities. Just a guess.

[rhchodur]

Going to do some more testing with forced kernel messages and see what happens. Post results back. At least then it will point to logger as the issue and what we were doing in testing.

[rhchodur]

Generating kernel messages outside of logger works, does not explain why logger does not create a kernel message in testing when using -p option.

Once a get a Linux box up for testing I am going to see if logger on that will do the same thing, or if it is just ESX.

Thanks.

[rreynol]

I tried it on several of our linux servers and the output goes to /var/log/messages when you issue logger -p kern.info TEST.

If you change it to logger -p local7.info TEST then it goes to /var/log/boot.log

[rhchodur]

Is it actually marked as a kernel message? ESX does not show in the messages log what facility it came from, was not till I logged it to an external syslogd did I see the facility and then the fact that logger was passing them as user messages and not kernel during testing.

Not sure if there is a setting for syslogd to have it also show the facilities on ESX in the /var/log/messages?

[rreynol]

logger -p local6.warning TEST will write to /var/log/vmkwarning and /var/log/vmkernel

logger -p local6.notice TEST will write to /var/log/vmkernel only

Could it be that your issue with kernel.info is that anything that is *.info will go to /var/log/messages?

[MikeAvery]

You can also use the -t option to force the "appearance" of whichever facility you like regardless of the actual facility the message is routed through.

\[root@vmhost root]# logger -p local0.notice -t VMware.Admin foo && tail -1 /var/log/messages

Jun 15 11:53:38 vmhost VMware.Admin: foo

\[root@vmhost root]#

Optionally, to avoid syslogd handling and routing of the message, you can use the -f switch to write directly to a file or the -u option to submit it via socket to a remote syslog daemon.