Hello,
We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and we noticed a gap of logs in our syslog server (kiwi) since then.
I did a bit of troubleshooting but Rsyslog (the syslog client running on VCSA) is completely new to me.
I use this command to restart Rsyslog:
systemctl restart rsyslog
Right after starting up Rsyslog, logs are being sent to our syslog server.
~10min later, no more logs are sent.
The vCenter log file in our syslog server stops getting updated.
I did a tcpdump in our vCenter and I see that the vCenter stops sending logs.
Using UDP or TCP doesn't fix the issue.
I looked for errors in various log files in the vCenter but can't find anything.
This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after restarting Rsyslog:
2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="http://www.rsyslog.com"] exiting on signal 15.
2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]
2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start
Rsyslog is still running based on this command
systemctl status rsyslog.service
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
Docs: man:rsyslogd(8)
Main PID: 22235 (rsyslogd)
Tasks: 12
Memory: 5.7M
CPU: 191ms
CGroup: /system.slice/rsyslog.service
└─22235 /usr/sbin/rsyslogd -n
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main process exited, code=killed, status=9/KILL
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging Service.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit entered failed state.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed with result 'signal'.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging Service...
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging Service.
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start
(real hostname has been replaced by vcenter.domain.local)
I created a ticket at VMware support, but the agent wasn't able to find any errors as well and she suggested to take a backup of our vCenter and reinstall with a restore to get a fresh install of Photon OS since Rsyslog is integrated in Photon OS. I'm not going to do that now, maybe as a last troubleshooting step.
In the meantime, do you guys have an idea? Wrong Rsyslog config?
Thx for your help.
I upgraded rsyslog and I haven't seen a failure going on 4 days on one VCSA. I upgraded my second VCSA this morning and it is still going as well. I'll keep everyone posted if either fails again. Disclaimer: I did not talk to VMWare support before doing this and do not know if it is officially supported, but I did let them know what I did and they didn't say anything about it. For anyone that wants to try it, from the vcsa shell: "tdnf upgrade rsyslog.x86_64".
I have nearly the same issue, the difference being that mine will sometimes last for 4 days. I can restart rsyslog and it starts working again. I have opened a case, SR9065686109, but no resolution yet. Problem began the day I updated to u3.
We also have this issue.
SR is 19066261109. Just opened it today.
Hopefully as more and more people open this, we should see some traction from vmware.
As a side note, we saw the same issue on a newly deployed vCenter server with no hosts attached.
Same problem here. Since upgrading VCSA to 6.7U3 we get 10-15 minutes worth of syslog before it stops without warning. Tried both TCP and UDP. Nothing interesting in the journal.
Opened SR19067744409.
I upgraded rsyslog and I haven't seen a failure going on 4 days on one VCSA. I upgraded my second VCSA this morning and it is still going as well. I'll keep everyone posted if either fails again. Disclaimer: I did not talk to VMWare support before doing this and do not know if it is officially supported, but I did let them know what I did and they didn't say anything about it. For anyone that wants to try it, from the vcsa shell: "tdnf upgrade rsyslog.x86_64".
I'm wondering if we're running into this: rsyslog failing with SEGV with dynafile + buffers · Issue #3772 · rsyslog/rsyslog · GitHub
Which version of rsyslogd did 6.7U2 have? U3 seems to have been released with a year-old version!
Anyone seen resolution on this? Our case is still open... fastie87 @jcm_g lulu62
Hey.
We've been working the following solution up through our vCenters:
Commands to run on each vcenter - in order
do you have any answer from vmware?
This is the response from VMware support I got at the beginning of the month.
Hello ...,
Greetings!
6.7 P01 release is suspected to be released within the next two months.
I checked and found that the next release (6.7P01) will include a newer rsyslog version which will fix the issue. Its version is rsyslog-8.1907.0-1.ph1.x86_64.rpm.
It is not supported to install a singular package on the VCSA appliance however, one of our customers just updated us that he installed the package and it worked.
Kind Regards,
...
For the time being, we manually updated rsyslog on our vCenters.
VMware does not support or recommend to update syslog rpm or any other rpm on vCSA.
During next patch update you might get the below error:
"Test transaction failed to update package
out=
error=error: Failed dependencies:"
For now restating rsyslogd service periodically using crontab will be temporary workaround.
AFAIK this will be fixed in next release. You may open a SR with GSS to validate the fix.
Vijay2027 How often are you restarting the service? Is there a chance of losing logs when the service is restarting?
Interesting thing my coworker found on this. We installed 6.7U3 a long while back and remote syslog worked just fine. It just stopped working one day. After some digging, we found on the day it stopped working, we had installed a security patch (VMSA-2019-0018 ), which just happens to be part of 6.7U3a. https://www.vmware.com/security/advisories/VMSA-2019-0018.html
We're going to open a case with VMware about it so they have the information, but will likely use crontab to restart the service periodically.
Every 8 hours.
Is there a chance of losing logs when the service is restarting?
I will have to verify this. Will update thread once I have valid inputs. Thanks.
Upgrading the package individually is not supported.
For a work around use the following:
1. Create a file “syslog_restart.cron” under ‘/etc/cron.d’ folder
2. Edit the “syslog_restart.cron” and add below content:
0 2 * * * root /usr/bin/systemctl restart syslog
3. Save the file
0 2 * * * : This means the syslog service will be restarted everyday at 2AM.
So this is a customizable parameter.
You can set these values to the time when you want to restart the rsyslog service.
Confirming that this is still a problem in VCSA 6.7U3f that was released now in April. Stops working afte ~10 minutes.
Lars