VMware Cloud Community
lulu62
Enthusiast
Enthusiast
Jump to solution

Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting

Hello,

We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and we noticed a gap of logs in our syslog server (kiwi) since then.

I did a bit of troubleshooting but Rsyslog (the syslog client running on VCSA) is completely new to me.

I use this command to restart Rsyslog:

systemctl restart rsyslog

Right after starting up Rsyslog, logs are being sent to our syslog server.

~10min later, no more logs are sent.

The vCenter log file in our syslog server stops getting updated.
I did a tcpdump in our vCenter and I see that the vCenter stops sending logs.
Using UDP or TCP doesn't fix the issue.


I looked for errors in various log files in the vCenter but can't find anything.

This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after restarting Rsyslog:

2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="http://www.rsyslog.com"] exiting on signal 15.

2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]

2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]

2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start

Rsyslog is still running based on this command

systemctl status rsyslog.service

● rsyslog.service - System Logging Service

   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)

   Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago

     Docs: man:rsyslogd(8)

           http://www.rsyslog.com/doc/

Main PID: 22235 (rsyslogd)

    Tasks: 12

   Memory: 5.7M

      CPU: 191ms

   CGroup: /system.slice/rsyslog.service

           └─22235 /usr/sbin/rsyslogd -n

Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main process exited, code=killed, status=9/KILL

Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging Service.

Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit entered failed state.

Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed with result 'signal'.

Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging Service...

Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging Service.

Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime  [v8.37.0 try http://www.rsyslog.com/e/2442 ]

Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.37.0]

Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start

(real hostname has been replaced by vcenter.domain.local)

I created a ticket at VMware support, but the agent wasn't able to find any errors as well and she suggested to take a backup of our vCenter and reinstall with a restore to get a fresh install of Photon OS since Rsyslog is integrated in Photon OS. I'm not going to do that now, maybe as a last troubleshooting step.

In the meantime, do you guys have an idea? Wrong Rsyslog config?

Thx for your help.

1 Solution

Accepted Solutions
jcm_g
Contributor
Contributor
Jump to solution

I upgraded rsyslog and I haven't seen a failure going on 4 days on one VCSA. I upgraded my second VCSA this morning and it is still going as well. I'll keep everyone posted if either fails again. Disclaimer: I did not talk to VMWare support before doing this and do not know if it is officially supported, but I did let them know what I did and they didn't say anything about it. For anyone that wants to try it, from the vcsa shell: "tdnf upgrade rsyslog.x86_64".

View solution in original post

17 Replies
jcm_g
Contributor
Contributor
Jump to solution

I have nearly the same issue, the difference being that mine will sometimes last for 4 days. I can restart rsyslog and it starts working again. I have opened a case, SR9065686109, but no resolution yet. Problem began the day I updated to u3.

Reply
0 Kudos
fastie87
Contributor
Contributor
Jump to solution

We also have this issue.

SR is 19066261109. Just opened it today.

Hopefully as more and more people open this, we should see some traction from vmware.

As a side note, we saw the same issue on a newly deployed vCenter server with no hosts attached.

Reply
0 Kudos
rm_bk
Enthusiast
Enthusiast
Jump to solution

Same problem here.  Since upgrading VCSA to 6.7U3 we get 10-15 minutes worth of syslog before it stops without warning.  Tried both TCP and UDP.  Nothing interesting in the journal.

Opened SR19067744409.

Reply
0 Kudos
jcm_g
Contributor
Contributor
Jump to solution

I upgraded rsyslog and I haven't seen a failure going on 4 days on one VCSA. I upgraded my second VCSA this morning and it is still going as well. I'll keep everyone posted if either fails again. Disclaimer: I did not talk to VMWare support before doing this and do not know if it is officially supported, but I did let them know what I did and they didn't say anything about it. For anyone that wants to try it, from the vcsa shell: "tdnf upgrade rsyslog.x86_64".

rm_bk
Enthusiast
Enthusiast
Jump to solution

I'm wondering if we're running into this: rsyslog failing with SEGV with dynafile + buffers · Issue #3772 · rsyslog/rsyslog · GitHub

Which version of rsyslogd did 6.7U2 have?  U3 seems to have been released with a year-old version!

Reply
0 Kudos
rm_bk
Enthusiast
Enthusiast
Jump to solution

Anyone seen resolution on this?  Our case is still open...  fastie87​ @jcm_g lulu62

Reply
0 Kudos
fastie87
Contributor
Contributor
Jump to solution

Hey.

We've been working the following solution up through our vCenters:

Commands to run on each vcenter - in order

  1. Snapshot VCSA
  2. Run # tdnf upgrade tdnf --refresh
    1. This will update/refresh/check the repository for up-to-date packages and show lines of "refreshing metadata". "Nothing to do" is expected.
  3. Run # tdnf upgrade rsyslog.x86_64
    1. This forces the update to the rsyslog package
  4. Run # systemctl status rsyslog.service
    1. Checks the service status
  5. Run # systemctl restart rsyslog.service
    1. Restarts the ryslog.service
  6. Run # systemctl status rsyslog.service
    1. Check status again
  7. Check syslog server for inbound messages
  8. After 24 hours - Delete snapshot
Reply
0 Kudos
srhh
Contributor
Contributor
Jump to solution

do you have any answer from vmware?

Reply
0 Kudos
lulu62
Enthusiast
Enthusiast
Jump to solution

This is the response from VMware support I got at the beginning of the month.

Hello ...,

Greetings!

6.7 P01 release is suspected to be released within the next two months.

I checked and found that the next release (6.7P01) will include a newer rsyslog version which will fix the issue. Its version is rsyslog-8.1907.0-1.ph1.x86_64.rpm.

It is not supported to install a singular package on the VCSA appliance however, one of our customers just updated us that he installed the package and it worked.

Kind Regards,

...

Reply
0 Kudos
lulu62
Enthusiast
Enthusiast
Jump to solution

For the time being, we manually updated rsyslog on our vCenters.

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

VMware does not support or recommend to update syslog rpm or any other rpm on  vCSA.

During next patch update you might get the below error:

"Test transaction failed to update package

out=

error=error: Failed dependencies:"

For now restating rsyslogd service periodically using crontab will be temporary workaround.

AFAIK this will be fixed in next release. You may open a SR with GSS to validate the fix.

dprows33
Contributor
Contributor
Jump to solution

Vijay2027 How often are you restarting the service?  Is there a chance of losing logs when the service is restarting?

David Prows Fort Wayne VMUG Leader
Reply
0 Kudos
dprows33
Contributor
Contributor
Jump to solution

Interesting thing my coworker found on this.  We installed 6.7U3 a long while back and remote syslog worked just fine.  It just stopped working one day.  After some digging, we found on the day it stopped working, we had installed a security patch (VMSA-2019-0018 ), which just happens to be part of 6.7U3a. https://www.vmware.com/security/advisories/VMSA-2019-0018.html

We're going to open a case with VMware about it so they have the information, but will likely use crontab to restart the service periodically.

David Prows Fort Wayne VMUG Leader
Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

Every 8 hours.

Is there a chance of losing logs when the service is restarting?

I will have to verify this. Will update thread once I have valid inputs. Thanks.

Reply
0 Kudos
jmarr132007
VMware Employee
VMware Employee
Jump to solution

Upgrading the package individually is not supported.

For a work around use the following:

1. Create a file “syslog_restart.cron” under ‘/etc/cron.d’ folder

2. Edit the “syslog_restart.cron” and add below content:

              0 2 * * * root /usr/bin/systemctl restart syslog

3. Save the file

0 2 * * * : This means the syslog service will be restarted everyday at 2AM.

So this is a customizable parameter.

You can set these values to the time when you want to restart the rsyslog service.

Reply
0 Kudos
larstr
Champion
Champion
Jump to solution

Confirming that this is still a problem in VCSA 6.7U3f that was released now in April. Stops working afte ~10 minutes.

Lars

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

KB VMware Knowledge Base  updated with the fix.

This is resolved in vSphere 6.7 U3g, Build 16046470.