1 2 Previous Next 17 Replies Latest reply on May 12, 2020 12:39 PM by Vijay2027

    Rsyslog in vCenter 6.7U3 (Photon OS) stops working ~10min after starting

    lulu62 Novice

      Hello,

       

      We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and we noticed a gap of logs in our syslog server (kiwi) since then.

      I did a bit of troubleshooting but Rsyslog (the syslog client running on VCSA) is completely new to me.

       

      I use this command to restart Rsyslog:

       

      systemctl restart rsyslog

       

      Right after starting up Rsyslog, logs are being sent to our syslog server.

       

      ~10min later, no more logs are sent.

      The vCenter log file in our syslog server stops getting updated.
      I did a tcpdump in our vCenter and I see that the vCenter stops sending logs.
      Using UDP or TCP doesn't fix the issue.


      I looked for errors in various log files in the vCenter but can't find anything.

      This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after restarting Rsyslog:

       

      2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="http://www.rsyslog.com"] exiting on signal 15.

      2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]

      2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]

      2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start

       

      Rsyslog is still running based on this command

       

      systemctl status rsyslog.service

       

      ● rsyslog.service - System Logging Service

         Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)

         Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago

           Docs: man:rsyslogd(8)

                 http://www.rsyslog.com/doc/

      Main PID: 22235 (rsyslogd)

          Tasks: 12

         Memory: 5.7M

            CPU: 191ms

         CGroup: /system.slice/rsyslog.service

                 └─22235 /usr/sbin/rsyslogd -n

       

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main process exited, code=killed, status=9/KILL

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging Service.

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit entered failed state.

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed with result 'signal'.

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging Service...

      Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging Service.

      Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime  [v8.37.0 try http://www.rsyslog.com/e/2442 ]

      Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.37.0]

      Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start

       

       

      (real hostname has been replaced by vcenter.domain.local)

       

      I created a ticket at VMware support, but the agent wasn't able to find any errors as well and she suggested to take a backup of our vCenter and reinstall with a restore to get a fresh install of Photon OS since Rsyslog is integrated in Photon OS. I'm not going to do that now, maybe as a last troubleshooting step.

       

      In the meantime, do you guys have an idea? Wrong Rsyslog config?

       

      Thx for your help.

        1 2 Previous Next