VMware vCSA 6.7 too much noisy ! Need Warning syslog level only!

[AntexMv]

Dear Community
Please advise

Regardless syslog level Warning - it sends tons of syslog messages including Info, Notes and even Debug level

Please advise how it is possible to limit this flood with Waring syslog messages only.
Regards, AntexMv

[scott28tt]

@AntexMv 

Moderator: Moved to vCenter Server Discussions

[vxprthu]

HI @AntexMv 

Can you please go to General tab and check the logging settings there?

Regards,

[AntexMv]

Dear megotloves

Thank you for your attention

I double check that setting and it is correct -what we need -Warning level

Please kindly checked attached photo

Any other suggestions ?

Regards, AntexMv

[AntexMv]

Dear megotloves

Again thank you for your attention

I double check that setting in General tab and it is correct -what we need -Warning level

I tried to extract and change  Host Profile in Policies and Profiles tab. I found there some string which mentions syslog logging severity level

Host Profile in Policies -> Host Profiles (need to extract from server/host or create new)->Configure-> Advanced Configuration Settings -> Host Profile Log Configuration -> Log Level WARN

Now start testing. Let's see.

Regards, AntexMv

[vxprthu]

HI @AntexMv 

With host profile only the hypervisors' configuration could be done, not the vcenter's.

If you can share some log entries, we could figure out what is the source and recommend solution to change.

Regards, 

[AntexMv]

Dear Enthusiast

Thank you again for your reply

"With host profile only the hypervisors' configuration could be done, not the vcenter's." - I agree

Today I tried to change Syslog Level everywhere I found it 🙂

And still some services (updatemgr and vpxd-svcs) send Info and even Debug messages

Kindly look at attached files

I did not find where to change Log Level for those services  (if those are correct source) even through CLI

Thank you in advance for your advice

Regards, 

[vxprthu]

HI @AntexMv 

To change updatemgr's log level, you should use the old flex gui. Then go to home  > administration > system configuration > services > update manager > manage.

You can check other services here, you may found some where you can change.

Also there is an option to change syslog settings in cli.

You can edit the (r)syslog.conf like this:

*.error;*.crit;*.alert @Syslogserversip:port;RSYSLOG_SyslogProtocol23Format

The restart the service:

systemctl restart rsyslog

Then you can test with the logger:

logger -p syslog.error "This should go through"
logger -p syslog.info "This should not go through"

And of course, please take backup/snapshot of the appliance and/or the files you changed.

Regards,

[AntexMv]

Dear megotloves

Thank you much for your support

I tried everything. I found INFO in  Flex, CLI and replace them for WARNING

I modified 3 files - rsyslog.conf ; rsyslog.conf.orig ; rsyslog.conf.rpmnew

You provide me with useful information - thank you much.

But seems we are missing something as I still receiving  INFO and DEBUG messages. 

And test shows that INFO goes through

Kindly look at attached files

May be you have other ideas

Regards, AntexMv

################################################################################
############################# VMware Rsyslog Configuration ####################
################################################################################
###### Module declarations ######
module( load="imtcp"
streamdriver.name="gtls"
streamdriver.mode="1"
streamdriver.authmode="anon"
gnutlsprioritystring="NONE:+AES-128-GCM:+AES-256-CBC:+AES-128-CBC:+ECDHE-RSA:+ECDHE-ECDSA:+RSA:+AEAD:+SHA384:+SHA256:+SHA1:+COMP-NULL:+VERS-TLS1.2:+SIGN-RSA-SHA224:+SIGN-RSA-SHA256:+SIGN-RSA-SHA384:+SIGN-RSA-SHA512:+SIGN-DSA-SHA224:+SIGN-DSA-SHA256:+SIGN-ECDSA-SHA224:+SIGN-ECDSA-SHA256:+SIGN-ECDSA-SHA384:+SIGN-ECDSA-SHA512:+CURVE-SECP256R1:+CURVE-SECP384R1:+CURVE-SECP521R1:+CTYPE-OPENPGP:+CTYPE-X509:-CAMELLIA-256-CBC:-CAMELLIA-192-CBC:-CAMELLIA-128-CBC:-CAMELLIA-256-GCM:-CAMELLIA-128-GCM"
)
input(type="imtcp" port="1514")
$ModLoad imuxsock.so

$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% %app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% %app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% %hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100 ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5% full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
*.warning;*.error;*.crit;*.alert @Syslogserversip:port;RSYSLOG_SyslogProtocol23Format

#
# cron log entries for GEN003160
#
cron.* -/var/log/cron

#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log

[vxprthu · Accepted solution]

HI,)

It's my bad, i wasn't clear. You need to edit /etc/vmware-syslog/syslog.conf instead of rsyslog.conf

Where i said @Syslogserversip:port; you have to update to yours (if it's not already there, but im sure its there.)

Probably you will see something like this in /etc/vmware-syslog/syslog.conf:

*.* 192.168.1.1:514;RSYSLOG_SyslogProtocol23Format

and that needs to be updated to this one:

*.warn;*.err;*.crit;*.alert 192.168.1.1:514;RSYSLOG_SyslogProtocol23Format

Then reboot rsyslog service.

And don't forget to change back the rsyslog files to their original one.

Cheers,

[AntexMv]

Dear megotloves

Seems it works now!!!

Thank you very much for your assistance!!!

Test 

logger -p syslog.error "This should go through"
logger -p syslog.info "This should not go through"

also clearly works!

This was nice lesson, indeed   🙂

If you don't mind, have a look at other post - may be we together  could solve the issue ?

Failed listing records from zone ...,, error 21 revmdnsd vcenter

Dear Community 

Please advise. vCSA 6.7 constantly sends Error 21  Failed listing records from zone vsphere.local.... error 21

DNS Server works and nslookup allows to resolve names. DNS configured for 2 zones. And allows to  resolve names. 

Did not find  any descriptions in vmware documentation or even Google. May be somebody encountered such errors and and could advice remediation procedure

Regards, AntesMv

https://communities.vmware.com/t5/vCenter-Server-Discussions/Failed-listing-records-from-zone-error-...

Regards, AntexMv

[AntexMv]

Dear Community 

Please advise. vCSA 6.7 constantly sends Error 21  Failed listing records from zone vsphere.local.... error 21

DNS Server works and nslookup allows to resolve names. DNS configured for 2 zones. And allows to  resolve names. 

Did not find  any descriptions in vmware documentation or even Google. May be somebody encountered such errors and and could advice remediation procedure

Regards, AntesMv

[AntexMv]

Dear Enthusiast

Long time since I we chat. Let me ask again your kind advise

Few days ago  I reinstalled vCSA. Version 6_7_0_48 Everything seems OK. But again I have issue with logging 

Seems everything is clear and algorithm straight forward. But every time I try to modify syslog.conf file, Syslog Server configuration disappearing from Syslog for vCenter Server Appliance Management menu. if only I change from *.* to anything (*.warn or *.error) for example - vCSA Manager loose Forwarding configuration. I tried different abbreviation or digits instead of name. I changed permissions and reloaded rsyslog or syslog daemons. Nothing help. Very strange. Version 6.7.0.48 does not accept any changes. Coud you please advise what it could be ?

Thank you much in advance, AntexMv

syslog.conf

*.* @192.168.X.Y:514;RSYSLOG_SyslogProtocol23Format
*.* @@192.168.X.V:514;RSYSLOG_SyslogProtocol23Format
*.* @192.168.X.Z:514;RSYSLOG_SyslogProtocol23Format

syslog_modified_full_name_levels_v_6_7_0_48000.conf

*.warning;*.error;*.crit;*.alert @192.168.X.Y:514;RSYSLOG_SyslogProtocol23Format
*.warning;*.error;*.crit;*.alert @192.168.X.V:514;RSYSLOG_SyslogProtocol23Format
*.warning;*.error;*.crit;*.alert @192.168.X.Z:514;RSYSLOG_SyslogProtocol23Format