VMware Cloud Community
erasedhammer
Contributor
Contributor
‎07-02-2021 07:46 AM

vCenter upgrade from 6.7 to 7 too many logs

I just upgraded to vCenter 7.0.2.00200 and changed the log level to error, hoping to reduce the millions of logs being sent to my syslog server.

Unfortunately not much has changed, the logs I am getting are still at the info and debug level and I am receiving over 70,000 logs per hour.

Back on vcenter 6.7 my syslog server was only getting 1500 logs per hour.

Here is a screenshot, it is very obvious when I did the upgrade from 6.7 to 7.

vcenter7vs6-7logs.png

 

Why isn't the verbosity level affect remote syslog? Why is 7 generating so many more logs?

Preview file
13 KB
0 Kudos
16 Replies
Vijay2027
Expert
Expert
‎07-05-2021 08:28 PM

On the syslog server can you check which service from VC is logging more?

0 Kudos
erasedhammer
Contributor
Contributor
‎07-08-2021 02:24 PM

I am working on getting that, I don't have the logs separated into fields, so I cant filter on that. Just splitting out each service I see in the logs manually so far, they all seem pretty equal.

Will post when I finish the counts

0 Kudos
erasedhammer
Contributor
Contributor
‎07-08-2021 03:17 PM

There are so many services logging. I have no idea why vmware insists on logging so much crap.

But my problem isn't noisy services, its that I still see INFO and DEBUG logs when I have clearly set the global logging level to ERROR.

 

0 Kudos
erasedhammer
Contributor
Contributor
‎10-25-2021 05:51 PM

I have to say, vcenter 7 update 3 made the log situation worse.

Log level set to ERROR.

At first, I was receiving over a million logs per minute, then it went down to 70,000 logs per 10 seconds. And it appears to have stabilized at 100 to 1000 logs per 10 seconds.

Most of the logs coming in are tagged with verbose and info. I would say that it ignores the log level that I set, except I can set the level to trivia and get over 100,000 logs per 10 seconds.

I just gotta say, it is absolutely disgusting how much it logs.

0 Kudos
padulka
Hot Shot
Hot Shot
‎10-26-2021 07:32 AM

You can check if there are the logs describes in the following KB:

https://kb.vmware.com/s/article/80803

 

0 Kudos
erasedhammer
Contributor
Contributor
‎10-26-2021 01:07 PM

Doesn't look like it, among the millions of logs, Pod only shows up 60 times

0 Kudos
padulka
Hot Shot
Hot Shot
‎10-26-2021 01:17 PM

could you execute 'du'  command on /storage/log and share the output?

0 Kudos
erasedhammer
Contributor
Contributor
‎10-26-2021 01:23 PM

Does vcenter still log locally if the remote syslog host is specified?

 

du /storage/log
85500   /storage/log/vmware/vpostgres
16      /storage/log/vmware/dbcc
100     /storage/log/vmware/vmdir
4       /storage/log/vmware/decommission
45096   /storage/log/vmware/lookupsvc/tomcat
57976   /storage/log/vmware/lookupsvc
636     /storage/log/vmware/hvc
11916   /storage/log/vmware/certificatemanagement
20436   /storage/log/vmware/vtsdb
59096   /storage/log/vmware/applmgmt
20      /storage/log/vmware/vpxd/drmdump/domain-c24
24      /storage/log/vmware/vpxd/drmdump
164     /storage/log/vmware/vpxd/telemetry/stage
308     /storage/log/vmware/vpxd/telemetry/prod
476     /storage/log/vmware/vpxd/telemetry
4       /storage/log/vmware/vpxd/uptime/error
8       /storage/log/vmware/vpxd/uptime
212328  /storage/log/vmware/vpxd
8       /storage/log/vmware/vcha
35084   /storage/log/vmware/vapi/endpoint
35088   /storage/log/vmware/vapi
620     /storage/log/vmware/topologysvc
4       /storage/log/vmware/netdumper
4       /storage/log/vmware/cis-license/slms
24      /storage/log/vmware/cis-license/dlfs/VMware VirtualCenter Server/7.0/7.0.3.0
24      /storage/log/vmware/cis-license/dlfs/VMware VirtualCenter Server/7.0/7.0.5.0
52      /storage/log/vmware/cis-license/dlfs/VMware VirtualCenter Server/7.0
56      /storage/log/vmware/cis-license/dlfs/VMware VirtualCenter Server
180     /storage/log/vmware/cis-license/dlfs/VMware ESX Server/6.0/6.7.2.1
184     /storage/log/vmware/cis-license/dlfs/VMware ESX Server/6.0
208     /storage/log/vmware/cis-license/dlfs/VMware ESX Server/7.0/7.0.5.0
208     /storage/log/vmware/cis-license/dlfs/VMware ESX Server/7.0/7.0.3.1
420     /storage/log/vmware/cis-license/dlfs/VMware ESX Server/7.0
608     /storage/log/vmware/cis-license/dlfs/VMware ESX Server
108     /storage/log/vmware/cis-license/dlfs/VMware VSAN/7.0/7.0.3.0
112     /storage/log/vmware/cis-license/dlfs/VMware VSAN/7.0
116     /storage/log/vmware/cis-license/dlfs/VMware VSAN
20      /storage/log/vmware/cis-license/dlfs/VMware Tanzu/7.0/7.0.0.1
16      /storage/log/vmware/cis-license/dlfs/VMware Tanzu/7.0/7.0.0.0
40      /storage/log/vmware/cis-license/dlfs/VMware Tanzu/7.0
44      /storage/log/vmware/cis-license/dlfs/VMware Tanzu
828     /storage/log/vmware/cis-license/dlfs
11164   /storage/log/vmware/cis-license
104     /storage/log/vmware/capengine
804     /storage/log/vmware/perfcharts/tomcat
3980    /storage/log/vmware/perfcharts
4       /storage/log/vmware/vmcam
22936   /storage/log/vmware/wcp
68      /storage/log/vmware/vmon/vapi
19996   /storage/log/vmware/vmon
19000   /storage/log/vmware/sso/tomcat
201148  /storage/log/vmware/sso
88      /storage/log/vmware/vmware-updatemgr/vum-server/hostUpgrade
37496   /storage/log/vmware/vmware-updatemgr/vum-server
37692   /storage/log/vmware/vmware-updatemgr
100100  /storage/log/vmware/trustmanagement
18988   /storage/log/vmware/vstats
144     /storage/log/vmware/deploy
33392   /storage/log/vmware/analytics
5092    /storage/log/vmware/vmafdd
37728   /storage/log/vmware/vdtc
16      /storage/log/vmware/rbd
4       /storage/log/vmware/vsm/web
1252    /storage/log/vmware/vsm
2032    /storage/log/vmware/rsyslogd
4884    /storage/log/vmware/vmdird
5344    /storage/log/vmware/observability/vapi
165476  /storage/log/vmware/observability
8       /storage/log/vmware/vmcamd
4728    /storage/log/vmware/vmcad
39984   /storage/log/vmware/vmware-sps
8       /storage/log/vmware/vmafd
7436    /storage/log/vmware/cloudvm
7392    /storage/log/vmware/eam/web
108876  /storage/log/vmware/eam
53972   /storage/log/vmware/vpxd-svcs
10540   /storage/log/vmware/vsphere-ui/logs/access
113076  /storage/log/vmware/vsphere-ui/logs
113168  /storage/log/vmware/vsphere-ui
4       /storage/log/vmware/sca/work/Tomcat/localhost/ROOT
8       /storage/log/vmware/sca/work/Tomcat/localhost
12      /storage/log/vmware/sca/work/Tomcat
16      /storage/log/vmware/sca/work
30908   /storage/log/vmware/sca
2824    /storage/log/vmware/vlcm
176     /storage/log/vmware/certificateauthority
7984    /storage/log/vmware/pod
61796   /storage/log/vmware/content-library
21828   /storage/log/vmware/envoy
16      /storage/log/vmware/syslog
1372    /storage/log/vmware/applmgmt-audit
156     /storage/log/vmware/upgrade
760     /storage/log/vmware/rhttpproxy
1648    /storage/log/vmware/vsan-health
4       /storage/log/vmware/infraprofile/vapi
9592    /storage/log/vmware/infraprofile
4       /storage/log/vmware/vmware-imagebuilder
1959584 /storage/log/vmware
16      /storage/log/lost+found
4       /storage/log/remote
1959608 /storage/log

0 Kudos
padulka
Hot Shot
Hot Shot
‎10-26-2021 01:33 PM

You need to investigate inside:

1959584 /storage/log/vmware

If you need to solve asap you can expand size of partition but it's only a workaround.

 

0 Kudos
erasedhammer
Contributor
Contributor
‎10-26-2021 01:44 PM

The biggest offenders in /storage/log/vmware

213852 /storage/log/vmware/vpxd
201476 /storage/log/vmware/sso
165596 /storage/log/vmware/observability
113140 /storage/log/vmware/vsphere-ui/logs
113232 /storage/log/vmware/vsphere-ui
108876 /storage/log/vmware/eam
100216 /storage/log/vmware/trustmanagement

 

I am not running out of space for vcenter itself, I simply want to know why vcenter syslogs so much even when set to error. And potentially be able to quiet down the logs so they are actually usable.

0 Kudos
ivanerben
Enthusiast
Enthusiast
‎03-21-2022 02:46 AM

I see same behavior on one vcenter after upgrade. So question is, what is solution to this log flood problem?

0 Kudos
ivanerben
Enthusiast
Enthusiast
‎03-21-2022 03:33 AM

So I have asked our TAM and he pointed me to vcenter log level settings, which we had set to verbose instead of info.

Configure Logging Options (vmware.com)

Solved.

0 Kudos
panmarmi
Contributor
Contributor
‎10-13-2023 12:01 AM

Hello, 

this solution doesn't work for me. The number of logs after changing to verbose increased from 500 MB per day (info level) to 100 MB per hour. vCenter version 6.7 logged 10 MB per day at the info level.

0 Kudos
TekXtreme1
Contributor
Contributor
‎10-13-2023 03:15 AM

We are experiencing the same issue since upgrade. If we change the Global Logging settings to ‘Warning’, we are still being flooded with INFO messages from Local0.

Logged into VCentre Server and grep Local0 within /etc/vmware-syslog,  we can see that it's still configured?

If we check a 6.7 deployment compared to 7.0 there are files in /etc/vmware-syslog/ vmware-services-<Component>.conf (x40), these files do not exist in 6.7. Next step is try and understand these files and there configuration.



0 Kudos
michalso
Contributor
Contributor
‎11-29-2023 05:25 AM

Support doesn't help. this topic has been going on for several months.

Perhaps it would be a good idea to return to the configuration from version 6.7

For example: 

#$IncludeConfig /etc/vmware-syslog/vmware-services-*.conf

 

0 Kudos
panmarmi
Contributor
Contributor
‎11-30-2023 10:30 PM

disabling additional configuration files (similar to version 6.7) resulted in a reduction in the number of logs from 500MB to 3MB per day.

0 Kudos