VMware Cloud Community
NeoProzett
Contributor
Contributor

Vcsa 6.7 b19832974 content-library logs extremely growing

Hi all

My friends have problems with vcsa 6.7 b19832974
After upgrading b19300125 to b19832974

(Clean install too)


/storage/log fills up very fast by growing file /storage/log/vmware/content-library/content-library-runtime.log.stdout

Parts of lines after service statup

Starting service process with pid: 90954.
2332 [main] WARN com.vmware.vsphere.vcde.logging.helper.LoggerHelper - Log4j is not configured correctly! Default configuration will be used!
2335 [main] DEBUG org.springframework.context.annotation.ConfigurationClassEnhancer - Successfully enhanced com.vmware.cl.config.ClVapiConfig; enhanced class name is: com.vmware.cl.config.ClVapiConfig$$EnhancerBySpringCGLIB$$f32309e1
2335 [main] DEBUG org.springframework.context.annotation.ConfigurationClassPostProcessor - Replacing bean definition 'clVapiConfig' existing class 'com.vmware.cl.config.ClVapiConfig' with enhanced class 'com.vmware.cl.config.ClVapiConfig$$EnhancerBySpringCGLIB$$f32309e1'
...
2422 [main] DEBUG com.vmware.vcde.configuration.impl.MemoryBasedConfigurationImpl - Total memory MB: 333, base 512, increment 0
2425 [main] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Eagerly caching bean 'defaultConfiguration' to allow for resolving potential circular references
...
2462 [main] INFO com.vmware.vcde.configuration.impl.ConfigurationServiceImpl - Set default value activity.cacheResultTimeToLive = 86400
2462 [main] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Eagerly caching bean 'configurationService' to allow for resolving potential circular references
2475 [main] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Finished creating instance of bean 'configurationService'
...

After clean service statup file will grow to 8-20 MB

Part of lines during deploy ovf

136671 [transferService-http-f9e7f0a0-b22b-4387-a491-590582e3aabe-001-SVCVM-1.vmdk-upload] DEBUG org.apache.http.wire - >> "text deleted"

....

text deleted = "[0xb9][0xd6][0x9c][0xc9] etc..."

During the deployment of OVF, this file will grow in 2-3 minutes and /storage/log is completely full

What the problem?

Very ThanX!

P.S.

New, clean vcsa with 1 host (esxi 6.5) with 3 vms
After deploy 4th vm with 20Gb vmdk:

Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 652K 7.9G 1% /dev/shm
tmpfs 7.9G 696K 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda3 11G 4.4G 5.8G 43% /
tmpfs 7.9G 1.5M 7.9G 1% /tmp
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/core_vg-core 50G 52M 47G 1% /storage/core
/dev/mapper/db_vg-db 9.8G 102M 9.1G 2% /storage/db
/dev/sda1 120M 34M 78M 31% /boot
/dev/mapper/seat_vg-seat 25G 72M 24G 1% /storage/seat
/dev/mapper/updatemgr_vg-updatemgr 99G 69M 94G 1% /storage/updatemgr
/dev/mapper/log_vg-log 9.8G 6.1G 3.2G 66% /storage/log
/dev/mapper/dblog_vg-dblog 15G 86M 14G 1% /storage/dblog
/dev/mapper/archive_vg-archive 50G 64M 47G 1% /storage/archive

and content-library-runtime.log.stdout is 6 256 902 Kb

After deploy next vm with 20Gb vmdk

Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 656K 7.9G 1% /dev/shm
tmpfs 7.9G 696K 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda3 11G 4.4G 5.8G 43% /
tmpfs 7.9G 1.5M 7.9G 1% /tmp
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/core_vg-core 50G 52M 47G 1% /storage/core
/dev/mapper/db_vg-db 9.8G 102M 9.1G 2% /storage/db
/dev/sda1 120M 34M 78M 31% /boot
/dev/mapper/seat_vg-seat 25G 72M 24G 1% /storage/seat
/dev/mapper/updatemgr_vg-updatemgr 99G 69M 94G 1% /storage/updatemgr
/dev/mapper/log_vg-log 9.8G 9.2G 0 100% /storage/log
/dev/mapper/dblog_vg-dblog 15G 86M 14G 1% /storage/dblog
/dev/mapper/archive_vg-archive 50G 64M 47G 1% /storage/archive

and content-library-runtime.log.stdout is 9 568 324 Kb (deploy not finished)

This not normal !!!

 

0 Kudos
5 Replies
NeoProzett
Contributor
Contributor

I find temporary solution

Might be useful to anyone

 

Enable SSH on VCSA
Logon as root
type shell to run bash
execute service-control --stop content-library

Need to edit edit 2 files: log4j.properties and vdc.properties in /etc/vmware-content-library

I use sftp for edit files remotely

 

/etc/vmware-content-library/log4j.properties:

 

File content (changes/adds is  green):
P.S. i changed all levels DEBUG to WARN


# Copyright (c) 2022 VMware, Inc. All rights reserved. VMware Confidential
# Logging configuration
name=CLSLogger
packages=com.vmware.vdcs.activation.logging

rootLogger.level=WARN,file
rootLogger.appenderRef.rolling.ref=file

# content library configuration
logger.clvapi.name=com.vmware.vapi
logger.clcis.name=com.vmware.cis
logger.cl.name=com.vmware.cl
logger.clcsf.name=com.vmware.csf
logger.cliso.name=com.vmware.iso
logger.clovfs.name=com.vmware.ovfs
logger.cltransfer.name=com.vmware.transfer
logger.clvmtx.name=com.vmware.vmtx
logger.clregistry.name=com.vmware.registry
logger.clvcde.name=com.vmware.vcde
logger.clnsx.name=com.vmware.vcenter.nsx
logger.clvcloudactivity.name=com.vmware.vcloud.activity
logger.clvcloudpbm.name=com.vmware.vcloud.pbm
logger.clvdcs.name=com.vmware.vdcs
logger.clvdcstask.name=com.vmware.vdcs.task
logger.clvsphere.name=com.vmware.vsphere

# content library log level
logger.clvapi.level=WARN
logger.clcis.level=WARN
logger.cl.level=WARN
logger.clcsf.level=WARN
logger.cliso.level=WARN
logger.clovfs.level=WARN
logger.cltransfer.level=WARN
logger.clvmtx.level=WARN
logger.clregistry.level=WARN
logger.clvcde.level=WARN
logger.clnsx.level=WARN
logger.clvcloudactivity.level=WARN
logger.clvcloudpbm.level=WARN
logger.clvdcs.level=WARN
logger.clvdcstask.level=WARN
logger.clvsphere.level=WARN

#CUSTOM
log4j.logger.com.vmware.vsphere.vcde=WARN
log4j.logger.org.hibernate=WARN

#CUSTOM
## End of build type based log levels.
log4j.logger.org.springframework=WARN
log4j.logger.org.apache=WARN
log4j.logger.com.vmware.vim.vmomi=WARN

#CUSTOM
# Logging level of the performance benchmarking logger
log4j.logger.httpclient.wire=WARN

# External package configuration
logger.extvmomiclient.name=com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase
logger.extvmomiclient.level=ERROR
logger.extcisauth.name=com.vmware.cis.authorization.client.impl.ClientSideCache
logger.extcisauth.level=WARN
logger.extvapiproto.name=com.vmware.vapi.protocol.server.msg.json.JsonServerConnection
logger.extvapiproto.level=WARN
logger.extciscommon.name=com.vmware.cis.common.sso.impl.ClientHelperImpl
logger.extciscommon.level=WARN
logger.extcisauthz.name=com.vmware.vapi.cis.authz.impl.AuthorizationFilter
logger.extcisauthz.level=WARN
logger.extspringframework.name=org.springframework
logger.extspringframework.level=WARN
logger.extspringjpa.name=org.springframework.orm.jpa.JpaTransactionManager
logger.extspringjpa.level=WARN
logger.extspringcore.name=org.springframework.core.env.StandardEnvironment
logger.extspringcore.level=WARN

# RollingFile appender properties
appender.rolling.type=RollingFile
appender.rolling.name=file
appender.rolling.fileName=${sys:vcde.log.dir}/cls.log
appender.rolling.filePattern=${sys:vcde.log.dir}/cls.log.%i.gz
appender.rolling.policies.type=Policies
appender.rolling.policies.startup.type=OnStartupTriggeringPolicy
appender.rolling.policies.size.type=SizeBasedTriggeringPolicy
#appender.rolling.policies.size.size=50MB
appender.rolling.policies.size.size=10MB
appender.rolling.strategy.type=DefaultRolloverStrategy
appender.rolling.strategy.max=50
appender.rolling.strategy.fileIndex=min
appender.rolling.layout.type=PatternLayout
appender.rolling.layout.charset=UTF-8
appender.rolling.layout.pattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} | %-8p | %-16operationID | %-25t | %-30c{1} | %m%n

 

/etc/vmware-content-library/vdc.properties:

 

VcServer.clExtensionPaths=cl-extension.xml
VcServer.isoExtensionPaths=iso-extension.xml
VcServer.ovfExtensionPaths=ovf-extension.xml
VcServer.vmtxExtensionPaths=vmtx-extension.xml
activity.logPhaseTimeWarnMs=5000
activity.maxConcurrent=20000
cell.uuid=8e9fe4cc-4e8e-4b41-a204-367a6330ac1e
certificate.alias=vpxd-extension
cls.endpoint=cls
diagnostics.enable=false
httpServerEndpoint.maxIdleTime=360000
httpServerEndpoint.port=16666
jdbc.properties.path=${VMWARE_CFG_DIR}-vpx/vcdb.properties
sca.url=http\://localhost\:18091/sca/sdk
sca_config_path=/etc/vmware-content-library/config
security.key=KmJfI05JPEg2dD41c14/dQ\=\=
solution_user.owner_id=vpxd-extension-08267b37-0251-4bc2-b1ce-2b316da73335@vsphere.local
vc.ip=vcsa-test1.domain2.lan

 

after files edit start content library:

 

service-control --start content-library

 

Regards

El_Chomps_CR
Contributor
Contributor

 A new "content-library-runtime.log.stdout" file in the "/storage/log/vmware/content-library" directory is created allowing you to remove the old large file is that correct?  

0 Kudos
erki72
Enthusiast
Enthusiast

Hi @El_Chomps_CR 

Hi @NeoProzett 

I'm facing the same question. I made the adjustments above. Service restarted.

The big files are still there and new ones have been created and numbered. Can I delete the large files "content-library-runtime.log-2"?

Thanks

Tags (1)
0 Kudos
Ajay1988
Expert
Expert

Fixed in 6.7 U3s. Please upgrade asap.

vCenter Appliance 6.7 Update 3s (6.7.0.54000)2022-10-0620540798

https://kb.vmware.com/s/article/89009

 

 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
erki72
Enthusiast
Enthusiast

just updated to version 7