vCenter 7U3 - Log exhaustion - WCP Service

[rom2311]

Hey everyone,

I have an environment with a vCenter version 7U3 build - 21477706.
We have an error in the environment regarding Log exhaustion in the /storage/log disk.

We tried inspecting further following the KBs: 
1. Troubleshooting vCenter Appliance /storage/log directory is 80% or more full (83070) (vmware.com)
2. vCenter Server Appliance disk space is full (76563) (vmware.com)

The only thing I've noticed is with the /storage/log/vmware/wcp folder having a huge log file that takes up almost all of the disk space.

It can be seen in the screenshot that the problematic file is - stdstream.log.stderr
I've tried finding more information regarding the file and the WCP service but I couldn't really understand the purpose and use of the service and if deleting the file and restarting the service is the solution.

Any help would be appreciated, 
Thanks, Rom.

[battybishop]

Workload Control Plane (WCP), is a service that runs within the vCenter appliance and is responsible for managing the integration of Kubernetes with vSphere.

Have you checked the /var/log/vmware/wcp/wcpsvc.log to identify issues?

[markey165]

You could try running cat /storage/log/vmware/wcp/stdstream.log.stderr to examine the contents and see what's causing it to get so large.   If its easier, you could also SCP the file off and examine it with notepad++

I've had something similar with a different service which was a known issue, and i recall the fix there was to simply zero size the file, but obviously it would useful to know what's causing it to get so large in the first place. Its obviously had (or is still having) some kind of issue that's causing that log to fill with errors. If in doubt, and you can't determine the cause, I'd recommend raising a support call, they should be able to advise you of best course of action!

HTH

[Ajay1988]

This could be a bug. I recall similar was fixed way back in early release of 7.0 . This could be something else but similar . Better to open SR with VMware Support and share VC bundle. 

Can you go share output of below commands.

du /storage/log/vmware/ -ahx . | sort -rh | head -n 20
du -hs * | sort -rh | head -10

[rom2311]

Hey, 

Sorry for the late response, been sick the last 2 weeks..
@battybishop , I did check that log and the only thing repeating multiple times - 

@markey165 I've tried looking into the /storage/log/vmware/wcp/stdstream.log.sderr log itself, but it seems unreadable. I'm now trying to download it and watch it on notepad.

@Ajay1988, It does make sense, as I recall a similar problem in earlier releases as well, thought maybe I just couldn't find the KBs that talk about it..
Here is the output of du /storage/log/vmware/ -ahx . | sort -rh | head -n 20 -

Anyways, I will try to upgrade the vCenter to 7u3n and see if it solves the problem and in the meantime I will open a SR as well.

[Ajay1988]

Upgrade to 7.0 U3n . Should fix the issue. Zero out or truncate /storage/log/vmware/wcp/stdstream.log.stderr . You might have to stop wcp service. Then upgrade and check.

I don't have issue in my lab.

root@is-dhcp42-38 [ ~ ]# rpm -qa | grep -i wcp-
VMware-wcp-0.0.23-21882390.x86_64
root@is-dhcp42-38 [ ~ ]# vpxd -v
VMware VirtualCenter 7.0.3 build-21958406

Can you also provide below output :
tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr

[rom2311]

Hey @Ajay1988, 

This is the output for tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr - 

Still, no real error I know how to solve.. We have decided to upgrade next week so I'll keep you posted 🙂

[battybishop]

I think upgrading to the latest version is your best bet

[liambourke]

I am having a similar issue.

What does the stdstream.log file do?

[navina]

Hope this helps
https://kb.vmware.com/s/article/94709?lang=en_US

[jrgalleg]

I have the same issue.

What I can see is 2 possible solutions:

https://kb.vmware.com/s/article/94709

https://kb.vmware.com/s/article/95479

Can you please suggest me the best option?

My VCenter version is: vCenter Server 7.0 U3o

Thx in advance.

[markey165]

@jrgalleg - I'd refer you to the 2nd link you posted (95749) and follow the instructions for the workaround. Specifically :

To mitigate this problem, please take a fresh backup of the vCenter Server Appliance, then remove all stdstream.log* files from the folder and restart the wcp service:

# service-control --stop wcp; rm /var/log/vmware/wcp/stdstream.log*; service-control --start wcp

In other words, delete (using the rm command) the files that end with a number (as shown in Black below), but don't delete the one highlighted in red, as that one is the live log file. The others are backups. If you wish to take a copy of the files to be on the safe side (although its unlikely you would need them as they are just log files) you can use WinSCP to connect over Port 22 (the SSH port) and copy them off

1.4M  -rw-------  1 wcp  lwisRegReader 1.4M <DATE> <TIME> stdstream.log-1.stderr
34M  -rw-------  1 wcp  lwisRegReader  34M <DATE> <TIME>stdstream.log-4.stderr
111M  -rw-------  1 wcp  lwisRegReader 111M <DATE> <TIME> stdstream.log.stderr1.1G  -rw-------  1 wcp  lwisRegReader 1.1G <DATE> <TIME> stdstream.log-3.stderr
1.6G  -rw-------  1 wcp  lwisRegReader 1.6G <DATE> <TIME> stdstream.log-0.stderr 

 
Once you have restarted the service you can then monitor the log to see if the problem returns. If it does, I would then raise a call with VMware support to see what is causing the logs to grow so big.

HTH

[Scott_BHF]

Thanks

This worked a treat for me

[markey165]

@Scott_BHF No worries, glad it helped you 😊