Solved: Re: vsantraces loging location

marnow · ‎03-19-2018

Hi,

I noticed an odd thing on one of my host, the df-u shows 100% usage on SD partition:

vfat 299712512 299130880 581632 100% /vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0

vfat 261853184 165019648 96833536 63% /vmfs/volumes/ef09f1e9-363d4c9f-a811-a0dead161d32

vfat 261853184 165007360 96845824 63% /vmfs/volumes/cc2c81e1-f01e34ad-36d4-b84e1a85ea62

I'v started digging into it and found:

[root@dh-vhost01:/vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0] ls -lh

total 768

drwxr-xr-x 1 root root 8 Jan 2 18:36 packages

drwxr-xr-x 1 root root 8 Jul 14 2017 var

drwxr-xr-x 1 root root 8 Mar 19 16:20 vsantraces

quite strange, I don't remember setting/redirecting the logs to this partition!

the vsantraces also exist in Ramdisk....

Ramdisk Size Used Available Use% Mounted on

root 32M 4M 27M 12% --

etc 28M 376K 27M 1% --

opt 32M 4K 31M 0% --

var 48M 2M 45M 5% --

tmp 256M 56M 199M 22% --

iofilters 32M 0B 32M 0% --

epd 50M 0B 50M 0% --

hostdstats 2053M 17M 2035M 0% --

vsantraces 300M 67M 232M 22% --

Both locations are active, the logs are there and rotating....

Have anyone idea how to fix this and eliminate vsantraces from the SD /vmfs partition?

marnow · ‎04-06-2018

The issue has been resolved. I reboot the host after a regular maintenance task. I don't vsantraces on the volume.

View solution in original post

TheBobkin · ‎03-19-2018

Hello marnow,

You can check where vsantraces are logging using:

# esxcli vsan trace get

Configure using:

# esxcli vsan trace get <options>

Or if you want to see more granular options and/or change them by editing /etc/vmware/vsan/vsantraced.conf

Restart the vsantrace services after making any changes to apply:

# /etc/init.d/vsantraced restart

Bob

marnow · ‎03-19-2018

It is properly configured:

[root@dh-*****01:/tmp/scratch] esxcli vsan trace get

VSAN Traces Directory: /vsantraces

Number Of Files To Rotate: 8

Maximum Trace File Size: 45 MB

Log Urgent Traces To Syslog: true

But why is still writing to VFAT partition on SD card?

TheBobkin · ‎03-19-2018

Hello marnow,

Check the .conf, do you have urgent traces going to syslog and if so, where is syslog set to? (e.g. /scratch/log by default points to a default-sized 300MB vfat partition)

Can you share the contents, ls -lah and du -h from those two locations? (/vsantraces and /vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0)

Bob

marnow · ‎03-19-2018

Only uncommentated two lines in .conf

VSANTRACED_LAST_SELECTED_VOLUME="/vsantraces"

VSANOBSERVER_MAX_MB_SIZE="10"

I do have sysyog configured, all logs going to vRalize Log Insight.

esxcli system syslog config get

Default Network Retry Timeout: 180

Dropped Log File Rotation Size: 100

Dropped Log File Rotations: 10

Enforce SSLCertificates: true

Local Log Output: /scratch/log

Local Log Output Is Configured: false

Local Log Output Is Persistent: false

Local Logging Default Rotation Size: 1024

Local Logging Default Rotations: 8

Log To Unique Subdirectory: false

Message Queue Drop Mark: 90

Remote Host: udp://vrlog.sys.****:514

/vsantraces

/vsantraces] ls -lah

total 61780

drwxr-xr-x 1 root root 512 Mar 19 20:34 .

drwxr-xr-x 1 root root 512 Mar 19 13:49 ..

-rw-r--r-- 1 root root 1.3M Mar 19 18:24 vsanObserver--2018-03-19T18h04m01s.gz

-rw-r--r-- 1 root root 1.3M Mar 19 18:46 vsanObserver--2018-03-19T18h26m01s.gz

-rw-r--r-- 1 root root 1.3M Mar 19 19:08 vsanObserver--2018-03-19T18h48m01s.gz

-rw-r--r-- 1 root root 1.3M Mar 19 19:30 vsanObserver--2018-03-19T19h10m25s.gz

-rw-r--r-- 1 root root 1.3M Mar 19 19:52 vsanObserver--2018-03-19T19h32m01s.gz

-rw-r--r-- 1 root root 1.3M Mar 19 20:14 vsanObserver--2018-03-19T19h54m02s.gz

-rw-r--r-- 1 root root 1.2M Mar 19 20:34 vsanObserver--2018-03-19T20h16m02s.gz

-rw-r--r-- 1 root root 6.6M Mar 19 19:30 vsantraces--2018-03-19T19h20m25s888.gz

-rw-r--r-- 1 root root 7.1M Mar 19 19:40 vsantraces--2018-03-19T19h30m25s671.gz

-rw-r--r-- 1 root root 6.6M Mar 19 19:50 vsantraces--2018-03-19T19h40m26s130.gz

-rw-r--r-- 1 root root 6.7M Mar 19 20:00 vsantraces--2018-03-19T19h50m26s241.gz

-rw-r--r-- 1 root root 7.3M Mar 19 20:10 vsantraces--2018-03-19T20h00m26s181.gz

-rw-r--r-- 1 root root 6.9M Mar 19 20:20 vsantraces--2018-03-19T20h10m26s508.gz

-rw-r--r-- 1 root root 6.9M Mar 19 20:30 vsantraces--2018-03-19T20h20m25s546.gz

-rw-r--r-- 1 root root 3.2M Mar 19 20:34 vsantraces--2018-03-19T20h30m24s984.gz

-rw-r--r-- 1 root root 192 Mar 19 20:30 vsantraces.index

-rw-r--r-- 1 root root 3.2K Mar 19 20:10 vsantracesLSOM--2018-03-19T20h00m26s286.gz

-rw-r--r-- 1 root root 3.2K Mar 19 20:20 vsantracesLSOM--2018-03-19T20h10m26s623.gz

-rw-r--r-- 1 root root 3.2K Mar 19 20:30 vsantracesLSOM--2018-03-19T20h20m25s657.gz

-rw-r--r-- 1 root root 2.8K Mar 19 20:32 vsantracesLSOM--2018-03-19T20h30m25s086.gz

-rw-r--r-- 1 root root 96 Mar 19 20:30 vsantracesLSOM.index

-rw-r--r-- 1 root root 19.6K Mar 19 20:10 vsantracesUrgent--2018-03-19T20h00m26s225.gz

-rw-r--r-- 1 root root 20.5K Mar 19 20:20 vsantracesUrgent--2018-03-19T20h10m26s555.gz

-rw-r--r-- 1 root root 19.5K Mar 19 20:30 vsantracesUrgent--2018-03-19T20h20m25s592.gz

-rw-r--r-- 1 root root 9.6K Mar 19 20:34 vsantracesUrgent--2018-03-19T20h30m25s036.gz

-rw-r--r-- 1 root root 96 Mar 19 20:30 vsantracesUrgent.index

:/vsantraces] du -h

60.6M .

/vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0

/vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0] ls -lah

total 768

drwxr-xr-x 1 root root 512 Mar 19 20:37 ..

drwxr-xr-x 1 root root 8 Jan 2 18:36 packages

drwxr-xr-x 1 root root 8 Jul 14 2017 var

drwxr-xr-x 1 root root 8 Mar 19 20:30 vsantraces

:/vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0] du -h

205.9M ./packages/6.5.0/vmtools

1001.5K ./packages/6.5.0/floppies

207.2M ./packages/6.5.0

263.5K ./packages/var/db/locker/vibs

256.0K ./packages/var/db/locker/profiles

775.5K ./packages/var/db/locker

1.0M ./packages/var/db

1.3M ./packages/var

208.7M ./packages

256.0K ./var/core

512.0K ./var

64.9M ./vsantraces

274.4M .

TheBobkin · ‎03-19-2018

Hello marnow,

I bet the contents of /vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0/vsantraces and /vsantraces are identical other than the current file for each of vsantraces, vsantracesLSOM, vsantracesUrgent and minus whatever rolled over.

This is likely due to having scratch set to /scratch/log, check where this is set to:

https://kb.vmware.com/s/article/1033696

Bob

marnow · ‎03-19-2018

Thanks Bob!

Yes, the contents in both location are identical, name and size wise

The scratch partition is not configured, /etc/vmware/locker.conf is empty.

Scratch is in the Ramdisk and sent to vRlog

esxcli system visorfs ramdisk list

Ramdisk Name System Include in Coredumps Reserved Maximum Used Peak Used Free Reserved Free Maximum Inodes Allocated Inodes Used Inodes Mount Point

------------ ------ -------------------- --------- ----------- --------- ---------- ----- ------------- -------------- ---------------- ----------- ---------------------------

root true true 32768 KiB 32768 KiB 4252 KiB 4256 KiB 87 % 87 % 26624 5280 5250 /

etc true true 28672 KiB 28672 KiB 376 KiB 420 KiB 98 % 98 % 4096 1024 649 /etc

opt true true 0 KiB 32768 KiB 4 KiB 4 KiB 99 % 0 % 8192 1024 31 /opt

var true true 5120 KiB 49152 KiB 2700 KiB 4724 KiB 94 % 47 % 8192 1056 799 /var

tmp false false 2048 KiB 262144 KiB 58888 KiB 108400 KiB 77 % 0 % 8192 576 198 /tmp

iofilters false false 0 KiB 32768 KiB 0 KiB 0 KiB 100 % 0 % 10240 32 1 /var/run/iofilters

epd false true 0 KiB 51200 KiB 0 KiB 0 KiB 100 % 0 % 8192 32 1 /epd

hostdstats false false 0 KiB 2102272 KiB 17752 KiB 17752 KiB 99 % 0 % 8192 32 5 /var/lib/vmware/hostd/stats

vsantraces false true 0 KiB 307200 KiB 79328 KiB 227064 KiB 74 % 0 % 8192 32 28 /vsantraces

lrwxrwxrwx 1 root root 23 Jan 2 19:20 productLocker -> /locker/packages/6.5.0/

lrwxrwxrwx 1 root root 4 Dec 18 10:42 sbin -> /bin

lrwxrwxrwx 1 root root 12 Jan 2 19:20 scratch -> /tmp/scratch

lrwxrwxrwx 1 root root 49 Jan 2 19:20 store -> /vmfs/volumes/5968d919-81f56ea0-fe93-246e963e72f0

drwxr-xr-x 1 root root 512 Feb 12 15:27 tardisks

drwxr-xr-x 1 root root 512 Jan 2 19:03 tardisks.noauto

drwxrwxrwt 1 root root 512 Mar 19 21:40 tmp

drwxr-xr-x 1 root root 512 Jan 2 19:03 usr

drwxr-xr-x 1 root root 512 Jan 2 19:03 var

drwxr-xr-x 1 root root 512 Jan 2 19:03 vmfs

drwxr-xr-x 1 root root 512 Jan 2 19:03 vmimages

lrwxrwxrwx 1 root root 18 Dec 18 10:42 vmupgrade -> /locker/vmupgrade/

drwxr-xr-x 1 root root 512 Mar 19 21:40 vsantraces

[root@dh-vhost01:~]

marnow · ‎04-06-2018

The issue has been resolved. I reboot the host after a regular maintenance task. I don't vsantraces on the volume.