ArrowSIVAC
Enthusiast
Enthusiast

/tmp running out of Space - vSphere 5 - LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var defined. Looking for CIMPLE_HOME

vSphere 5.1.0  1065491

I noted disconnect of the host from vCenter though I could log into host directly via vSphere client.  So I knew the service was running but vCenter could not communicate.

System log noted

Host Error log1

The ramdisk 'tmp' is full.  As a result, the file /tmp/.SwapInfoSysSwap.lock.

LOCK.9133 could not be written.

error

10/22/2013 10:45:53 PM

vsphere1.acme.com

....and other various log events about being out of space

Host Error Log 2

The ramdisk 'tmp' is full.  As a result, the file /tmp/OcpLdn could not be

written.

error

10/22/2013 10:45:51 PM

vsphere1.acme.com

The ramdisk 'tmp' is full.  As a result, the file

/tmp/cimple_log_err_messages could not be written.

error

10/22/2013 10:13:49 PM

vsphere1.acme.com

....etc...

Logged into host via SSH and noticed a few things out of place.

-rw-r--r--    1 root root        5.6M Oct 23 02:13 cimple_log_err_messages

drwxr-xr-x    1 root root         512 Oct 23 01:02 ipp

-rw-------    1 root root           0 Oct 19 18:12 k5Vi9Y

-rw-------    1 root root           0 Oct 19 01:56 mGryZ3

-rw-r--r--    1 root root      186.1M Oct 23 02:16 mili2d.log

-rw-------    1 root root           0 Oct 19 18:11 ojl59g

/tmp # date

Wed Oct 23 02:23:56 UTC 2013

/tmp # hostname

vsphere1.acme.com

/tmp #

Odd as today is Oct 22.. so something

/tmp is part of the root volume and as such is very critical

Log Parse

cimple_log_err_messages

mili2d.log

#Newest Message:

2013/08/22 02:25:29:LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var defined. Looking for CIMPLE_HOME

#Oldest Message:

2013/10/22 06:36:07:LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var defined. Looking for CIMPLE_HOME

# Log file is not set to rotate.. that is first "Bad" thing.  Second is that it is calling for a system variable not set

~ # env

SSH_CLIENT=10.1.40.192 62660 22

MAIL=/var/mail/root

USER=root

OLDPWD=/vmfs/volumes

TERMINFO=/usr/share/terminfo

HOME=/

SSH_TTY=/dev/char/pty/t0

VI_USERNAME=root

LOGNAME=root

TERM=xterm

PATH=/bin:/sbin

LANG=en_US.UTF-8

SHELL=/bin/sh

PWD=/

LC_ALL=en_US.UTF-8

SSH_CONNECTION=10.1.40.192 62660 10.101.44.35 22

TMOUT=0

~ #

# This file seems to be a component of the standard CIM management interface of vSphere

#Newest Message:

Thu Aug 22 02:23:23 2013,8630,   ERROR:sync_targets_with_file:There is no savetgt file to opend. So returning success.

#Oldest Message:

Tue Oct 22 06:28:06 2013,12036,   ERROR:Send_Ioctl_PayLoad: actual response length is 0

Tue Oct 22 06:28:06 2013,12036,   ERROR:additional status is:0x0

Various other notable error entries....

Thu Sep 26 12:02:50 2013,12036,   ERROR:additional status is:0x0

Thu Sep 26 12:02:50 2013,12036,   ERROR:Unable to process command ioctl_result = -1 errno = 9 err_str = Bad file descriptor

# Log file is not set to rotate.. that is first "Bad" thing.  Second is that it is calling for a system variable not set

# My guess is this is some kind of log file by ?Emulex? or ?Qlogic? (OEM adapter cards in use for this server) and their driver has logging turned way up and is not doing lots of logging of IO events

Questions:

1) Has anyone else seen this?

2) Why is /tmp part of /  and as such impact the host in such a manner? (or is it that vendors are putting logs in incorrect paths?)

3) How do I fix lack of log rotation on files in /tmp

4) What are best practices in relation to running out of /  file space such as to avoid this?

The only odd thing I did notice also was that the date is off. It is one day in future.  This obviously is an issue (though NTP servers are set). It may play into this conversation but basic log management seems to be issue IMAO.

Thanks,

0 Kudos
6 Replies
samdoyle
VMware Employee
VMware Employee

This is a pretty old post but I got this hit when I was searching for the same error so figured I would dump the info here for others that stumble upon it.

I just recently upgraded CIMPLE and starting seeing these show up. Obviously not very clear on what is going on and not much in terms of searching shows up much. Turns out there is now .cimplerc file that is expected to configure some options like logging and log limits. The location of that is either to be set when building cimple via the --cimplehome-envvar configuration if this is not set then you need to set the environment variable CIMPLE_HOME.

A sample .cimplerc looks like this

# FATAL - provides the highest level log messages.

# ERR - default configuration.

# WARN - captures warning messages.

# INFO - provides additional detailed information.

# DBG - provides the lowest level log messages.

LOG_LEVEL=ERR

# This is in bytes

# Set to 5MB

MAX_LOG_FILE_SIZE=5242880

MAX_LOG_BACKUP_FILES=5

If you don't set a max size you will have a single log that grows unbounded, if you don't set max backup files you get an unbounded number of backup files.

Hope this helps,

S.D.

0 Kudos
terryhovertex
Contributor
Contributor

Hi Samdoyle,

Would you mind to clarify that what is the path of .cimplerc located?

Regards,

Terry

0 Kudos
samdoyle
VMware Employee
VMware Employee

It needs to be explicitly set through the environment variable CIMPLE_HOME or through configure with the --cimplehome-envvar.

If neither is set, no logs are created and you get the constant logging you are seeing which is unfortunately at an inappropriate error level for this type of message.

In previous releases it was fixed through HOME env.

S.D.

0 Kudos
terryhovertex
Contributor
Contributor

mm.....so what is the default path of CIMPLE_HOME and how can i add this env var?  Thanks.

0 Kudos
ArrowSIVAC
Enthusiast
Enthusiast

As I started this thread, I wanted to follow up with some details that I have learned.

CIM issues from the vendor in this example was / is IBM, but this issue is by no means just an IBM issue.  I have validated almost every other vendor has done the same thing and made the same mistake in one degree or another.

Each of the various vendors have compiled their own versions of vSphere to reflect the subset of HBAs and corresponding drivers into the installation media and updates. And each of the vendors have 'struggled' with the same mistake of leaving CIM loggers in debug mode, or left to log to default space of /tmp. This fills up /tmp and the OS "goes offline" as it is not able to create ssh key files etc that vCenter needs to connect / communicate.

As you can expect, the implementation of vSphere has a single root volume group and with USB boot keys, has limited file system space (not that internal boot or SAN boot has much better as it just lengthens time before they experience the errors).

Emulex and Intel CIM agents (Though I have heard of other OEM adapters doing same) are being left on and dumping to /tmp  This is the core of the issue.  I have worked with two of the vendors who have realized the issue and in upcoming distributions have made adjustments to decrease this logging (note I did not say eliminate).  The CIM loggers from each vendor come in at different levels and some systems do not even have those OEM cards and would not have the logging issue, so it is hard to nail down all the sources.  What I can say is they have stated goals to redress this issue such that CIM logging is reflected per VMWare's direction into proper system event trapping.

I have yet to test, and have this upcoming soon, but was assured that using the OEM upgrade package includes in it the adjustments and fixes for CIM loggers, so no manual intervention is required.  Obviously reboot will clear out /tmp and "solve" the issue (well, reset the timer till it happens again) but reboot is not a fix!

I will post as I learn more

0 Kudos
samdoyle
VMware Employee
VMware Employee

It depends on what is using the CIMPLE environment. In the case of VAMI, which leverages SFCBD, the init script /etc/init.d/vami-sfcb is one place to put it.

e.g.

export CIMPLE_HOME=/opt/vmware/var/log/cimple

then create your .cimplerc for example

CIMPLERC=/opt/vmware/var/log/cimple/.cimplerc

if [[ ! -e $CIMPLERC ]]; then

    echo "$CIMPLERC file is missing, creating default !"

    mkdir -p $(dirname $CIMPLERC)

    cat > $CIMPLERC << "EOF"

# FATAL - provides the highest level log messages.

# ERR - default configuration.

# WARN - captures warning messages.

# INFO - provides additional detailed information.

# DBG - provides the lowest level log messages.

LOG_LEVEL=ERR

# This is in bytes

# Set to 5MB

MAX_LOG_FILE_SIZE=5242880

MAX_LOG_BACKUP_FILES=5

EOF

fi

terryhovertex wrote:

mm.....so what is the default path of CIMPLE_HOME and how can i add this env var?  Thanks.

0 Kudos