ESXi5 syslog.log error (?) messages

BlueBolt · ‎12-06-2012

Hello

ESXi 5.0.0, 623860

I get hundreds of these messages at 5-second intervals in syslog.log. Cannot find any description or discussion about this particular message - ring any bells with anyone? I've restarted services twice with no change.

2012-11-30T21:57:02Z sfcb-CIMXML-Processor[6573115]: TicketCache --- Can't query '/var/run/sfcb/c2200646-6076-43a3-b567-5a3eb2480b6a', Error: 'Value too large for defined data type'
2012-11-30T21:57:07Z sfcb-CIMXML-Processor[6573115]: TicketCache --- Can't query '/var/run/sfcb/c2200646-6076-43a3-b567-5a3eb2480b6a', Error: 'Value too large for defined data type'
2012-11-30T21:57:12Z sfcb-CIMXML-Processor[6571076]: TicketCache --- Can't query '/var/run/sfcb/c2200646-6076-43a3-b567-5a3eb2480b6a', Error: 'Value too large for defined data type'

The contents of /var/run/sfcb/c2200646-6076-43a3-b567-5a3eb2480b6a and one other /var/run/sfcb/... file is simply:

root~

Normal logging activity, non-fatl but annoying logging bug/feature or a runaway something?

Thanks for any light shed on it!

- Cal Sawyer @ Bluebolt

nielse · ‎12-06-2012

Do you get an error or warning when you restart the services regarding sfcbd or sfcbd-watchdog?

Could you run the following commands via SSH:

/etc/init.d/sfcbd status

/etc/init.d/sfcbd-watchdog status

You can also restart them using "restart" - they might give an error/warning...

@nielsengelen - http://foonet.be - VCP4/5

spravtek · ‎12-06-2012

I see I'm a bit late to the party ... But yeah ... http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203779...

nielse · ‎12-06-2012

Never too late . But yes the KB u provided should be the solution but normally a restart should remove the files.

@nielsengelen - http://foonet.be - VCP4/5

BlueBolt · ‎12-07-2012

Thanks to everyone for the quick replies and advice! Yes, i've read that KB and tried it's recommendations, but the error mssgs in syslog.log persist, although referencing newly-created /var/run/sfcb/[UUID] files. Looking back through syslog.log, these messages go back some time, although the log has been rotated (first entry is 2012-11-30T18:54:06Z)

BTW: If i run service.sh retart, i get the mentioned sfcb start error, but sfcb is always found to be running - i sort of wrote that off to an error in the service.sh script. Here's a full stop/start/check-logs run:

~ # /etc/init.d/sfcbd status
sfcbd is running
~ # /etc/init.d/sfcbd
~ # /etc/init.d/sfcbd-watchdog status
sfcbd is running.

~ # /etc/init.d/sfcbd-watchdog stop

~ # ls -l /var/run/sfcb/
-rw-------    1 root     root                  4 Dec 6 12:36 042f9f80-be10-4312-be6e-388a33635e82
-rw-------    1 root     root                  4 Dec 6 21:49 07c7f010-0027-4685-8ca3-83d0e86f197f
-rw-------    1 root     root                  4 Dec 6 12:40 592ce030-7604-41ae-949d-65ba950cf08d
-rw-------    1 root     root                  4 Dec 7 06:57 61680923-5242-413a-9ac7-2e1e32f8e0e1
-rw-------    1 root     root                  4 Dec 6 18:46 9e46c5bf-5ea3-489f-a717-4713aeebadd6
-rw-------    1 root     root                  4 Dec 6 15:40 a351494e-ff61-44be-9a10-6357bc34c098
-rw-------    1 root     root                  4 Dec 7 00:52 c91ba4de-2871-453f-988d-9fff58fd65e1
-rw-------    1 root     root                  4 Dec 7 03:55 d1d41ab5-5c09-4a9f-9138-30385e108761

(doesn't look like sfcb has expired it's runfiles)

~ # rm -fv /var/run/sfcb/*
~ # ls -l /var/run/sfcb/

<nada>

~ # /etc/init.d/sfcbd-watchdog start
~ # ls -l /var/run/sfcb/

<nada>

~ # /etc/init.d/sfcbd-watchdog status
sfcbd is running.
~ # /etc/init.d/sfcbd status
sfcbd is running

~ # tail /var/log/syslog.log
2012-12-07T08:37:54Z sfcbd-watchdog: Watchdog active: interval 60 seconds, pid 6992813
2012-12-07T08:37:54Z sfcbd-watchdog: starting sfcbd
2012-12-07T08:37:54Z sfcbd: Starting sfcbd
2012-12-07T08:37:55Z sfcb-sfcb[6990922]: --- Log syslog level: 3
2012-12-07T08:38:00Z cimslp: --- Using /etc/sfcb/sfcb.cfg
2012-12-07T08:38:06Z sfcb-CIMXML-Processor[6995040]: TicketCache --- Can't query '/var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1', Error: 'Value too large for defined data type'
2012-12-07T08:38:11Z sfcb-CIMXML-Processor[6995040]: TicketCache --- Can't query '/var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1', Error: 'Value too large for defined data type'
2012-12-07T08:38:16Z sfcb-CIMXML-Processor[6995040]: TicketCache --- Can't query '/var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1', Error: 'Value too large for defined data type'
2012-12-07T08:38:23Z sfcb-CIMXML-Processor[6995040]: TicketCache --- Can't query '/var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1', Error: 'Value too large for defined data type'
2012-12-07T08:38:23Z sfcb-CIMXML-Processor[6995040]: TicketCache --- Can't query '/var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1', Error: 'Value too large for defined data type'

~ # cat /var/run/sfcb/a2a8612a-1784-4703-a52a-69f6fda4afb1
root~ #

* ?? What is the "correct" content of these /var/run/sfcb/[UUID] files?

* A few minutes, later, this popped up in the shell:

~ # cannot stat /var/run/vmware/tickets/vmtck-524431be-c67e-ce

so ...

~ # ls -l /var/run/vmware/tickets/vmtck-524431be-c67e-ce
-rw------- 1 root root 42 Dec 7 08:48 /var/run/vmware/tickets/vmtck-524431be-c67e-ce

* and it stats fine for me!

~ # stat /var/run/vmware/tickets/vmtck-524431be-c67e-ce
File: "/var/run/vmware/tickets/vmtck-524431be-c67e-ce"
Size: 42            Blocks: 8          IO Block: 131072 regular file
Device: 1h/1d    Inode: 18446744072816497911 Links: 1
Access: (0600/-rw-------) Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-12-07 08:49:30.000000000
Modify: 2012-12-07 08:48:40.000000000
Change: 2012-12-07 08:48:40.000000000

Perhaps the server is obliquely asking for a reboot?

cheers

- cal

spravtek · ‎12-07-2012

What hardware are you using? Did you install something extra, like extra CIM providers? the error you see could have something to do with CIM providers ...

sfcb-CIMXML-Processor

It seems it wants to query something but the value is too large ...

BlueBolt · ‎12-07-2012

You may have already exceeded my knowledge of VMWare I did install HP ESXi Utilities Offline Bundle a couple of days ago, but nothing before that. The system hasn't been restarted to make the HP utils take effect - they list in esxcli software vib list --rebooting-image:

char-hpilo            500.9.0.0.9-1OEM.500.0.0.434156     Hewlett-Packard PartnerSupported 2012-12-05
hpacucli              9.20-9.0                            Hewlett-Packard PartnerSupported 2012-12-05
hpbootcfg             01-01.02                            Hewlett-Packard PartnerSupported 2012-12-05
hponcfg               04-00.10                            Hewlett-Packard PartnerSupported 2012-12-05

All of the other VIBs are VMware/VMwareCertified

Hardware is an HP dl360G7 with upgraded p410i and internal HP SAS HDDs

cheers

spravtek · ‎12-07-2012

Interesting...

I work a lot with HP and haven't run into this before ... Though maybe I'm just missing it

Well, the thing to test is to see if a reboot might help (when possible), maybe something isn't registered correctly... Besides that I'd need to investigate further

BlueBolt · ‎12-07-2012

Yes, need to reboot later this year to see if this all just disappears. Could be a while before that happens, so if i go silent for a while that's what's (not) happening. Many thanks to everyone for their help and advice.