failed to write log, No space left on device

[Gayathri_Rathna]

Hi All,

Anyone came across the error . The below error is on ESXi 5.1 1483097

2016-02-24T06:42:38.152Z vmsyslog.main            : ERROR   ] Watchdog 28960927 fired (child 28960928 died with status 256)!

2016-02-24T06:42:38.248Z vmsyslog                 : CRITICAL] vmsyslogd daemon starting (29468839)

2016-02-24T06:42:38.257Z vmsyslog.main            : ERROR   ] Watchdog 28960927 exiting

2016-02-24T06:42:41.235Z vmsyslog.loggers.file    : ERROR   ] Gzip logfile /scratch/log/vmkernel0.gz failed <type 'exceptions.MemoryError'>

2016-02-24T06:42:41.274Z vmsyslog.main            : ERROR   ] <vmkernel> failed to write log, disabling

2016-02-24T06:42:41.279Z vmsyslog.main            : ERROR   ] failed to send vob: [Errno 28] No space left on device

Scratch partition size is 4GB and it is utilized 1%

Syslog service was stopped by something. Because of that no logs were written on vmkernal.

MY questions are:

1. was the above error really indicate that scratch partition running out of space
2. Under what circumstance we will get this error
3. What are the possibilities that syslogd services got stopped
4. If syslog got stopped will it be started automatically by the system

Can some one answer.

[msripada]

Hello Gayathri,

Can you run the command vdf -h and check the /tmp and ramdisk space on the ESXi host

As per the ESXi build, this should be resolved as per this KB as per the build you are already on ESXi 5.1 update 2

You can try to increase some more (as advised to 60MB)  memory resources by following the steps in the above kb

To increase the maximum memory limit of the logging resource group:

1. In vSphere Client, click to select the ESXi host.
2. Click the Configuration tab.
3. Under Software, Click System Resource Allocation.
4. Click Advanced
5. Under System Resource Pools, expand host > vim > vmvisor and click logging.
6. Click Edit settings. The Edit Logging Resource Allocation dialog box appears.
7. In the Edit Logging Resource Allocation dialog box, set the Limit under Memory Resources to 48 MB.
   
   Note: In some instances 48MB may not be enough to resolve this issue depending on the ESXi host load.  Increasing the Memory resources to 60MB should resolve this.
   
   
8. Click OK.
9. Log in to the ESXi host as the root user. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
10. Check the watchdog and vmsyslogd process IDs (PIDs)by running the command:
    
    ps -Cc|grep vmsyslogd
    
    You see output similar to:
    
    5462 5462 vmsyslogd
    4523 4523 wdog-4523
    
11. Using the PIDs from Step 10, run the kill command for the watchdog process first and then the vmsyslogd process: <br style="box-sizing: border-box;">
    
    For example:
    
    kill -9 4523
    kill -9 5462
    
    
12. Restart the vmsyslogd process by running the command:
    
    /usr/lib/vmware/vmsyslog/bin/vmsyslogd

Thanks,

MS