ESXi 6.5 /var/log no space left - pysikal console don't react

[MLoeschner]

Hello folks, I hope that you can help me here. Following problem:

- All VM's are still running

- Login in the Web Gui is no longer possible

- Login possible in the physical console

- At first I thought that the login is locked

- Unlock command returned the following response: pam_tally2 update fsync failed for tallylog no space left on device - After that, the console no longer reacts to inputs, i.e. I can enter a command, but there is no reaction, no output

- therefore I cannot delete a log file I suspect \var\log is finally full now.

How do I get out of the impasse? What would happen if I restarted the server? Does a restart give the ESXi some free space again, so that a login and further analysis are possible? I don't dare to restart and hope for help, confirmation. Best regards

Manfred

[Lalegre]

@MLoeschner,

A reboot could free some space for sure, however, your VMs will suffer some impact. If your ESXi is in a cluster with HA, the VMs should reboot on another available host, otherwise, they will be powered off not properly.

If the ESXi is 100% not responsive and you cannot even login, I believe you will need to reboot it.

[connecttoduy]

Hello

You can login ssh to host and run command 

#df -h

and check where system is full --> and check KB for deleted file be able delete for free space

Regards,

Duy

[MLoeschner]

Many thanks for your response. A reboot seems to be the only option. The only question is whether it will boot. I will have no choice but to try it.

[MLoeschner]

Many thanks for your response. A login to the SSH shell is no longer possible. I can log into the shell on the host, but no more commands are accepted. I think the only way is to reboot and hope it boots.

[e_espinel]

Hello.
If you have access to the console, from there you can enable SSH access

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-F9234F4A-FD6A-477B...

After enabling SSH access, then you can login from an SSH session (putty) to the ESXi host and run commands. You must use the user root and your password

[MLoeschner · Accepted solution]

Hello,

tThank you for the answer, but unfortunately it doesn't help. SSH is activated, but a login in the SSH Shell (Putty) is not possible because /var/log no space left. It seems to me that this is a very rare phenomenon.

[MLoeschner]

Hello, Thanks to all. The problem is solved, a reboot has freed up the storage space (the log memory is on the RAM disk) and afterwards it is possible to log in to the SSH shell again.

Greetings from Hannover