How to catch signals like SIGTERM before a esxi host shuts down?

[dee2]

Hi 

I want to do some cleanup before the esxi host is shut down. I using trap to catch signals and trigger a function.

This is code I have written called ts.sh

function trigger_sigint ()

{

echo "This is a response to interrupt signal SIGINT(2)";

}

function trigger_sigterm ()

{

echo -e "\nSystem shutdown signal revd.";

log "System shutdown signal revd." log_file

}

trap trigger_sigint SIGINT

trap trigger_sigterm SIGTERM

while [[ true ]] ; do

sleep 1;

done

When I run this script in the background and send signals to it using command
kill -SIGTERM %1

the script calls the corresponding function defined above and prints the message.

But when I shutdown the esxi host, it does not print any message.
Also nothing is printed in the log_file.

So how do I get function trigger_sigterm to be called automatically before the esxi host shuts down?

Which signal does it send?  and how to catch it?

Thanks

-Deepali

[Gizzie]

When an ESXi host is shut down or powered off, the VMkernel, which is the core of the ESXi hypervisor, manages the shutdown process. The VMkernel sends a special ACPI power-off signal to the virtual machines running on the host, instructing them to shut down gracefully. This signal is not directly catchable by user-space scripts or processes like SIGTERM. Therefor, your shell script running on ESXi host will not receive the signal when the host is shut down, and your custom signal handler function will not be executed automatically.

[dee2]

Does esx host write any log message or send any signal to the kernel before shutdown.
How does vsphere shutdown vms when esx host (physical server) is shutdown

[dee2]

Hi Gizzie,

Thanks for the answer.

In case of a problem I want to know if the esxi host was shutdown deliberately or if it crashed/ unreachable.
Is there any way to find that out.

Thanks

[Gizzie]

There are a few ways where you can check this:

• ESXi Logs located in the /var/log directory on the ESXi host
  • vmkernel.log for information about vmkernel (obviously), kernel-level events, hardware issues and/or system messages
  • vmksummary.log for VMkernel activities (including system boot, uptime and shutdown events)
• ESXi Event Logs that can be accessed via esxcli or vSphere Client. Here you can find details from shutdown events, system errors, and user-initiated actions.

[dee2]

Hi Gizzie,

Thanks for some more insight.

I see following messages in vmksummary.log 

[root@Poweredge-R720-ESXi6:/var/log] tail -f vmksummary.log | grep -i "powering off"
2023-08-03T02:37:00Z bootstop: Host is powering off

The log was printed in a few seconds after I shutdown from command line using esxcli system shutdown command.

But how do I get a notification when the above line is printed in the vmksummary.log that I can catch in a script/function and do some clean up

How do I use esxcli to get a signal when that 'Host is powering off' is written to the log file.

Thanks again,

[Sachchidanand]

I am not sure if any direct method is available for the same, but in a different way you can can setup syslog and sync your log files to remote syslog server. On syslog server write a bash script for getting this specific message and trigger an alert as soon as this message appears.

Regards,

Sachchidanand 

[dee2]

Hi Sachchidanand,

Thanks for the reply. That's what I was thinking, But how do I do it,
On esxi host I have to keep tailing the log messages in a continuous loop and look for the lines like "powering off" .

someething. like

If I tail in one shell and give shutdown command in a different shell I see this log message in the shell that is tailing
"2023-08-03T02:37:00Z bootstop: Host is powering off"

SO I wrote this script.

while [[ true ]] ; do

echo "Looking for poweroff"

signal=$(tail -n0 -f /var/log/vmksummary.log | grep -i "powering off")

echo "$signal"

if [[ "${signal}" ]]; then

// trigger alert

fi

sleep 1;

done

But tail is a blocking call. it does not return any value in the script above.

Also this makes the script heavy on cpu cycles as it is in continuous polling mode.

Is there any way like subscription or registering for an event alert. and not blocking

Thank you . would appreciate any ideas.

Deepali

[Sachchidanand]

I don't think there is a direct alert on ESXi but if you are managing your hosts via vCenter, you can configure alarm based on your requirement like host shut down and there you can send mail, snmp traps or write script(see screenshot).

Otherwise, syslog is the another way where you can use cron job to continuosly monitor your logs and execute alert as soon as you get the specific message.

Regards,

Sachchidanand