How to troubleshoot unplanned host restart - Initiated through DCUI

[mquast]

Hello.

We had a restart of one host in a cluster with threee hosts running 6.0.0 (4600944) in HA mode.

Warning on the host: "

The host is being rebooted through the Direct

Console User Interface (DCUI). Please consult

ESXi Embedded and vCenter Server Setup Guide

or follow the Ask VMware link for more

information.

warning

24.07.2017 08:21:27

"

Info	24.07.2017 08:22:20	VSPHERE.LOCAL\vpxd-extension-9324dfa0-4535-11e7-be07-0050568348d4Benutzer VSPHERE.LOCAL\vpxd-extension-9324dfa0-4535-11e7-be07-0050568348d4@127.0.0.1 als VMware vim-java 1.0 angemeldet
Info	24.07.2017 08:21:52	vCenter gestartet.

How to troubleshoot this restart ? I looked into the vmware-vpx\vpxd.log and this looks to me as if theres was new vCenter Agent installed on the host.

Was this restart initiated through the vCenter Server?

Thanks.

Michael

[vijayrana968]

You can check /var/log/vmksummary.log file to find out reboot events.

[mquast]

Thanks, but theres nothing to find in this log.

2017-07-24T06:27:30Z bootstop: Host has booted

2017-07-24T07:00:02Z heartbeat: up 0d0h33m57s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4583248kB]] [[37643 sleep 0%max] [37648 sh 0%max] [37651 python 0%max]]

2017-07-24T08:00:01Z heartbeat: up 0d1h33m56s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4583268kB]] [[41732 sleep 0%max] [41734 sh 0%max] [41736 python 0%max]]

2017-07-24T09:00:01Z heartbeat: up 0d2h33m57s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4583280kB]] [[44241 sleep 0%max] [44243 sh 0%max] [44245 python 0%max]]

2017-07-24T10:00:02Z heartbeat: up 0d3h33m57s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4583260kB]] [[46788 sleep 0%max] [46793 sh 0%max] [46795 python 0%max]]

2017-07-24T11:00:01Z heartbeat: up 0d4h33m56s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4646768kB]] [[49309 sleep 0%max] [49320 sh 0%max] [49322 python 0%max]]

2017-07-24T12:00:01Z heartbeat: up 0d5h33m56s, 4 VMs; [[36810 vmx 2097152kB] [36836 vmx 4194304kB] [36762 vmx 4646848kB]] [[51834 sleep 0%max] [51841 sh 0%max] [51843 python 0%max]]

[dekoshal]

Check this vmware kb to determine why an esxi host shutdown or restarted.

Determining why an ESXi/ESX host was powered off or restarted (1019238) | VMware KB

If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.

Best Regards,

Deepak Koshal

CNE|CLA|CWMA|VCP4|VCP5|CCAH

[mquast]

Hi.

I checked the logs, but found nothing helpful. Nevertheless maybe the vpxd.log from the vCenter is helpful. Unfortunatley in german. Maybe somebody can help? Problem startet at beginning of log. Maybe because AD services were down or AD services were down because of problem with ESXi host. Don't know yet.

Thanks,

Michael

[dekoshal]

This log looks like after the reboot of esxi server.

look into var/run/log where you will find the archived files. vmksummary and vmkernel log files will provide you necessary info.

If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.

Best Regards,

Deepak Koshal

[mquast]

Hey,

thank you. I already looked into these files and I think they are also starting after the restart.

Michael

[dekoshal]

Yes, these logs also looks like after the reboot.

[mquast]

Hi.

Any other place where I may find the reason for the restart?

What do setup that next time I have proper logs? Setup an external syslog server and point all three hosts to that syslog server?

Michael

[vijayrana968]

If esxi rebooted due to reason other than VMkernel, you won't find these logs after reboot. These logs will clear. Please refer to below KB for logs persistent location config.

Determining why an ESXi/ESX host was powered off or restarted (1019238) | VMware KB  

ESXi 4.x/5.x/6.x

1. To determine the reason for abrupt shut down or reboot of a VMware ESXi host:
   
   Note: By default, VMware ESXi logs do not persist upon a reboot. If a VMware ESXi host experiences an abrupt reboot due to reasons other than a VMkernel error, the logs do not persist and you do not have access to the logs prior to the reboot to determine the cause. The steps in this section assume that the VMware ESXi host is configured to redirect the logs to a location where the logs persist. For more information on how to configure a VMware ESXi host to redirect the logs to an alternate location, see Configure Syslog on ESXi Hosts in the Basic Administration Guide for your version of ESXi.
   
   
   1. If the ESXi host is currently turned off, turn the host back on.
      
      
   2. Ensure that there are no hardware lights that may indicate a hardware issue. For more information, engage the hardware vendor.
      
      
   3. Determine where the logs are being redirected to:
      
      
      1. Open vSphere Client.
      2. Connect to the ESXi host or vCenter Server managing the ESXi host.
      3. Provide the credentials of an administrative user.
      4. Select the ESXi host in the Inventory.
      5. Click the Configuration tab.
      6. Click Advanced Settings.
      7. In the Advanced Settings dialog, verify the location where the log files are being redirected:
         
         Note: If either of these settings are not properly configured, then logs do not persist upon a reboot and may limit the amount of information that can be gathered for troubleshooting.
         
         
         • Syslog > Local > Syslog.Local.DatastorePath contains the location of the logs if they are redirected to a VMFS volume.
         • Syslog > Remote > Syslog.Remote.Hostname contains the IP address or hostname of the syslog server that houses the logs for this host.
           
           
   4. Navigate to the location of the log files, and based on the modified date of the files, open the log file using your preferred editor.