ESXi Logging

[1an3]

Hello

I have a strange issue which I cannot get to the bottom of, and I am struggling to get support to help.

A few weeks ago, I had an issue with a host randomly disconnecting and reconnecting in vcenter. I could not see a cause, so I logged a support call. As part of that, we discovered that the vpxa.log had not updated since August 2016

Logs go to [local datastore]\.locker\

There is free space on the datastore.

A reboot of the host fixed the issue, but temporarily. A restart of the mgmt agents from the DCUI does not resolve the issue.

The vpxa.log on that host has stopped updating on 18th Feb

There are no errors or alarms logged against the host in VC.

The running of the host is unaffected.

Various hosts are affected, mostly ESXI6 U2 (slightly different patch levels)

I am at a loss, can anyone point me in the direction of any kb articles?

You can use this PowerCLI to check the date/time of the last line in vpxa.log for each host in your cluster.

Get-Cluster | Get-VMHost | Get-Log -Key vpxa |Select @{N='VMHost';E={$_.Host}},@{N='Log';E={$_.Entries[-1]}} | Format-Table -Autosize

[msripada]

Hello 1an3,

May I know if the ESXi hosts are configured with logging to a specific datastore or is it local datastore?

How about the other ESXi hosts, are all of them (issue specific hosts) are logging to a single datastore ??

Thanks,

MS

[Rahim_from_PSPL]

Try setting up syslog collector on your vCenter; and keep last 5 logs retention.

I remember there were some bugs in vsphere5.5 for logs.

Syslog collector is always a best practice for audit purpose. Do not configure dump collector until you need memory dump which is useless IMO.

[iamamit]

You should use common/shared datastore to store ESXi Logs instead of local one. I remember, there was similar kind of issue with my VM infra but did not see it repeated after I set the sys log collector to common datastore.

You can check the below KB for step by step process of setting up sys log collector using vSphere Client.

http://www.techiessphere.com/2016/12/configuration-issues-system-logs-on-non-persistent-storage.html

Thanks,

Amit

[1an3]

They are each configured to use their own local datastore. Esxi is installed to SD card, logs etc go to a datastore on local HDD pair. All VM storage is on SAN.

[1an3]

I have log insight appliance as a collector but this is the local vpxa logs (esxi 6)

No matter what the syslog collector, there should still be local logs, right?

[1an3]

> You should use common/shared datastore to store ESXi Logs instead of local one.

Why is this?

The only issue you see is if you boot a host with no 'persistent' storage - creating a local datastore means logs write there by default.

And for a while the logs write there, but for some reason they have now stopped.

[iamamit]

There are multiple reasons, but the most valid reasons are (as per my previous experiences):

1. When your ESXi crashes or corrupted due to any reasons, you will lose all the logs saved on local datastore. So, its good idea to have shared datastore so that in case something happens to your ESXi host, at-least you would be having some logs to investigate.

2. Some time local datastore misbehave for external write operations like saving logs on local datastore, its good to switch to shared datastore in this case.

3. If your ESXi host is installed on SD card, there many be available space issue in future which may lead to unwanted interruptions to your ESXi host. So, if you would be having the logs on shared datastore you even migrate some VMs to other datastores in space crunch situation to make sure at least you have enough available space for saving your ESXi logs.

Hope its helpful..

Thanks,

Amit

[1an3]

>When your ESXi crashes or corrupted due to any reasons, you will lose all the logs saved on local datastore.

No you won't - they're on *disk* not in memory. Logfiles persist across reboots.

The local datastores have plenty of space (400GB+) so space on these datastores is not an issue - they are not used for running VMs.

[iamamit]

>>When your ESXi crashes or corrupted due to any reasons, you will lose all the logs saved on local datastore

I mean, the complete crash or os corrupted situation where even after reboot you will not get your ESXi up.. In my case, I had lost everything (OS, Logs etc..). I had to rebuild the OS after that..

[1an3]

You will still have logs if you syslog off to syslog collector (or similar) working correctly....