Logs for Deleted VM

[bradyrandolph]

Hi,

We recently had a VM that was mysteriously deleted, what logs can I check to find out how and when this was removed?

Thanks in advance,

Brady

[esiebert7625]

In VirtualCenter look at the Tasks and Events for the ESX server, there should be a task for Destroy Virtual Machine and by what user. Sort on the Name column to make it easier to find the "Destroy Virtual Machine" task.

[bradyrandolph]

That is exactly what I did. Any other logs on the ESX host I can search?

[esiebert7625]

hostd would be the log file to look at, I don't believe it will tell what user did it though. You migh thave to also check the authentication logs.

• You can check several log files on the ESX server based on the problem you are experiencing, these include:

o Vmkernel - /var/log/vmkernel – records activities related to the virtual machines and ESX server

o Vmkernel Warnings - /var/log/vmkwarning – records activities with the virtual machines

o Vmkernel Summary - /var/log/vmksummary - Used to determine uptime and availability statistics for ESX Server; human-readable summary found in /var/log/vmksummary.txt

o ESX Server host agent log - /var/log/vmware/hostd.log - Contains information on the agent that manages and configures the ESX Server host and its virtual machines (Search the file date/time stamps to find the log file it is currently outputting to.)

o Service Console - /var/log/messages - Contain all general log messages used to troubleshoot virtual machines or ESX Server

o Web Access - /var/log/vmware/webAccess - Records information on Web-based access to ESX Server

o Authentication log - /var/log/secure - Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.

o VirtualCenter agent - /var/log/vmware/vpx - Contains information on the agent that communicates with VirtualCenter

o Virtual Machines - The same directory as the affected virtual machine’s configuration files; named vmware.log - Contain information when a virtual machine crashes or ends abnormally

[kharbin]

ssh into the host. su to root.

cd /var/log/vmware

grep "Destroy VM complete" hostd*.log

If done through VC, it will show here and return which hostd log file it was found in. Open the file with an editor and look for "Destroy VM called", this will show you the day/time the event happened. Also, make sure its the right VM, its .vmx is listed in the messages.

If there is no record of the delete here, then cd to /root

grep rm .bash_history

This will return any remove commands issued by the root user. Won't tell you when, just that it happened.

If not here, check each user.

cd /home/username

grep rm .bash_history

Do for each users home directory.

Also check /var/log/secure for something similar below for evidence of ssh login

Aug 10 15:36:12 host1 xinetd\[851]: START: vmware-authd pid=14039 from=192.168.200.187 (obviously the IP range will be different)

You can also run the command 'last'

This tells you everyone thats logged on in the last few weeks.

Hope it helps

Ken Harbin

www.esXpress.com

[esiebert7625]

Nice detailed info Ken!

[MarshKY]

I had this happen once in the Horizon View. The number of virtual machines specified in the View Administrator was less than the actual number of users. This is because the number was decreased in the View Administrator but all of the VMs had a user assigned to them. After I unassigned a user, it automatically deleted the VM. Not sure if this is the case but hope it helps