6 Replies Latest reply on Jun 7, 2017 7:34 AM by MarshKY

    Logs for Deleted VM

    bradyrandolph Enthusiast

      Hi,

       

      We recently had a VM that was mysteriously deleted, what logs can I check to find out how and when this was removed?

       

      Thanks in advance,

       

      Brady

        • 1. Re: Logs for Deleted VM
          esiebert7625 Guru
          vExpert

          In VirtualCenter look at the Tasks and Events for the ESX server, there should be a task for Destroy Virtual Machine and by what user. Sort on the Name column to make it easier to find the "Destroy Virtual Machine" task.

          • 2. Re: Logs for Deleted VM
            bradyrandolph Enthusiast

            That is exactly what I did.  Any other logs on the ESX host I can search?

            • 3. Re: Logs for Deleted VM
              esiebert7625 Guru
              vExpert

              hostd would be the log file to look at, I don't believe it will tell what user did it though. You migh thave to also check the authentication logs.

               

              •     You can check several log files on the ESX server based on the problem you are experiencing, these include:

              o     Vmkernel - /var/log/vmkernel – records activities related to the virtual machines and ESX server

              o     Vmkernel Warnings - /var/log/vmkwarning – records activities with the virtual machines

              o     Vmkernel Summary - /var/log/vmksummary - Used to determine uptime and availability statistics for ESX Server; human-readable summary found in /var/log/vmksummary.txt

              o     ESX Server host agent log - /var/log/vmware/hostd.log - Contains information on the agent that manages and configures the ESX Server host and its virtual machines (Search the file date/time stamps to find the log file it is currently outputting to.)

              o     Service Console - /var/log/messages - Contain all general log messages used to troubleshoot virtual machines or ESX Server

              o     Web Access - /var/log/vmware/webAccess - Records information on Web-based access to ESX Server

              o     Authentication log - /var/log/secure - Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.

              o     VirtualCenter agent - /var/log/vmware/vpx - Contains information on the agent that communicates with VirtualCenter

              o     Virtual Machines - The same directory as the affected virtual machine’s configuration files; named vmware.log - Contain information when a virtual machine crashes or ends abnormally

              • 4. Re: Logs for Deleted VM
                kharbin Master

                ssh into the host. su to root.

                cd /var/log/vmware

                grep "Destroy VM complete" hostd*.log

                 

                If done through VC, it will show here and return which hostd log file it was found in. Open the file with an editor and look for "Destroy VM called", this will show you the day/time the event happened. Also, make sure its the right VM, its .vmx is listed in the messages.

                 

                If there is no record of the delete here, then cd to /root

                 

                grep rm .bash_history

                 

                This will return any remove commands issued by the root user. Won't tell you when, just that it happened.

                 

                If not here, check each user.

                 

                cd /home/username

                grep rm .bash_history

                 

                Do for each users home directory.

                 

                Also check /var/log/secure for something similar below for evidence of ssh login

                 

                Aug 10 15:36:12 host1 xinetd\[851]: START: vmware-authd pid=14039 from=192.168.200.187 (obviously the IP range will be different)

                 

                You can also run the command 'last'

                 

                This tells you everyone thats logged on in the last few weeks.

                 

                 

                Hope it helps

                 

                Ken Harbin

                www.esXpress.com

                • 5. Re: Logs for Deleted VM
                  esiebert7625 Guru
                  vExpert

                  Nice detailed info Ken!

                  • 6. Re: Logs for Deleted VM
                    MarshKY Lurker

                    I had this happen once in the Horizon View. The number of virtual machines specified in the View Administrator was less than the actual number of users. This is because the number was decreased in the View Administrator but all of the VMs had a user assigned to them. After I unassigned a user, it automatically deleted the VM. Not sure if this is the case but hope it helps