One of my vm was migrated from host a (esx 3.5) to host b (esx 3.5). When I checked the vm's (RHEL5 32 bit) message log, all the services seemed to have been restart without the vm having been powerrd off, restarted, shut down or lost power. I have not seen this before... what caused this and how could it be prevented.
The is from the esx side:
DRS migrated studyabroad from esx-7sc to esx-6sc in cluster CLS Cluster in X
info 1/14/2010 12:44:40 PM Changed resource allocation for studyabroad
info 1/14/2010 12:44:23 PM Migrating studyabroad off host esx-7sc in X
info 1/14/2010 12:44:04 PM Migrating studyabroad from esx-7sc to esx-6sc in X
info 1/14/2010 12:44:04 PM Task: Migrate Virtual Machine
From the vm
Jan 14 12:40:52 study-abroad syslogd 1.4.1: restart.
Jan 14 12:40:52 study-abroad kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 14 12:40:52 study-abroad kernel: Linux version 2.6.18-164.10.1.el5 (mockbuild@hs20-bc1-7.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed Dec 30 18:35:37 EST 2009
Jan 14 12:40:52 study-abroad kernel: BIOS-provided physical RAM map:
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 0000000000010000 - 000000000009f800 (usable)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 0000000000100000 - 000000003e6f0000 (usable)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 000000003e6f0000 - 000000003e6ff000 (ACPI data)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 000000003e6ff000 - 000000003e700000 (ACPI NVS)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 000000003e700000 - 000000003e800000 (usable)
Jan 14 12:40:52 study-abroad kernel: BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
Jan 14 12:40:53 study-abroad kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 14 12:40:53 study-abroad kernel: BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
Jan 14 12:40:53 study-abroad kernel: 104MB HIGHMEM available.
Jan 14 12:40:53 study-abroad kernel: 896MB LOWMEM available.
Jan 14 12:40:53 study-abroad kernel: found SMP MP-table at 000f6cd0
Jan 14 12:40:53 study-abroad kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Jan 14 12:40:53 study-abroad kernel: disabling kdump
Jan 14 12:40:53 study-abroad kernel: Using x86 segment limits to approximate NX protection
Jan 14 12:40:53 study-abroad kernel: DMI present.
Jan 14 12:40:53 study-abroad kernel: Using APIC driver default
Jan 14 12:40:53 study-abroad kernel: ACPI: PM-Timer IO Port: 0x1008
Jan 14 12:40:53 study-abroad kernel: ACPI: LAPIC (acpi_id0x00 lapic_id0x00 enabled)
Jan 14 12:40:53 study-abroad kernel: Processor #0 6:15 APIC version 17
Jan 14 12:40:53 study-abroad kernel: ACPI: LAPIC_NMI (acpi_id0x00 high edge lint0x1)
Jan 14 12:40:53 study-abroad kernel: ACPI: IOAPIC (id0x01 address0xfec00000 gsi_base[0])
Jan 14 12:40:53 study-abroad kernel: IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
Jan 14 12:40:53 study-abroad kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Jan 14 12:40:53 study-abroad kernel: Enabling APIC mode: Flat. Using 1 I/O APICs
Jan 14 12:40:53 study-abroad kernel: Using ACPI (MADT) for SMP configuration information
Jan 14 12:40:53 study-abroad kernel: Allocating PCI resources starting at 40000000 (gap: 3e800000:c0400000)
Jan 14 12:40:53 study-abroad kernel: TSC: Frequency read from the hypervisor
Jan 14 12:40:53 study-abroad kernel: Detected 3000.106 MHz processor.
Jan 14 12:40:53 study-abroad kernel: Built 1 zonelists. Total pages: 256000
Jan 14 12:40:53 study-abroad kernel: Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Jan 14 12:40:53 study-abroad kernel: Enabling fast FPU save and restore... done.
Jan 14 12:40:53 study-abroad kernel: Enabling unmasked SIMD FPU exception support... done.
Jan 14 12:40:53 study-abroad kernel: Initializing CPU#0
Jan 14 12:40:53 study-abroad kernel: CPU 0 irqstacks, hard=c0762000 soft=c0742000
Jan 14 12:40:53 study-abroad kernel: PID hash table entries: 4096 (order: 12, 16384 bytes)
Jan 14 12:40:53 study-abroad kernel: Console: colour VGA+ 80x25
Jan 14 12:40:53 study-abroad kernel: Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Jan 14 12:40:53 study-abroad mcstransd: mcstransd starting
Jan 14 12:40:53 study-abroad kernel: Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Jan 14 12:40:53 study-abroad kernel: Memory: 1006976k/1024000k available (2160k kernel code, 16124k reserved, 901k data, 228k init, 106432k highmem)
Jan 14 12:40:53 study-abroad kernel: Checking if this processor honours the WP bit even in supervisor mode... Ok.
Jan 14 12:40:53 study-abroad kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 6000.21 BogoMIPS (lpj=3000106)
Jan 14 12:40:53 study-abroad kernel: Security Framework v1.0.0 initialized
Jan 14 12:40:53 study-abroad kernel: SELinux: Initializing.
Jan 14 12:40:53 study-abroad kernel: selinux_register_security: Registering secondary module capability
Jan 14 12:40:53 study-abroad mcstransd: mcstransd initialized
Jan 14 12:40:53 study-abroad kernel: Capability LSM initialized as secondary
Jan 14 12:40:53 study-abroad kernel: Mount-cache hash table entries: 512
Jan 14 12:40:53 study-abroad kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 14 12:40:53 study-abroad kernel: CPU: L2 cache: 4096K
Jan 14 12:40:53 study-abroad kernel: Intel machine check architecture supported.
Jan 14 12:40:53 study-abroad kernel: Intel machine check reporting enabled on CPU#0.
Jan 14 12:40:53 study-abroad kernel: Checking 'hlt' instruction... OK.
Jan 14 12:40:53 study-abroad kernel: SMP alternatives: switching to UP code
Jan 14 12:40:53 study-abroad kernel: Freeing SMP alternatives: 14k freed
Jan 14 12:40:53 study-abroad kernel: ACPI: Core revision 20060707
Jan 14 12:40:53 study-abroad kernel: CPU0: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz stepping 08
Jan 14 12:40:53 study-abroad kernel: Total of 1 processors activated (6000.21 BogoMIPS).
Jan 14 12:40:53 study-abroad kernel: ENABLING IO-APIC IRQs
Jan 14 12:40:53 study-abroad kernel: ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 14 12:40:53 study-abroad kernel: Using local APIC timer interrupts.
Jan 14 12:40:53 study-abroad kernel: WARNING calibrate_APIC_clock: the APIC timer calibration may be wrong.
Jan 14 12:40:53 study-abroad kernel: Brought up 1 CPUs
Jan 14 12:40:53 study-abroad kernel: checking if image is initramfs... it is
Jan 14 12:40:53 study-abroad kernel: Freeing initrd memory: 3143k freed
Jan 14 12:40:53 study-abroad kernel: NET: Registered protocol family 16
Jan 14 12:40:53 study-abroad kernel: ACPI: bus type pci registered
Jan 14 12:40:53 study-abroad kernel: PCI: PCI BIOS revision 2.10 entry at 0xfd9a0, last bus=1
Jan 14 12:40:53 study-abroad kernel: PCI: Using configuration type 1
Jan 14 12:40:53 study-abroad kernel: Setting up standard PCI resources
Jan 14 12:40:53 study-abroad kernel: ACPI: Interpreter enabled
Jan 14 12:40:53 study-abroad kernel: ACPI: Using IOAPIC for interrupt routing
Jan 14 12:40:53 study-abroad kernel: ACPI: No dock devices found.
Jan 14 12:40:53 study-abroad kernel: ACPI: PCI Root Bridge PCI0 (0000:00)
Jan 14 12:40:53 study-abroad kernel: PCI quirk: region 1000-103f claimed by PIIX4 ACPI
Jan 14 12:40:53 study-abroad kernel: PCI quirk: region 1040-104f claimed by PIIX4 SMB
Jan 14 12:40:53 study-abroad kernel: ACPI: PCI Interrupt Link LNKA (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.
Jan 14 12:40:53 study-abroad kernel: ACPI: PCI Interrupt Link LNKB (IRQs 3 4 5 6 7 *9 10 11 14 15)
Jan 14 12:40:53 study-abroad kernel: ACPI: PCI Interrupt Link LNKC (IRQs 3 4 5 6 7 9 10 *11 14 15)
Jan 14 12:40:53 study-abroad kernel: ACPI: PCI Interrupt Link LNKD (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.
Jan 14 12:40:53 study-abroad kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Jan 14 12:40:53 study-abroad kernel: pnp: PnP ACPI init
Jan 14 12:40:53 study-abroad kernel: pnp: PnP ACPI: found 12 devices
Jan 14 12:40:53 study-abroad kernel: usbcore: registered new driver usbfs
Jan 14 12:40:53 study-abroad kernel: usbcore: registered new driver hub
Jan 14 12:40:53 study-abroad kernel: PCI: Using ACPI for IRQ routing
Jan 14 12:40:53 study-abroad kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
Jan 14 12:40:53 study-abroad kernel: NetLabel: Initializing
Jan 14 12:40:53 study-abroad kernel: NetLabel: domain hash size = 128
Jan 14 12:40:53 study-abroad kernel: NetLabel: protocols = UNLABELED CIPSOv4
Jan 14 12:40:53 study-abroad kernel: NetLabel: unlabeled traffic allowed by default
Jan 14 12:40:53 study-abroad kernel: PCI: Bridge: 0000:00:01.0
Jan 14 12:40:53 study-abroad kernel: IO window: disabled.
Jan 14 12:40:53 study-abroad kernel: MEM window: disabled.
Jan 14 12:40:53 study-abroad kernel: PREFETCH window: disabled.
Jan 14 12:40:53 study-abroad kernel: NET: Registered protocol family 2
Jan 14 12:40:53 study-abroad kernel: IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
Jan 14 12:40:53 study-abroad kernel: TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
Jan 14 12:40:53 study-abroad kernel: TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
Jan 14 12:40:53 study-abroad kernel: TCP: Hash tables configured (established 131072 bind 65536)
Jan 14 12:40:53 study-abroad kernel: TCP reno registered
Jan 14 12:40:53 study-abroad kernel: Simple Boot Flag at 0x36 set to 0x80
Jan 14 12:40:53 study-abroad kernel: apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
Jan 14 12:40:53 study-abroad kernel: apm: overridden by ACPI.
Jan 14 12:40:53 study-abroad kernel: audit: initializing netlink socket (disabled)
Jan 14 12:40:53 study-abroad kernel: type=2000 audit(1263490801.693:1): initialized
Jan 14 12:40:53 study-abroad kernel: highmem bounce pool size: 64 pages
Jan 14 12:40:53 study-abroad kernel: Total HugeTLB memory allocated, 0
Jan 14 12:40:53 study-abroad kernel: VFS: Disk quotas dquot_6.5.1
Jan 14 12:40:53 study-abroad kernel: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Jan 14 12:40:53 study-abroad kernel: Initializing Cryptographic API
Jan 14 12:40:53 study-abroad kernel: alg: No test for crc32c (crc32c-generic)
Jan 14 12:40:53 study-abroad rpc.statd1790: Version 1.0.9 Starting
Jan 14 12:40:53 study-abroad kernel: ksign: Installing public key data
Jan 14 12:40:53 study-abroad kernel: Loading keyring
Jan 14 12:40:53 study-abroad kernel: - Added public key 1A9118D63F4F2E5A
Jan 14 12:40:53 study-abroad kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 14 12:40:53 study-abroad kernel: io scheduler noop registered
Jan 14 12:40:53 study-abroad kernel: io scheduler anticipatory registered
Jan 14 12:40:53 study-abroad kernel: io scheduler deadline registered
Jan 14 12:40:53 study-abroad kernel: io scheduler cfq registered (default)
Jan 14 12:40:53 study-abroad kernel: Limiting direct PCI/PCI transfers.
Jan 14 12:40:53 study-abroad kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 14 12:40:53 study-abroad kernel: ACPI: Processor CPU0 (supports 8 throttling states)
Jan 14 12:40:53 study-abroad kernel: Real Time Clock Driver v1.12ac
Jan 14 12:40:53 study-abroad kernel: Non-volatile memory driver v1.2
Jan 14 12:40:53 study-abroad kernel: Linux agpgart interface v0.101 (c) Dave Jones
Jan 14 12:40:53 study-abroad kernel: agpgart: Detected an Intel 440BX Chipset.
Jan 14 12:40:53 study-abroad kernel: agpgart: AGP aperture is 256M @ 0x0
Jan 14 12:40:53 study-abroad kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
Jan 14 12:40:53 study-abroad kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 12:40:53 study-abroad kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 12:40:53 study-abroad kernel: 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 12:40:53 study-abroad kernel: 00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 12:40:53 study-abroad kernel: brd: module loaded
Jan 14 12:40:53 study-abroad kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
Jan 14 12:40:53 study-abroad kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Jan 14 12:40:54 study-abroad kernel: PIIX4: IDE controller at PCI slot 0000:00:07.1
Jan 14 12:40:54 study-abroad kernel: PIIX4: chipset revision 1
Jan 14 12:40:54 study-abroad kernel: PIIX4: not 100% native mode: will probe irqs later
Jan 14 12:40:54 study-abroad kernel: ide0: BM-DMA at 0x1050-0x1057, BIOS settings: hda:DMA, hdb:pio
Jan 14 12:40:54 study-abroad kernel: hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive
...
Jan 14 12:41:01 study-abroad gpm2232: *** info http://startup.c(95):
Jan 14 12:41:01 study-abroad gpm2232: Started gpm successfully. Entered daemon mode.
Jan 14 12:41:14 study-abroad rhnsd2604: Red Hat Network Services Daemon starting up.
Jan 14 12:41:20 study-abroad smartd2691: smartd version 5.38 i686-redhat-linux-gnu Copyright (C) 2002-8 Bruce Allen
Jan 14 12:41:20 study-abroad smartd2691: Home page is http://smartmontools.sourceforge.net/
Jan 14 12:41:20 study-abroad smartd2691: Opened configuration file /etc/smartd.conf
Jan 14 12:41:20 study-abroad smartd2691: Configuration file /etc/smartd.conf parsed.
Jan 14 12:41:20 study-abroad smartd2691: Device: /dev/sda, opened
Jan 14 12:41:20 study-abroad smartd2691: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features
Jan 14 12:41:20 study-abroad smartd2691: Unable to register SCSI device /dev/sda at line 32 of file /etc/smartd.conf
Jan 14 12:41:20 study-abroad smartd2691: Device /dev/sda not available
Jan 14 12:41:20 study-abroad smartd2691: Monitoring 0 ATA and 0 SCSI devices
Jan 14 12:41:20 study-abroad smartd2693: smartd has fork()ed into background mode. New PID=2693.
Is the time accurate? Because VMotion started 4 minutes later according to logs.
---
MCSA, MCTS, VCP, VMware vExpert '2009
Is the time accurate? Because VMotion started 4 minutes later according to logs.
---
MCSA, MCTS, VCP, VMware vExpert '2009
The time stamps are correct.
I found this in the logs:
Jan 14 12:39:38 esx-7sc vmkernel: 5:14:36:21.178 cpu0:1289)World: vm 1331: 901: Starting world vmm0:studyabroad with flags 8
Jan 14 12:39:38 esx-7sc vmkernel: 5:14:36:21.179 cpu0:1289)Sched: vm 1331: 5333: adding 'vmm0:studyabroad': group 'host/user/pool0/pool2': cpu: shares=-3 min=500 minLimit=-1 max=-1
My question is:
is this a vm reboot and who what caused it?
VM usually reports high CPU and memory utilization a few minutes after startup, so it looks like VM restart caused VMotion.
Investigate VM and ESX logs further. I suppose from 12:35:00 to 12:40:00
---
MCSA, MCTS, VCP, VMware vExpert '2009
My question then is how do I find out what causes the vm to restart?
Is there somewhere in the vm that record the restart, reboot or power off of the vm by the vm user or VI3?
There is no universal answer. Investigate your logs.
---
MCSA, MCTS, VCP, VMware vExpert '2009