VMware Cloud Community
RoFz
Contributor
Contributor

ESXi 4.0 freeze on PowerEdge T110

Hello,

Our ESXi 4.0 froze last night around 1 AM (GMT -3). Unfortunately the server was rebooted this morning by another person from the IT staff who did not take time to analyze the LEDs of the server or diagnose the situation any further. So far the only relevant piece of information we have is ESX's messages log (which is copied below):

Last log entry is from Apr 25 04:47:56. System restarted at Apr 25 12:03:47.

Apr 25 04:00:57 Hostd: [2013-04-25 04:00:57.591 43977B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:01:01 crond[5169]: USER root pid 19111095 cmd /sbin/auto-backup.sh #first minute of every hour (run every hour)
Apr 25 04:02:28 Hostd: [2013-04-25 04:02:28.082 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:02:41 Hostd: [2013-04-25 04:02:41.141 43977B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x443422d0] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
Apr 25 04:02:41 Hostd: [2013-04-25 04:02:41.141 43977B90 verbose 'Vmomi'] Arg version: "26"
Apr 25 04:02:41 Hostd: [2013-04-25 04:02:41.141 43977B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
Apr 25 04:02:41 Hostd: [2013-04-25 04:02:41.141 43977B90 info 'Vmomi'] Result: (vmodl.fault.RequestCanceled) {    dynamicType = <unset>,     faultCause = (vmodl.MethodFault) null,     msg = "",  }
Apr 25 04:02:41 Hostd: [2013-04-25 04:02:41.142 437F7DC0 error 'SoapAdapter'] PendingRequest: HTTP Transaction failed, closing connection: N7Vmacore15SystemExceptionE(Connection reset by peer)
Apr 25 04:02:56 Hostd: [2013-04-25 04:02:56.700 43A3AB90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:03:58 Hostd: [2013-04-25 04:03:58.565 43A3AB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:05:16 Hostd: [2013-04-25 04:05:16.307 43AFDB90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/iac01/iac01.vmx'] Actual VM overhead: 174080000 bytes
Apr 25 04:05:16 Hostd: [2013-04-25 04:05:16.307 43AFDB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:05:30 Hostd: [2013-04-25 04:05:30.041 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:07:00 Hostd: [2013-04-25 04:07:00.531 43977B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:07:36 Hostd: [2013-04-25 04:07:36.317 43A3AB90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 81809408 bytes
Apr 25 04:07:36 Hostd: [2013-04-25 04:07:36.317 43A3AB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:07:45 Hostd: [2013-04-25 04:07:45.699 439B8B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6033MB to 8139MHz/6032MB
Apr 25 04:07:56 Hostd: [2013-04-25 04:07:56.702 43977B90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:08:31 Hostd: [2013-04-25 04:08:31.011 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:08:45 Hostd: [2013-04-25 04:08:45.700 437F7DC0 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6032MB to 8139MHz/6033MB
Apr 25 04:10:01 Hostd: [2013-04-25 04:10:01.495 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:11:16 Hostd: [2013-04-25 04:11:16.335 43977B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/iac01/iac01.vmx'] Actual VM overhead: 173547520 bytes
Apr 25 04:11:16 Hostd: [2013-04-25 04:11:16.336 43977B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:11:32 Hostd: [2013-04-25 04:11:32.740 43AFDB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:11:56 Hostd: [2013-04-25 04:11:56.339 43977B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/iac01/iac01.vmx'] Actual VM overhead: 174071808 bytes
Apr 25 04:11:56 Hostd: [2013-04-25 04:11:56.339 43977B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:12:36 Hostd: [2013-04-25 04:12:36.341 439B8B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 80752640 bytes
Apr 25 04:12:36 Hostd: [2013-04-25 04:12:36.342 439B8B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:12:56 Hostd: [2013-04-25 04:12:56.342 439F9B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 81276928 bytes
Apr 25 04:12:56 Hostd: [2013-04-25 04:12:56.343 439F9B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:12:56 Hostd: [2013-04-25 04:12:56.702 43ABCB90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:13:03 Hostd: [2013-04-25 04:13:03.231 43977B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:14:33 Hostd: [2013-04-25 04:14:33.721 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:16:04 Hostd: [2013-04-25 04:16:04.208 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:17:35 Hostd: [2013-04-25 04:17:35.225 43A7BB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:17:56 Hostd: [2013-04-25 04:17:56.703 43A3AB90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:19:05 Hostd: [2013-04-25 04:19:05.703 43A7BB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:20:36 Hostd: [2013-04-25 04:20:36.182 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:22:06 Hostd: [2013-04-25 04:22:06.672 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:22:41 Hostd: [2013-04-25 04:22:41.367 43A7BB90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x44104688] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
Apr 25 04:22:41 Hostd: [2013-04-25 04:22:41.367 43A7BB90 verbose 'Vmomi'] Arg version: "26"
Apr 25 04:22:41 Hostd: [2013-04-25 04:22:41.367 43A7BB90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
Apr 25 04:22:41 Hostd: [2013-04-25 04:22:41.368 43A7BB90 info 'Vmomi'] Result: (vmodl.fault.RequestCanceled) {    dynamicType = <unset>,     faultCause = (vmodl.MethodFault) null,     msg = "",  }
Apr 25 04:22:41 Hostd: [2013-04-25 04:22:41.368 43AFDB90 error 'SoapAdapter'] PendingRequest: HTTP Transaction failed, closing connection: N7Vmacore15SystemExceptionE(Connection reset by peer)
Apr 25 04:22:56 Hostd: [2013-04-25 04:22:56.704 43AFDB90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:23:38 Hostd: [2013-04-25 04:23:38.466 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:25:08 Hostd: [2013-04-25 04:25:08.950 43AFDB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:26:39 Hostd: [2013-04-25 04:26:39.441 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:26:45 Hostd: [2013-04-25 04:26:45.712 43AFDB90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6033MB to 8139MHz/6032MB
Apr 25 04:27:45 Hostd: [2013-04-25 04:27:45.713 43A3AB90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6032MB to 8139MHz/6033MB
Apr 25 04:27:56 Hostd: [2013-04-25 04:27:56.704 439B8B90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:28:09 Hostd: [2013-04-25 04:28:09.928 43977B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:29:16 Hostd: [2013-04-25 04:29:16.415 43977B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 81801216 bytes
Apr 25 04:29:16 Hostd: [2013-04-25 04:29:16.415 43977B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:29:41 Hostd: [2013-04-25 04:29:41.419 437F7DC0 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:30:36 Hostd: [2013-04-25 04:30:36.422 43A3AB90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 80744448 bytes
Apr 25 04:30:36 Hostd: [2013-04-25 04:30:36.423 43A3AB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:31:11 Hostd: [2013-04-25 04:31:11.909 43977B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:31:16 Hostd: [2013-04-25 04:31:16.425 43A3AB90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/fw-2.0/fw-2.0.vmx'] Actual VM overhead: 81268736 bytes
Apr 25 04:31:16 Hostd: [2013-04-25 04:31:16.425 43A3AB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:32:42 Hostd: [2013-04-25 04:32:42.400 43A3AB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:32:56 Hostd: [2013-04-25 04:32:56.706 437F7DC0 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:34:12 Hostd: [2013-04-25 04:34:12.887 437F7DC0 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:35:44 Hostd: [2013-04-25 04:35:44.694 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:35:45 Hostd: [2013-04-25 04:35:45.721 43A7BB90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6033MB to 8139MHz/6032MB
Apr 25 04:36:45 Hostd: [2013-04-25 04:36:45.721 439B8B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6032MB to 8139MHz/6033MB
Apr 25 04:36:56 Hostd: [2013-04-25 04:36:56.453 43ABCB90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/iac01/iac01.vmx'] Actual VM overhead: 173015040 bytes
Apr 25 04:36:56 Hostd: [2013-04-25 04:36:56.453 43ABCB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:37:15 Hostd: [2013-04-25 04:37:15.182 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:37:16 Hostd: [2013-04-25 04:37:16.454 43977B90 verbose 'vm:/vmfs/volumes/4c333313-f745fb6c-b91d-001b215d617d/iac01/iac01.vmx'] Actual VM overhead: 173539328 bytes
Apr 25 04:37:16 Hostd: [2013-04-25 04:37:16.455 43977B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
Apr 25 04:37:56 Hostd: [2013-04-25 04:37:56.707 43AFDB90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:38:45 Hostd: [2013-04-25 04:38:45.673 439B8B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:40:16 Hostd: [2013-04-25 04:40:16.163 437F7DC0 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:41:47 Hostd: [2013-04-25 04:41:47.826 439F9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:42:41 Hostd: [2013-04-25 04:42:41.611 43977B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x44090428] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
Apr 25 04:42:41 Hostd: [2013-04-25 04:42:41.611 43977B90 verbose 'Vmomi'] Arg version: "26"
Apr 25 04:42:41 Hostd: [2013-04-25 04:42:41.612 43977B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
Apr 25 04:42:41 Hostd: [2013-04-25 04:42:41.612 43977B90 info 'Vmomi'] Result: (vmodl.fault.RequestCanceled) {    dynamicType = <unset>,     faultCause = (vmodl.MethodFault) null,     msg = "",  }
Apr 25 04:42:41 Hostd: [2013-04-25 04:42:41.612 439F9B90 error 'SoapAdapter'] PendingRequest: HTTP Transaction failed, closing connection: N7Vmacore15SystemExceptionE(Connection reset by peer)
Apr 25 04:42:56 Hostd: [2013-04-25 04:42:56.708 439B8B90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 04:43:18 Hostd: [2013-04-25 04:43:18.319 439B8B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:44:48 Hostd: [2013-04-25 04:44:48.811 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:46:19 Hostd: [2013-04-25 04:46:19.293 43ABCB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:46:45 Hostd: [2013-04-25 04:46:45.727 439F9B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6033MB to 8139MHz/6032MB
Apr 25 04:47:45 Hostd: [2013-04-25 04:47:45.727 43ABCB90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 8139MHz/6032MB to 8139MHz/6033MB
Apr 25 04:47:51 Hostd: [2013-04-25 04:47:51.193 43A7BB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Apr 25 04:47:56 Hostd: [2013-04-25 04:47:56.708 43977B90 verbose 'DvsManager'] PersistAllDvsInfo called
Apr 25 12:03:47 vmklogger: Successfully daemonized.
Apr 25 12:03:47 vmkernel: TSC: 0 cpu0:0)Init: 293: cpu 0: early measured tsc speed is 2393999718 Hz
Apr 25 12:03:47 vmkernel: TSC: 14280 cpu0:0)Cpu: 346: id1.version 106e5
Apr 25 12:03:47 vmkernel: TSC: 28904 cpu0:0)CPUIntel: 333: Turbo Mode enabled by the BIOS
Apr 25 12:03:47 vmkernel: TSC: 32368 cpu0:0)CPUIntel: 361: C1E enabled by the BIOS
Apr 25 12:03:47 vmkernel: TSC: 36268 cpu0:0)Cpu: 408: APIC ID mask: 0xff000000
Apr 25 12:03:47 vmkernel: TSC: 39380 cpu0:0)Cpu: 1057: initial APICID=0x0
Apr 25 12:03:47 vmkernel: TSC: 42728 cpu0:0)MicrocodeIntel: 513: Update signature 300000000, Platform ID 4000000000000.
Apr 25 12:03:47 vmkernel: TSC: 47868 cpu0:0)Cpu: 1304: 36 physical bits, 48 virtual bits
Apr 25 12:03:47 vmkernel: TSC: 51632 cpu0:0)Cpu: 1836: MTRR: cap=0xd08 with 8 variable MTRRs on BSP 0
Apr 25 12:03:47 vmkernel: TSC: 55828 cpu0:0)Cpu: 1872: MTRR: deftype=0xc00, enabled bit=1 and default type=0
Apr 25 12:03:47 vmkernel: TSC: 60440 cpu0:0)Cpu: 1591: Variable MTRR 0: start=0x0 mask=0xf80000800 end=0x0 type=0
Apr 25 12:03:47 vmkernel: TSC: 65192 cpu0:0)Cpu: 1591: Variable MTRR 1: start=0x0 mask=0xfc0000800 end=0x0 type=0
Apr 25 12:03:47 vmkernel: TSC: 69672 cpu0:0)Cpu: 1591: Variable MTRR 2: start=0x0 mask=0xf00000800 end=0x0 type=0
Apr 25 12:03:47 vmkernel: TSC: 73972 cpu0:0)Cpu: 1591: Variable MTRR 3: start=0x0 mask=0xfc0000800 end=0x0 type=0
Apr 25 12:03:47 vmkernel: TSC: 79476 cpu0:0)Cpu: 1744: start=0x0 type=0x606060606060606
Apr 25 12:03:47 vmkernel: TSC: 84844 cpu0:0)Cpu: 1744: start=0x80000 type=0x606060606060606
Apr 25 12:03:47 vmkernel: TSC: 88896 cpu0:0)Cpu: 1744: start=0xa0000 type=0x0

The symptons:

- guests stopped responding to ping

Other symptons are not available since it was manually power cycled by someone from the IT staff who did not conduct other diagnostics.

The data:

- VMware ESXi 4.0.0 build-261974

- Guest #1: CentOS 5.5 32 bit (2 GB RAM)

- Guest #2: pfSense 2.0.1 (128 MB RAM)

The hardware:

- Dell PowerEdge T110

- 8 GB RAM

- 2x250 GB SATA disks in RAID 1

- SAS 6iR Controller

- Intel Gigabit ET Dual Port

We are still waiting to shutdown the server again and check ESM logs, but until there we would like some insight on ESX's messages log file provided above.

Any help would be greatly appreciated.

0 Kudos
0 Replies