Monitor coredump, vm crashs

[onkelalbert]

Hi all

I have problems with one virtual machine. Every night at same time, the machine powering off itselfs. See the following logfile:

Dec 28 00:02:50.761: vcpu-1| debugCtlLow: 0x0  debugCtlHigh: 0x0
Dec 28 00:02:51.119: vcpu-0| MONITOR PANIC: vcpu-1:VMM fault: regs=0x2da8, exc=14, eip=0xe2117
Dec 28 00:02:51.119: vcpu-0| Core dump with build build-283373
Dec 28 00:02:51.135: vcpu-0| Writing monitor corefile "/vmfs/volumes/4ca07c12-e47080e2-45c9-001a4bec9998/vmweb20p10/vmware-core0.gz"
Dec 28 00:02:51.135: vcpu-0| CoreDump: dumping core with superuser privileges
Dec 28 00:02:51.183: vcpu-0| VMK Stack for vcpu 0 is at 3bad000
Dec 28 00:02:51.183: vcpu-0| Beginning monitor coredump
Dec 28 00:02:51.315: vcpu-0| End monitor coredump
Dec 28 00:02:51.316: vcpu-0| Beginning extended monitor coredump
Dec 28 00:02:51.316: vcpu-0| Writing anonymous pages at pos: 404000
Dec 28 00:02:52.648: vcpu-0| Writing monitor corefile "/vmfs/volumes/4ca07c12-e47080e2-45c9-001a4bec9998/vmweb20p10/vmware-core1.gz"
Dec 28 00:02:52.648: vcpu-0| CoreDump: dumping core with superuser privileges
Dec 28 00:02:52.657: vcpu-0| VMK Stack for vcpu 1 is at 3bb1000
Dec 28 00:02:52.658: vcpu-0| Beginning monitor coredump
Dec 28 00:02:52.791: vcpu-0| End monitor coredump
Dec 28 00:02:52.792: vcpu-0| Beginning extended monitor coredump
Dec 28 00:02:52.792: vcpu-0| Writing anonymous pages at pos: 404000
Dec 28 00:02:54.136: vcpu-0| Msg_Post: Error
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic] *** VMware ESX Server internal monitor error ***
Dec 28 00:02:54.136: vcpu-0| vcpu-1:VMM fault: regs=0x2da8, exc=14, eip=0xe2117
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic.report] Please report this problem by selecting menu item Help > VMware on the Web > Request Support, or by going to the Web page "http://www.vmware.com/info?id=8&logFile=%2fvmfs%2fvolumes%2f4ca07c12%2de47080e2%2d45c9%2d001a4bec999...". Please provide us with the log file (/vmfs/volumes/4ca07c12-e47080e2-45c9-001a4bec9998/vmweb20p10/vmware.log) and the core file (/vmfs/volumes/4ca07c12-e47080e2-45c9-001a4bec9998/vmweb20p10/vmware-core[0-1].gz).
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic.serverdebug] If the problem is repeatable, please set 'Use Debug Monitor' to 'Yes' in the 'Misc' section of the Configure Virtual Machine Web page. Then reproduce the incident and file it according to the instructions.
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic.vmSupport] To collect files to submit to VMware support, run "vm-support".
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic.entitlement] We will respond on the basis of your support entitlement.
Dec 28 00:02:54.136: vcpu-0| [msg.log.monpanic.finish] We appreciate your feedback,
Dec 28 00:02:54.136: vcpu-0|   -- the VMware ESX Server team.
Dec 28 00:02:54.136: vcpu-0| ----------------------------------------
Dec 28 00:02:54.139: vcpu-0| Exiting vcpu-0
Dec 28 00:02:55.137: vmx| VTHREAD watched thread 4 "vcpu-0" died
Dec 28 00:02:55.238: mks| VTHREAD watched thread 0 "vmx" died

Dec 28 00:02:55.764: vcpu-1| VTHREAD watched thread 0 "vmx" died

I have the following configuration:

- ESX 3.5 build 283373

- Virtual machine with two cores and 1GB Memory

- SLES 11 SP1 installed

Have you any ideas? Thanx for your help.

Cheers

[vDeepak]

I think its a known issue that is caused by VMM. virtual machine monitor.

Have a look at Vmware KB arctile 1021174 (I dont knw why ..but cant copy the link ).

Thanks

[onkelalbert]

Hi

I've updated the ESX Servers now, like it was said in the KB, but still having the issue.

I've tried to read the coredump with gdb, but there was a segfault while reading (is that normal..?)

Uhm, could get some information with strings, see below.

Minor version mismatch in vmm-vmk interface (%#x)
This version of the monitor (%#x) not supported by vmkernel

This virtual machine has tried to execute an invalid part of memory. It is possible that the virtual disk or CD media you are using is not bootable or that a program or driver running in the virtual machine has malfunctioned.

1394: Guest access beyond available registers. Offset(0x%x) Size(0x%x)
1394: Unexpected guest register access size (%d)
MACVMKPend%d
MACVMKChannel
Vmxnet
%s%d
%sIntr%d
VmxnetWakePattern: bad sequence, ignoring
VmxnetCommand 0x%x failed: 0x%x
VLANCE: Not fully initialized, no mapping of tx ring
ADAPTER ID: %d, BADX: 0x%x, CADX: 0x%x, NXDA: 0x%x, XMTRL: %d
VLANCE: Not fully initialized, no mapping of rx ring
ADAPTER ID: %d, BADR: 0x%x, CADR: 0x%x, NRDA: 0x%x, RCVRL: %d
E1000: -- read not allowed on IMC
E1000: Failed to stop vmkernel, status %#x.
E1000: Failed to start vmkernel, status %#x.
E1000_%u.%u

Memory allocation failure
BUG 3485:
The amount of memory given to the guest OS is too large. Try reducing it.

The last one was really interesting, because until the last night the vm was running with 1GB memory. After the last crash, I've set 4 GB, but the error was there, no matter how much memory was given.
Also, I haven't found anything about this BUG 3485...

Kernelversion is 2.6.27.48-0.1-vmi

Thanks