VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
7 Replies Last post: Jul 5, 2009 3:46 PM by DSTAVERT  

DELL Power Edge 840: Freezes periodically posted: Jul 4, 2009 3:48 PM

Click to view WiAemUser's profile Lurker 5 posts since
Jul 4, 2009
We have 3 identical machines from DELL:
DELL Power Edge 840
Intel Xeon CPU X3220 @ 2.40 Ghz
8 GB Memory

all running with:
ESX 3i 3.5.0 build 163429 (u4?)

All this machines stops working and freezes periodically:
One machine every 7 days, the other two every 14 days (+- some hours).
We don't get any purple screen, just a freezed machine. Only switching power off and restart helps.

In the downloaded 'diagnostic data' we found this informations in the kernel-log:

7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object 1 4 57f 0 0 0 0 0 0 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309

7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05

7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1cc00284 172 0 0 0 0 0: Out of memory

7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309

7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05

7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1d400284 174 0 0 0 0 0: Out of memory

7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05

7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory

7:01:15:14.165 cpu1:3221)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu1:3221)WARNING: Heap: 1522: Heap_Align(mainHeap, 3328/3328 bytes, 4 align) failed. caller: 0x642270

7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309

7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu1:3221)Backtrace for current CPU #1, worldID=3221, ebp=0x3c57ee0
7:01:15:14.165 cpu1:3221)0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38, 0x64239c, 0x83b5f5
7:01:15:14.165 cpu1:3221)0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4, 0x0, 0xad0001
7:01:15:14.165 cpu1:3221)0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34, 0xd33d6c53, 0x5320c
7:01:15:14.166 cpu1:3221)0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54, 0x88ffc0, 0x800
7:01:15:14.166 cpu1:3221)0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000, 0x0, 0x1246
VMware ESX Server 3i Releasebuild-163429
Out of memory for timers
frame=0x3c57d84 ip=0x62b234 cr2=0x8298bb58 cr3=0xd2f03000 cr4=0x669
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3c57ee0 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff
0:1033/BCFlush *1:3221/vmm0:SXXX 2:1766935/vmm0:SXXX 3:1540/sfcbd
@BlueScreen: Out of memory for timers
0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38
0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4
0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34
0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54
0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000
VMK uptime: 7:01:15:14.167 TSC: 1462405465651878
7:01:09:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x904146:0x4213d58
7:01:11:30.555 cpu2:1033)Heartbeat: 470: PCPU 3 didn't have a heartbeat for 180 seconds. may be locked up
7:01:11:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x647e93:0x4213ae0
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory

7:01:15:14.154 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.154 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05


What do you think about this lines?
Any help would be really appreciated!

Re: DELL Power Edge 840: Freezes periodically

1. Jul 4, 2009 7:25 PM in response to: WiAemUser
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
I would download and run memtest. Is all the RAM from Dell? Since it isn't on the HCL what did you need to do to get ESXi to install?

Re: DELL Power Edge 840: Freezes periodically

3. Jul 5, 2009 10:38 AM in response to: WiAemUser
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
memtest.org.

Your machine is not on the HCL. Did you need to do anything special to get ESXi installed? Some people with 840's did need to work at getting it installed because of the hardware.

Start capturing log files.

Re: DELL Power Edge 840: Freezes periodically

5. Jul 5, 2009 1:45 PM in response to: WiAemUser
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
One of the problems with logfiles is that some ESXi logs don't survive a reboot. That means you don't have any reference immediately after an incident since everything just prior to the incident is wiped. You can set up realtime logging. ESXi supports syslog. Configure it from the VI client. There are syslog servers for just about every platform.

Re: DELL Power Edge 840: Freezes periodically

7. Jul 5, 2009 3:46 PM in response to: WiAemUser
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
I don't know what the log references. I would suspect it is a driver issue or memory. Low down issues rather than configuration. To that end I suggested memtest and asked whether you needed to do anything special to install. I also pointed out that your server wasn't on the HCL. I assume here that you hardware firmware is up to date but only you know for sure. The point is to eliminate things that it isn't.

If you have a service contract with VMware it might be time to use it. File an SR. Be prepared to provide as much information as you can.

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities