We have 3 identical machines from DELL:
DELL Power Edge 840
Intel Xeon CPU X3220 @ 2.40 Ghz
8 GB Memory
all running with:
ESX 3i 3.5.0 build 163429 (u4?)
All this machines stops working and freezes periodically:
One machine every 7 days, the other two every 14 days (+- some hours).
We don't get any purple screen, just a freezed machine. Only switching power off and restart helps.
In the downloaded 'diagnostic data' we found this informations in the kernel-log:
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object 1 4 57f 0 0 0 0 0 0 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1cc00284 172 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1d400284 174 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu1:3221)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu1:3221)WARNING: Heap: 1522: Heap_Align(mainHeap, 3328/3328 bytes, 4 align) failed. caller: 0x642270
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu1:3221)Backtrace for current CPU #1, worldID=3221, ebp=0x3c57ee0
7:01:15:14.165 cpu1:3221)0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38, 0x64239c, 0x83b5f5
7:01:15:14.165 cpu1:3221)0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4, 0x0, 0xad0001
7:01:15:14.165 cpu1:3221)0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34, 0xd33d6c53, 0x5320c
7:01:15:14.166 cpu1:3221)0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54, 0x88ffc0, 0x800
7:01:15:14.166 cpu1:3221)0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000, 0x0, 0x1246
VMware ESX Server 3i Releasebuild-163429
Out of memory for timers
frame=0x3c57d84 ip=0x62b234 cr2=0x8298bb58 cr3=0xd2f03000 cr4=0x669
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3c57ee0 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff
0:1033/BCFlush *1:3221/vmm0:SXXX 2:1766935/vmm0:SXXX 3:1540/sfcbd
@BlueScreen: Out of memory for timers
0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38
0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4
0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34
0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54
0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000
VMK uptime: 7:01:15:14.167 TSC: 1462405465651878
7:01:09:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x904146:0x4213d58
7:01:11:30.555 cpu2:1033)Heartbeat: 470: PCPU 3 didn't have a heartbeat for 180 seconds. may be locked up
7:01:11:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x647e93:0x4213ae0
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory
7:01:15:14.154 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.154 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
What do you think about this lines?
Any help would be really appreciated!
DELL Power Edge 840
Intel Xeon CPU X3220 @ 2.40 Ghz
8 GB Memory
all running with:
ESX 3i 3.5.0 build 163429 (u4?)
All this machines stops working and freezes periodically:
One machine every 7 days, the other two every 14 days (+- some hours).
We don't get any purple screen, just a freezed machine. Only switching power off and restart helps.
In the downloaded 'diagnostic data' we found this informations in the kernel-log:
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object 1 4 57f 0 0 0 0 0 0 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1cc00284 172 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 1d400284 174 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
7:01:15:14.165 cpu0:1033)BC: 2546: Failed to flush buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory
7:01:15:14.165 cpu1:3221)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu1:3221)WARNING: Heap: 1522: Heap_Align(mainHeap, 3328/3328 bytes, 4 align) failed. caller: 0x642270
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.165 cpu3:1540)WARNING: Heap: 1522: Heap_Align(mainHeap, 20/20 bytes, 4 align) failed. caller: 0x88a309
7:01:15:14.165 cpu3:1540)WARNING: LinStubs: 2609: Couldn't allocate memory
7:01:15:14.165 cpu1:3221)Backtrace for current CPU #1, worldID=3221, ebp=0x3c57ee0
7:01:15:14.165 cpu1:3221)0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38, 0x64239c, 0x83b5f5
7:01:15:14.165 cpu1:3221)0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4, 0x0, 0xad0001
7:01:15:14.165 cpu1:3221)0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34, 0xd33d6c53, 0x5320c
7:01:15:14.166 cpu1:3221)0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54, 0x88ffc0, 0x800
7:01:15:14.166 cpu1:3221)0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000, 0x0, 0x1246
VMware ESX Server 3i Releasebuild-163429
Out of memory for timers
frame=0x3c57d84 ip=0x62b234 cr2=0x8298bb58 cr3=0xd2f03000 cr4=0x669
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3c57ee0 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff
0:1033/BCFlush *1:3221/vmm0:SXXX 2:1766935/vmm0:SXXX 3:1540/sfcbd
@BlueScreen: Out of memory for timers
0x3c57ee0:0x62b234Panic+0x17 stack: 0x83b5f5, 0x3c57efc, 0x3c57f38
0x3c57ef0:0x62b234Panic+0x17 stack: 0x83b5f5, 0xd00, 0x4
0x3c57f38:0x64239cTimer_AddTC+0x33b stack: 0x1, 0x0, 0x754d34
0x3c57f8c:0x76321dCpuSched_VcpuHalt+0x10c stack: 0x7, 0x81000000, 0x2f54
0x3c57fe0:0x62785dVMKCall+0x12c stack: 0x7, 0x2f54, 0x81000000
VMK uptime: 7:01:15:14.167 TSC: 1462405465651878
7:01:09:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x904146:0x4213d58
7:01:11:30.555 cpu2:1033)Heartbeat: 470: PCPU 3 didn't have a heartbeat for 180 seconds. may be locked up
7:01:11:30.555 cpu3:1540)NMI: 1625: Faulting eip:esp 0x647e93:0x4213ae0
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log buffers for object f530 28 3 4a0bddec e671cc02 2200fd57 1605d719 9800284 148 0 0 0 0 0: Out of memory
7:01:15:14.154 cpu0:1033)WARNING: Heap: 1397: Heap mainHeap already at its maximumSize. Cannot expand.
7:01:15:14.154 cpu0:1033)WARNING: Heap: 1522: Heap_Align(mainHeap, 400/400 bytes, 4 align) failed. caller: 0x7a0a05
What do you think about this lines?
Any help would be really appreciated!