Highlighted
Contributor
Contributor

ESXi 6.7 kernel panic during Heap_Free in dlmalloc

Jump to solution

I'm not sure if this is the right place to post this, but I couldn't find a bug tracker for ESXi.

The host has been running ESXi 6.0 and now 6.7 without issue for a couple years. After reboot, it's all back to normal; so, I don't think it's a hardware problem. I wasn't doing anything in particular when this happened. I.e., no easy repro.

Host: VMkernel esxi2 6.7.0 #1 SMP Release build-8169922 Apr  3 2018 14:48:22 x86_64 x86_64 x86_64 ESXi

Stack trace:

Line prefix: 2018-05-24T00:44:53.953Z, cpu13:2097958)

@BlueScreen: PANIC bora/vmkernel/main/dlmalloc.c:4924 - Usage error in dlmalloc

Code start: 0x41801cc00000 VMK uptime: 28:20:55:23.811

0x451a0a79b470:[0x41801cd08ca5]PanicvPanicInt@vmkernel#nover+0x439 stack: 0x7520676e69726f6e

0x451a0a79b510:[0x41801cd08ed8]Panic_NoSave@vmkernel#nover+0x4d stack: 0x451a0a79b570

0x451a0a79b570:[0x41801cd512c2]DLM_free@vmkernel#nover+0x657 stack: 0x431c091f1fb0

0x451a0a79b590:[0x41801cd4e4b0]Heap_Free@vmkernel#nover+0x115 stack: 0x451a0a79b630

0x451a0a79b5e0:[0x41801df056ca]CbtAsyncIODone@(hbr_filter)#<None>+0x2b stack: 0x459a40a26800

0x451a0a79b610:[0x41801ccc89a2]AsyncPopCallbackFrameInt@vmkernel#nover+0x4b stack: 0x451a0a79b670

0x451a0a79b640:[0x41801df072a9]Lwd_IssuePendingIO@(hbr_filter)#<None>+0x9a stack: 0x431c09235c50

0x451a0a79b670:[0x41801defa413]DemandLogReadTokenCallback@(hbr_filter)#<None>+0x1a4 stack: 0x41801cf76149

0x451a0a79b7a0:[0x41801ccc89a2]AsyncPopCallbackFrameInt@vmkernel#nover+0x4b stack: 0x459a40a0d000

0x451a0a79b7d0:[0x41801ccc89a2]AsyncPopCallbackFrameInt@vmkernel#nover+0x4b stack: 0x459a40b16b80

0x451a0a79b800:[0x41801cface08]VSCSI_FSVirtAsyncDone@vmkernel#nover+0x59 stack: 0x17eda3ffe77b66

0x451a0a79b810:[0x41801ccc89a2]AsyncPopCallbackFrameInt@vmkernel#nover+0x4b stack: 0x459a40a55398

0x451a0a79b840:[0x41801cc4d96b]FS_IOAccessDone@vmkernel#nover+0x68 stack: 0x1

0x451a0a79b860:[0x41801ccc89a2]AsyncPopCallbackFrameInt@vmkernel#nover+0x4b stack: 0x459a5d587d00

0x451a0a79b890:[0x41801cc678e8]FDSAsyncTokenIODone@vmkernel#nover+0x91 stack: 0x459a5a861900

0x451a0a79b8c0:[0x41801cf65b2a]SCSIDeviceCmdCompleteInt@vmkernel#nover+0x6f stack: 0x459a410ba240

0x451a0a79b930:[0x41801cf66a17]SCSIDeviceCmdCompleteCB@vmkernel#nover+0x2bc stack: 0x41801d8be5c4

0x451a0a79ba10:[0x41801cf68a2e]SCSICompleteDeviceCommand@vmkernel#nover+0xa7 stack: 0x0

0x451a0a79bb10:[0x41801d883fa0]nmp_CompleteCommandForDevice@com.vmware.vmkapi#v2_5_0_0+0x39 stack: 0x459a5b8cb340

0x451a0a79bb70:[0x41801d884468]nmp_CompleteCommandForPath@com.vmware.vmkapi#v2_5_0_0+0x61 stack: 0x418043400d40

0x451a0a79bcc0:[0x41801cf888c4]SCSICompletePathCommand@vmkernel#nover+0x1f5 stack: 0x430469218180

0x451a0a79bd90:[0x41801cf77849]SCSICompleteAdapterCommand@vmkernel#nover+0x13e stack: 0x900000200

0x451a0a79be20:[0x41801d5f1978]SCSILinuxWorldletFn@com.vmware.driverAPI#9.2+0x3f1 stack: 0x430874ce72e0

0x451a0a79bf80:[0x41801cd3e694]WorldletFunc@vmkernel#nover+0xf5 stack: 0x0

0x451a0a79bfe0:[0x41801cf081f2]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0

base fs=0x0 gs=0x418043400000 Kgs=0x0

Lemme know if you need more info.

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
VMware Employee
VMware Employee

This is a known issue with ESXi 6.7. We are currently still working on determining the root cause of the issue. We published the following KB https://kb.vmware.com/s/article/55650.

If you do experience this PSOD please open a Support Request with GSS to obtain a debug build for future troubleshooting.

James Walker VMware Support Moderator

View solution in original post

0 Kudos
5 Replies
Highlighted
Champion
Champion

You are not alone.

This is probably a bug, open a Service Request with VMware Technical Support ESXi 6.7 Keep Getting PANIC bora/vmkernel/main/dlmalloc.c: 4924 - Usage error in dlmalloc

0 Kudos
Highlighted
Contributor
Contributor

Hello; I tried to send you a private message but it failed. VMware support is tracking this issue for us also; however since the last time it occured the Sr. Solution Engineer who had been working the issue is now on PTO until July 3. Have you received any additional information as to a fix or workaround to avoid? We were told the issue was likely something related to VDR (vsphere replication).

Thanks,

John

0 Kudos
Highlighted
Contributor
Contributor

Hello, any workarounds for this issue ?

0 Kudos
Highlighted
VMware Employee
VMware Employee

This is a known issue with ESXi 6.7. We are currently still working on determining the root cause of the issue. We published the following KB https://kb.vmware.com/s/article/55650.

If you do experience this PSOD please open a Support Request with GSS to obtain a debug build for future troubleshooting.

James Walker VMware Support Moderator

View solution in original post

0 Kudos
Highlighted
Contributor
Contributor

Any progress on this issue, we are facing this issue as well with SRM 8.1 and vCenter/ESXi 6.7 ?

0 Kudos