A PSOD on Mutex usage when developing a block devi...

sstarrysky · ‎07-25-2014

Hi All:

I am porting a block device driver to ESXi 5.1. The module could work now, ESXi take our SSD as a local disk.

I've created Guest OS on it and run some tests, but this PSOD will eventually come out after a time period of several hours.

__shannon_read_cmd calls mutex_lock & mutex_unlock and this function is called nearly every second.

Does anyone have a clue on this?

Thanks in advance.

spike.

2014-05-21T21:48:31.548Z cpu0:2473)World: 8376: PRDA 0x418040000000 ss 0x0 ds 0x4018 es 0x4018 fs 0x4018 gs 0x4018

2014-05-21T21:48:31.558Z cpu0:2473)World: 8378: TR 0x4020 GDT 0x412206a61000 (0x402f) IDT 0x418026112000 (0xfff)

2014-05-21T21:48:31.568Z cpu0:2473)World: 8379: CR0 0x80010031 CR3 0x11b39c000 CR4 0x42768

2014-05-21T21:48:31.585Z cpu0:2473)Backtrace for current CPU #0, worldID=2473, ebp=0x412206a5a4f0

2014-05-21T21:48:31.595Z cpu0:2473)0x412206a5a4f0:[0x41802607ad3a]PanicvPanicInt@vmkernel#nover+0x61 stack: 0x3000000008, 0x412206a5a5

2014-05-21T21:48:31.607Z cpu0:2473)0x412206a5a5d0:[0x41802607b62b]Panic@vmkernel#nover+0xae stack: 0x0, 0x0, 0x4100061836c0, 0x0, 0x65

2014-05-21T21:48:31.620Z cpu0:2473)0x412206a5a640:[0x4180261c5dae]CpuSchedWait@vmkernel#nover+0xa1 stack: 0x0, 0x412206a68300, 0x41220

2014-05-21T21:48:31.633Z cpu0:2473)0x412206a5a660:[0x4180261c61e7]CpuSched_WaitIRQ@vmkernel#nover+0x1a stack: 0x412206a5a690, 0x412206

2014-05-21T21:48:31.645Z cpu0:2473)0x412206a5a690:[0x418026558da2]LinuxTask_WaitEvent@com.vmware.driverAPI#9.2+0x4d stack: 0x1, 0x4122

2014-05-21T21:48:31.658Z cpu0:2473)0x412206a5a6f0:[0x418026512b60]__down@com.vmware.driverAPI#9.2+0xcf stack: 0xffffffff, 0x29, 0x417f

2014-05-21T21:48:31.671Z cpu0:2473)0x412206a5a750:[0x41802652e6a5]__down_failed@com.vmware.driverAPI#9.2+0x34 stack: 0x410019581090, 0

2014-05-21T21:48:31.683Z cpu0:2473)0x412206a5a7e0:[0x418026a05b8d]__shannon_read_cmd@<None>#<None>+0x120 stack: 0xfffffeff06a5a810, 0x

2014-05-21T21:48:31.696Z cpu0:2473)0x412206a5a840:[0x418026a066c5]lunset_pick_request@<None>#<None>+0x110 stack: 0x20000000002, 0x8, 0

2014-05-21T21:48:31.709Z cpu0:2473)0x412206a5a910:[0x418026a0bc44]shannon_submit_aligned_bio@<None>#<None>+0x6b7 stack: 0x412401305e00

2014-05-21T21:48:31.721Z cpu0:2473)0x412206a5a940:[0x418026a2789f]shannon_disk_xfer_request@<None>#<None>+0xbe stack: 0x41240133e940,

2014-05-21T21:48:31.734Z cpu0:2473)0x412206a5a970:[0x418026a27974]shannon_disk_request@<None>#<None>+0x4f stack: 0x47, 0x418026a27924,

2014-05-21T21:48:31.747Z cpu0:2473)0x412206a5a9b0:[0x41802652f08d]__generic_unplug_device@com.vmware.driverAPI#9.2+0x4c stack: 0x41001

2014-05-21T21:48:31.759Z cpu0:2473)0x412206a5a9e0:[0x41802652f102]generic_unplug_device@com.vmware.driverAPI#9.2+0x65 stack: 0x0, 0x41

2014-05-21T21:48:31.772Z cpu0:2473)0x412206a5ab20:[0x418026532231]LinuxBlockCommand@com.vmware.driverAPI#9.2+0x9f0 stack: 0x0, 0x41802

2014-05-21T21:48:31.785Z cpu0:2473)0x412206a5abc0:[0x41802627a0ba]SCSIStartAdapterCommands@vmkernel#nover+0x2f9 stack: 0x418026395410,

2014-05-21T21:48:31.797Z cpu0:2473)0x412206a5ad50:[0x418026288db7]SCSIStartPathCommands@vmkernel#nover+0x1a02 stack: 0x4100050d80f0, 0

2014-05-21T21:48:31.810Z cpu0:2473)0x412206a5add0:[0x418026289400]SCSIIssueAsyncPathCommandDirect@vmkernel#nover+0x48f stack: 0x412206

2014-05-21T21:48:31.823Z cpu0:2473)0x412206a5ae40:[0x418026851a0e]nmp_SelectPathAndIssueCommand@com.vmware.vmkapi#v2_1_0_0+0x2c1 stack

2014-05-21T21:48:31.835Z cpu0:2473)0x412206a5aea0:[0x418026852ad7]nmp_IssueCommandToDevice@com.vmware.vmkapi#v2_1_0_0+0xc2 stack: 0x0,

2014-05-21T21:48:31.848Z cpu0:2473)0x412206a5af00:[0x41802684ecf2]nmp_DeviceStartLoop@com.vmware.vmkapi#v2_1_0_0+0x319 stack: 0x412400

2014-05-21T21:48:31.861Z cpu0:2473)0x412206a5af20:[0x41802684eef6]nmpDeviceStart@com.vmware.vmkapi#v2_1_0_0+0xbd stack: 0x4100050dac40

2014-05-21T21:48:31.873Z cpu0:2473)0x412206a5af50:[0x41802626ca06]SCSIStartDeviceInt@vmkernel#nover+0x45 stack: 0x1, 0x4124007f0410, 0

2014-05-21T21:48:31.886Z cpu0:2473)0x412206a5b010:[0x41802626e143]SCSIAsyncDeviceCommand@vmkernel#nover+0xd06 stack: 0x0, 0x4100000101

2014-05-21T21:48:31.899Z cpu0:2473)0x412206a5b060:[0x418026255581]FDS_CommonAsyncIO@vmkernel#nover+0x48 stack: 0x0, 0x4124001b5d80, 0x

2014-05-21T21:48:31.911Z cpu0:2473)0x412206a5b120:[0x418026252309]FDS_AsyncIO@vmkernel#nover+0x20c stack: 0x412206a5b2f0, 0x412206a5b3

2014-05-21T21:48:31.924Z cpu0:2473)0x412206a5b410:[0x4180269f043e]Fil3_FileIO@esx#nover+0xd89 stack: 0x410019930410, 0x41220e7e7000, 0

2014-05-21T21:48:31.937Z cpu0:2473)0x412206a5b4a0:[0x4180269f0a83]Fil3_FileIOWithRetry@esx#nover+0xb2 stack: 0x4124001b5d80, 0x0, 0x20

2014-05-21T21:48:31.949Z cpu0:2473)0x412206a5b4f0:[0x41802622c21f]FSSFileIO@vmkernel#nover+0xfe stack: 0x4124001b5d80, 0x9c273c0, 0x41

2014-05-21T21:48:31.962Z cpu0:2473)0x412206a5b510:[0x41802622d081]FSS_AsyncFileIO@vmkernel#nover+0x10 stack: 0x412206a5b5c0, 0x1944180

2014-05-21T21:48:31.975Z cpu0:2473)0x412206a5b610:[0x418026223523]VSCSI_FSCommand@vmkernel#nover+0x84a stack: 0x0, 0x412401039620, 0x4

2014-05-21T21:48:31.987Z cpu0:2473)0x412206a5b650:[0x418026217ecb]VSCSI_IssueCommandBE@vmkernel#nover+0x52 stack: 0x412206a5b6b0, 0x41

2014-05-21T21:48:32.000Z cpu0:2473)0x412206a5b700:[0x41802621c90c]VSCSI_HandleCommand@vmkernel#nover+0x447 stack: 0x60, 0x0, 0x22df3,

2014-05-21T21:48:32.013Z cpu0:2473)0x412206a5b7c0:[0x41802621cb93]VSCSI_VmkExecuteCommand@vmkernel#nover+0x1f2 stack: 0x412200001000,

2014-05-21T21:48:32.025Z cpu0:2473)0x412206a5b9a0:[0x41802622a265]LSIProcessReqInt@vmkernel#nover+0x864 stack: 0x410009c23340, 0x41000

2014-05-21T21:48:32.038Z cpu0:2473)0x412206a5ba00:[0x41802622a3ff]LSIProcessRequestRing@vmkernel#nover+0x72 stack: 0x0, 0x0, 0x410009c

2014-05-21T21:48:32.051Z cpu0:2473)0x412206a5ba30:[0x418026220cc4]VSCSI_WorldletCB@vmkernel#nover+0x9b stack: 0xba, 0x1, 0x4100016257b

2014-05-21T21:48:32.063Z cpu0:2473)0x412206a5bbb0:[0x41802610b21b]WorldletProcessQueue@vmkernel#nover+0x486 stack: 0x412206a5bc60, 0xb

2014-05-21T21:48:32.076Z cpu0:2473)0x412206a5bbf0:[0x41802610b895]WorldletBHHandler@vmkernel#nover+0x60 stack: 0x100000000000001, 0x41

2014-05-21T21:48:32.089Z cpu0:2473)0x412206a5bc70:[0x41802602083a]BH_Check@vmkernel#nover+0x185 stack: 0x412206a5bd70, 0x412206a5bd10,

2014-05-21T21:48:32.101Z cpu0:2473)0x412206a5bd70:[0x4180261bc9bc]CpuSchedIdleLoopInt@vmkernel#nover+0x13b stack: 0x206a5bda0, 0x41000

2014-05-21T21:48:32.114Z cpu0:2473)0x412206a5be60:[0x4180261c4c1f]CpuSchedDispatch@vmkernel#nover+0xabe stack: 0x0, 0x0, 0x41000618150

2014-05-21T21:48:32.127Z cpu0:2473)0x412206a5bed0:[0x4180261c5f4f]CpuSchedWait@vmkernel#nover+0x242 stack: 0x410000000000, 0x412206a5b

2014-05-21T21:48:32.140Z cpu0:2473)0x412206a5bef0:[0x4180261c61e7]CpuSched_WaitIRQ@vmkernel#nover+0x1a stack: 0x412206a5bf20, 0x418000

2014-05-21T21:48:32.152Z cpu0:2473)0x412206a5bf60:[0x4180261c6463]CpuSched_TimedWaitIRQ@vmkernel#nover+0xa6 stack: 0x0, 0x412206967000

2014-05-21T21:48:32.165Z cpu0:2473)0x412206a5bff0:[0x41802604810c]helpFunc@vmkernel#nover+0x20b stack: 0x0, 0x0, 0x0, 0x0, 0x0

2014-05-21T21:48:32.177Z cpu0:2473)0x412206a5bff8:[0x0]<unknown> stack: 0x0, 0x0, 0x0, 0x0, 0x0

2014-05-21T21:48:32.186Z cpu0:2473)VMware ESXi 5.1.0 [Releasebuild-1065491 x86_64]

NOT_IMPLEMENTED bora/vmkernel/sched/cpusched.c:8001

2014-05-21T21:48:32.199Z cpu0:2473)cr0=0x8001003d cr2=0xb7f3d0 cr3=0xbd93f000 cr4=0x216c

2014-05-21T21:48:32.207Z cpu0:2473)pcpu:0 world:2473 name:"helper31-10" (SH)

2014-05-21T21:48:32.214Z cpu0:2473)pcpu:1 world:2476 name:"helper31-13" (SH)

2014-05-21T21:48:32.221Z cpu0:2473)@BlueScreen: NOT_IMPLEMENTED bora/vmkernel/sched/cpusched.c:8001

2014-05-21T21:48:32.229Z cpu0:2473)Code start: 0x418026000000 VMK uptime: 0:05:41:12.774

2014-05-21T21:48:32.238Z cpu0:2473)0x412206a5a4f0:[0x41802607ad3a]PanicvPanicInt@vmkernel#nover+0x61 stack: 0x3000000008

2014-05-21T21:48:32.250Z cpu0:2473)0x412206a5a5d0:[0x41802607b62b]Panic@vmkernel#nover+0xae stack: 0x0

2014-05-21T21:48:32.260Z cpu0:2473)0x412206a5a640:[0x4180261c5dae]CpuSchedWait@vmkernel#nover+0xa1 stack: 0x0

2014-05-21T21:48:32.270Z cpu0:2473)0x412206a5a660:[0x4180261c61e7]CpuSched_WaitIRQ@vmkernel#nover+0x1a stack: 0x412206a5a690

2014-05-21T21:48:32.282Z cpu0:2473)0x412206a5a690:[0x418026558da2]LinuxTask_WaitEvent@com.vmware.driverAPI#9.2+0x4d stack: 0x1

2014-05-21T21:48:32.294Z cpu0:2473)0x412206a5a6f0:[0x418026512b60]__down@com.vmware.driverAPI#9.2+0xcf stack: 0xffffffff

2014-05-21T21:48:32.306Z cpu0:2473)0x412206a5a750:[0x41802652e6a5]__down_failed@com.vmware.driverAPI#9.2+0x34 stack: 0x410019581090

2014-05-21T21:48:32.319Z cpu0:2473)0x412206a5a7e0:[0x418026a05b8d]__shannon_read_cmd@<None>#<None>+0x120 stack: 0xfffffeff06a5a810

2014-05-21T21:48:32.331Z cpu0:2473)0x412206a5a840:[0x418026a066c5]lunset_pick_request@<None>#<None>+0x110 stack: 0x20000000002

2014-05-21T21:48:32.343Z cpu0:2473)0x412206a5a910:[0x418026a0bc44]shannon_submit_aligned_bio@<None>#<None>+0x6b7 stack: 0x412401305e00

2014-05-21T21:48:32.356Z cpu0:2473)0x412206a5a940:[0x418026a2789f]shannon_disk_xfer_request@<None>#<None>+0xbe stack: 0x41240133e940

2014-05-21T21:48:32.369Z cpu0:2473)0x412206a5a970:[0x418026a27974]shannon_disk_request@<None>#<None>+0x4f stack: 0x47

2014-05-21T21:48:32.380Z cpu0:2473)0x412206a5a9b0:[0x41802652f08d]__generic_unplug_device@com.vmware.driverAPI#9.2+0x4c stack: 0x41001

2014-05-21T21:48:32.393Z cpu0:2473)0x412206a5a9e0:[0x41802652f102]generic_unplug_device@com.vmware.driverAPI#9.2+0x65 stack: 0x0

2014-05-21T21:48:32.405Z cpu0:2473)0x412206a5ab20:[0x418026532231]LinuxBlockCommand@com.vmware.driverAPI#9.2+0x9f0 stack: 0x0

2014-05-21T21:48:32.417Z cpu0:2473)0x412206a5abc0:[0x41802627a0ba]SCSIStartAdapterCommands@vmkernel#nover+0x2f9 stack: 0x418026395410

2014-05-21T21:48:32.430Z cpu0:2473)0x412206a5ad50:[0x418026288db7]SCSIStartPathCommands@vmkernel#nover+0x1a02 stack: 0x4100050d80f0

2014-05-21T21:48:32.442Z cpu0:2473)0x412206a5add0:[0x418026289400]SCSIIssueAsyncPathCommandDirect@vmkernel#nover+0x48f stack: 0x412206

2014-05-21T21:48:32.455Z cpu0:2473)0x412206a5ae40:[0x418026851a0e]nmp_SelectPathAndIssueCommand@com.vmware.vmkapi#v2_1_0_0+0x2c1 stack

2014-05-21T21:48:32.468Z cpu0:2473)0x412206a5aea0:[0x418026852ad7]nmp_IssueCommandToDevice@com.vmware.vmkapi#v2_1_0_0+0xc2 stack: 0x0

2014-05-21T21:48:32.481Z cpu0:2473)0x412206a5af00:[0x41802684ecf2]nmp_DeviceStartLoop@com.vmware.vmkapi#v2_1_0_0+0x319 stack: 0x412400

2014-05-21T21:48:32.493Z cpu0:2473)0x412206a5af20:[0x41802684eef6]nmpDeviceStart@com.vmware.vmkapi#v2_1_0_0+0xbd stack: 0x4100050dac40

2014-05-21T21:48:32.506Z cpu0:2473)0x412206a5af50:[0x41802626ca06]SCSIStartDeviceInt@vmkernel#nover+0x45 stack: 0x1

2014-05-21T21:48:32.517Z cpu0:2473)0x412206a5b010:[0x41802626e143]SCSIAsyncDeviceCommand@vmkernel#nover+0xd06 stack: 0x0

2014-05-21T21:48:32.529Z cpu0:2473)0x412206a5b060:[0x418026255581]FDS_CommonAsyncIO@vmkernel#nover+0x48 stack: 0x0

2014-05-21T21:48:32.540Z cpu0:2473)0x412206a5b120:[0x418026252309]FDS_AsyncIO@vmkernel#nover+0x20c stack: 0x412206a5b2f0

2014-05-21T21:48:32.552Z cpu0:2473)0x412206a5b410:[0x4180269f043e]Fil3_FileIO@esx#nover+0xd89 stack: 0x410019930410

2014-05-21T21:48:32.563Z cpu0:2473)0x412206a5b4a0:[0x4180269f0a83]Fil3_FileIOWithRetry@esx#nover+0xb2 stack: 0x4124001b5d80

2014-05-21T21:48:32.575Z cpu0:2473)0x412206a5b4f0:[0x41802622c21f]FSSFileIO@vmkernel#nover+0xfe stack: 0x4124001b5d80

2014-05-21T21:48:32.586Z cpu0:2473)0x412206a5b510:[0x41802622d081]FSS_AsyncFileIO@vmkernel#nover+0x10 stack: 0x412206a5b5c0

2014-05-21T21:48:32.598Z cpu0:2473)0x412206a5b610:[0x418026223523]VSCSI_FSCommand@vmkernel#nover+0x84a stack: 0x0

2014-05-21T21:48:32.610Z cpu0:2473)base fs=0x0 gs=0x418040000000 Kgs=0x0

2014-05-21T21:48:32.616Z cpu0:2473)vmkernel 0x0 .data 0x0 .bss 0x0

2014-05-21T21:48:32.623Z cpu0:2473)chardevs 0x418026470000 .data 0x417fc0000000 .bss 0x417fc00008a0

2014-05-21T21:48:32.633Z cpu0:2473)user 0x418026475000 .data 0x417fc0400000 .bss 0x417fc0413240

2014-05-21T21:48:32.643Z cpu0:2473)procfs 0x4180264fb000 .data 0x417fc0800000 .bss 0x417fc0800220

2014-05-21T21:48:32.653Z cpu0:2473)vfat 0x4180264fe000 .data 0x417fc0c00000 .bss 0x417fc0c021c0

2014-05-21T21:48:32.662Z cpu0:2473)vmkapi_socket 0x418026508000 .data 0x417fc1000000 .bss 0x417fc1000700

2014-05-21T21:48:32.672Z cpu0:2473)vmkapi_v2_0_0_0_vmkernel_shim 0x41802650b000 .data 0x417fc1400000 .bss 0x417fc140b218

2014-05-21T21:48:32.683Z cpu0:2473)vmkplexer 0x41802650c000 .data 0x417fc1800000 .bss 0x417fc18004a0

2014-05-21T21:48:32.693Z cpu0:2473)vmklinux_9 0x418026510000 .data 0x417fc1c00000 .bss 0x417fc1c0e480

2014-05-21T21:48:32.702Z cpu0:2473)vmklinux_9_2_0_0 0x418026588000 .data 0x417fc2000000 .bss 0x417fc200a7f8

2014-05-21T21:48:32.712Z cpu0:2473)vmklinux_9_2_1_0 0x418026589000 .data 0x417fc2400000 .bss 0x417fc240a9a8

2014-05-21T21:48:32.722Z cpu0:2473)random 0x41802658a000 .data 0x417fc2800000 .bss 0x417fc2800740

2014-05-21T21:48:32.732Z cpu0:2473)usb 0x41802658e000 .data 0x417fc2c00000 .bss 0x417fc2c01fa0

2014-05-21T21:48:32.742Z cpu0:2473)ehci-hcd 0x4180265ac000 .data 0x417fc3000000 .bss 0x417fc3000500

2014-05-21T21:48:32.751Z cpu0:2473)usb-uhci 0x4180265b6000 .data 0x417fc3400000 .bss 0x417fc3400420

2014-05-21T21:48:32.761Z cpu0:2473)hid 0x4180265bc000 .data 0x417fc3800000 .bss 0x417fc3800600

2014-05-21T21:48:32.771Z cpu0:2473)procMisc 0x4180265c1000 .data 0x417fc3c00000 .bss 0x417fc3c00000

2014-05-21T21:48:32.781Z cpu0:2473)iscsi_trans 0x4180265c2000 .data 0x417fc4000000 .bss 0x417fc4001420

2014-05-21T21:48:32.791Z cpu0:2473)etherswitch 0x4180265cd000 .data 0x417fc4400000 .bss 0x417fc44114c0

2014-05-21T21:48:32.800Z cpu0:2473)netsched 0x4180265f8000 .data 0x417fc4800000 .bss 0x417fc48030c0

2014-05-21T21:48:32.810Z cpu0:2473)cnic_register 0x4180265fe000 .data 0x417fc4c00000 .bss 0x417fc4c002a0

2014-05-21T21:48:32.820Z cpu0:2473)tg3 0x418026600000 .data 0x417fc5000000 .bss 0x417fc50013e0

2014-05-21T21:48:32.830Z cpu0:2473)vmkapi_v2_0_0_0_iscsi_shim 0x418026627000 .data 0x417fc5400000 .bss 0x417fc5400b00

2014-05-21T21:48:32.840Z cpu0:2473)vmci 0x418026628000 .data 0x417fc5800000 .bss 0x417fc5805120

2014-05-21T21:48:32.850Z cpu0:2473)dell 0x418026646000 .data 0x417fc5c00000 .bss 0x417fc5c02460

2014-05-21T21:48:32.860Z cpu0:2473)healthchk 0x418026649000 .data 0x417fc6000000 .bss 0x417fc600fa20

2014-05-21T21:48:32.869Z cpu0:2473)teamcheck 0x41802665b000 .data 0x417fc6400000 .bss 0x417fc640fe60

2014-05-21T21:48:32.879Z cpu0:2473)vlanmtucheck 0x41802666a000 .data 0x417fc6800000 .bss 0x417fc680fc00

2014-05-21T21:48:32.889Z cpu0:2473)heartbeat 0x41802667b000 .data 0x417fc6c00000 .bss 0x417fc6c0fb00

2014-05-21T21:48:32.899Z cpu0:2473)shaper 0x418026689000 .data 0x417fc7000000 .bss 0x417fc7011a80

2014-05-21T21:48:32.909Z cpu0:2473)lldp 0x41802669a000 .data 0x417fc7400000 .bss 0x417fc7400040

2014-05-21T21:48:32.918Z cpu0:2473)cdp 0x41802669f000 .data 0x417fc7800000 .bss 0x417fc7811000

2014-05-21T21:48:32.928Z cpu0:2473)ipfix 0x4180266b3000 .data 0x417fc7c00000 .bss 0x417fc7c0fbc0

2014-05-21T21:48:32.938Z cpu0:2473)tcpip3 0x4180266c4000 .data 0x417fc8000000 .bss 0x417fc8008440

2014-05-21T21:48:32.948Z cpu0:2473)dvsdev 0x418026777000 .data 0x417fc8400000 .bss 0x417fc8400040

2014-05-21T21:48:32.958Z cpu0:2473)vdl2 0x41802677a000 .data 0x417fc8800000 .bss 0x417fc8800140

2014-05-21T21:48:32.967Z cpu0:2473)dvfilter 0x418026786000 .data 0x417fc8c00000 .bss 0x417fc8c00ea0

2014-05-21T21:48:32.977Z cpu0:2473)lacp 0x41802679e000 .data 0x417fc9000000 .bss 0x417fc9000120

2014-05-21T21:48:32.987Z cpu0:2473)vmkapi_v2_0_0_0_dvfilter_shim 0x4180267a3000 .data 0x417fc9400000 .bss 0x417fc9400b20

2014-05-21T21:48:32.998Z cpu0:2473)dvfilter-generic-fastpath 0x4180267a4000 .data 0x417fc9800000 .bss 0x417fc9810100

2014-05-21T21:48:33.008Z cpu0:2473)vmkstatelogger 0x4180267bb000 .data 0x417fc9c00000 .bss 0x417fc9c03620

2014-05-21T21:48:33.018Z cpu0:2473)svmmirror 0x4180267d8000 .data 0x417fca000000 .bss 0x417fca000100

2014-05-21T21:48:33.027Z cpu0:2473)cbt 0x4180267e2000 .data 0x417fca400000 .bss 0x417fca400080

2014-05-21T21:48:33.037Z cpu0:2473)migrate 0x4180267e4000 .data 0x417fca800000 .bss 0x417fca804fc0

2014-05-21T21:48:33.047Z cpu0:2473)esxfw 0x418026837000 .data 0x417fcac00000 .bss 0x417fcac109a0

2014-05-21T21:48:33.057Z cpu0:2473)dm 0x41802684a000 .data 0x417fcb000000 .bss 0x417fcb000000

2014-05-21T21:48:33.067Z cpu0:2473)nmp 0x41802684c000 .data 0x417fcb400000 .bss 0x417fcb403c20

2014-05-21T21:48:33.076Z cpu0:2473)vmw_satp_local 0x41802686c000 .data 0x417fcb800000 .bss 0x417fcb800050

2014-05-21T21:48:33.086Z cpu0:2473)vmw_satp_default_aa 0x41802686e000 .data 0x417fcbc00000 .bss 0x417fcbc00000

2014-05-21T21:48:33.096Z cpu0:2473)vmw_psp_lib 0x41802686f000 .data 0x417fcc000000 .bss 0x417fcc000390

2014-05-21T21:48:33.106Z cpu0:2473)vmw_psp_fixed 0x418026871000 .data 0x417fcc400000 .bss 0x417fcc400000

2014-05-21T21:48:33.115Z cpu0:2473)vmw_psp_rr 0x418026873000 .data 0x417fcc800000 .bss 0x417fcc800090

2014-05-21T21:48:33.125Z cpu0:2473)vmw_psp_mru 0x418026876000 .data 0x417fccc00000 .bss 0x417fccc00000

2014-05-21T21:48:33.135Z cpu0:2473)libata_92 0x418026878000 .data 0x417fcd000000 .bss 0x417fcd003960

2014-05-21T21:48:33.145Z cpu0:2473)libata_9_2_0_0 0x418026898000 .data 0x417fcd400000 .bss 0x417fcd401ea0

2014-05-21T21:48:33.155Z cpu0:2473)usb-storage 0x418026899000 .data 0x417fcd800000 .bss 0x417fcd804940

2014-05-21T21:48:33.164Z cpu0:2473)vmkapi_v2_0_0_0_nmp_shim 0x4180268a5000 .data 0x417fcdc00000 .bss 0x417fcdc00f58

2014-05-21T21:48:33.175Z cpu0:2473)hbr_filter 0x4180268a6000 .data 0x417fce000000 .bss 0x417fce000380

2014-05-21T21:48:33.184Z cpu0:2473)libfc_92 0x4180268ca000 .data 0x417fce400000 .bss 0x417fce401020

2014-05-21T21:48:33.194Z cpu0:2473)libfcoe_92 0x4180268e4000 .data 0x417fce800000 .bss 0x417fce800320

2014-05-21T21:48:33.204Z cpu0:2473)libfc_9_2_0_0 0x4180268ea000 .data 0x417fcec00000 .bss 0x417fcec00b48

2014-05-21T21:48:33.214Z cpu0:2473)libfcoe_9_2_0_0 0x4180268eb000 .data 0x417fcf000000 .bss 0x417fcf000288

2014-05-21T21:48:33.224Z cpu0:2473)ata_piix 0x4180268ec000 .data 0x417fcf400000 .bss 0x417fcf401080

2014-05-21T21:48:33.233Z cpu0:2473)lvmdriver 0x4180268ef000 .data 0x417fcf800000 .bss 0x417fcf802f40

2014-05-21T21:48:33.243Z cpu0:2473)deltadisk 0x418026903000 .data 0x417fcfc00000 .bss 0x417fcfc05340

2014-05-21T21:48:33.253Z cpu0:2473)sunrpc 0x418026927000 .data 0x417fd0000000 .bss 0x417fd00029c0

2014-05-21T21:48:33.263Z cpu0:2473)nfsclient 0x418026932000 .data 0x417fd0400000 .bss 0x417fd0403440

2014-05-21T21:48:33.273Z cpu0:2473)vprobe 0x418026948000 .data 0x417fd0800000 .bss 0x417fd080cb40

2014-05-21T21:48:33.282Z cpu0:2473)vmkibft 0x41802697f000 .data 0x417fd0c00000 .bss 0x417fd0c03380

2014-05-21T21:48:33.292Z cpu0:2473)iscsi_vmk 0x418026982000 .data 0x417fd1000000 .bss 0x417fd1005dc0

2014-05-21T21:48:33.302Z cpu0:2473)vmfs3 0x4180269ab000 .data 0x417fd1400000 .bss 0x417fd1400f00

2014-05-21T21:48:33.312Z cpu0:2473)shannon 0x418026a01000 .data 0x417fd1800000 .bss 0x417fd1800b20

Coredump to disk.

2014-05-21T21:48:33.373Z cpu0:2473)Slot 1 of 1.

DumpProgress: Faulting world regs Faulting world regs (0/9)

DumpProgress: Vmm code/data Vmm code/data (1/9)

DumpProgress: Vmk code/data/stack Vmk code/data/stack (2/9)

DumpProgress: Vmk data/heap Vmk data/heap (3/9)

DumpProgress: PCPU PCPU (4/9)

DumpProgress: World-specific data World-specific data (5/9)

DumpProgress: Xmap Xmap (6/9)

2014-05-21T21:49:58.980Z cpu0:2473)WARNING: Compress: 160: CompressCheckOutput failed: Limit exceeded [avail_in: 1020, avail_out: 0]

2014-05-21T21:49:58.993Z cpu0:2473)WARNING: Dump: 1478: Compression failure while dumping range 'XMap' 4096 bytes: 0xbad0006: Limit exceeded

2014-05-21T21:49:59.006Z cpu0:2473)XMap: 1644: Error while dumping l2e=145 (of 3072; 48066 validPages), l1e=342 (of 512): Limit exceeded

2014-05-21T21:49:59.028Z cpu0:2473)XMap: 1694: Error dumping xmap, may be truncated (have 447521 pages).

2014-05-21T21:49:59.037Z cpu0:2473)WARNING: Dump: 1819: Failed to flush compressed data with Limit exceeded

2014-05-21T21:49:59.049Z cpu0:2473)Dump: 2450: Compressed dump took 104858112 bytes total.

2014-05-21T21:49:59.057Z cpu0:2473) DiskDump: Partial Dump: Out of space o=0x63ff200 l=0x1000

Debugger waiting(world 2473) Debugger waiting(world 2473) Halting PCPU 1.

All

A PSOD on Mutex usage when developing a block device driver