Re: 5 ESXi same hardware . 2 of them crash

omidkosari · ‎08-28-2014

I have 5 ESXi with same hardware 2 of them crashes . Here is the vmkernel-log

2014-08-27T13:47:40.003Z cpu1:108537)World: 14296: VC opID hostd-85e1 maps to vmkernel opID 7c2abc39

2014-08-27T13:48:00.002Z cpu1:108531)World: 14296: VC opID hostd-23a4 maps to vmkernel opID 16c641b3

2014-08-27T13:48:01.033Z cpu0:32957)MCE: 1118: cpu0: MCA error detected via CMCI (Gbl status=0x0): Restart IP: invalid, Error IP: invalid, MCE in progress: no.

2014-08-27T13:48:01.033Z cpu0:32957)MCE: 222: cpu0: bank0: status=0x9000004000010005: (VAL=1, OVFLW=0, UC=0, EN=1, PCC=0, S=0, AR=0), ECC=no, Addr:0x0 (invalid), Misc:0x0 (invalid)

2014-08-27T13:48:01.033Z cpu0:32957)MCE: 231: cpu0: bank0: MCA recoverable error (CE): "Internal Parity Error."

2014-08-27T13:48:01.033Z cpu1:33242)World: 8773: PRDA 0x418040400000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b

2014-08-27T13:48:01.033Z cpu1:33242)World: 8775: TR 0x4020 GDT 0x4123876a1000 (0x402f) IDT 0x418014cf3000 (0xfff)

2014-08-27T13:48:01.033Z cpu1:33242)World: 8776: CR0 0x80010031 CR3 0x11e3e4000 CR4 0x42768

2014-08-27T13:48:01.039Z cpu1:33242)Backtrace for current CPU #1, worldID=33242, ebp=0x4119c0013dd0

2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8, 0x4119c0013e40, 0x41

2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001, 0x5, 0xbe200000

2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e, 0x6738ce, 0xf

2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8, 0x1196300

2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0, 0x13b, 0x0, 0x12a7495c, 0

2014-08-27T13:48:01.040Z cpu1:33242) [45m [33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64] [0m

NOT_REACHED bora/vmkernel/main/idt.c:1165

2014-08-27T13:48:01.040Z cpu1:33242)cr0=0x80010031 cr2=0x12a18be0 cr3=0x11e3e4000 cr4=0x42768

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:0 world:32949 name:"memMapKernel-0" (S)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:1 world:33242 name:"vobd" (U)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:2 world:32791 name:"CmdCompl-2" (S)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:3 world:33210 name:"vmsyslogd" (U)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:4 world:32783 name:"coalesceWorld-0" (S)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:5 world:53216 name:"vmm1:Analytics_VM" (V)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:6 world:33212 name:"vmsyslogd" (U)

2014-08-27T13:48:01.040Z cpu1:33242)pcpu:7 world:32784 name:"netCoalesce2World" (S)

2014-08-27T13:48:01.040Z cpu1:33242)@BlueScreen: NOT_REACHED bora/vmkernel/main/idt.c:1165

2014-08-27T13:48:01.040Z cpu1:33242)Code start: 0x418014c00000 VMK uptime: 2:02:57:31.656

2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8

2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001

2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e

2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8

2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0

2014-08-27T13:48:01.042Z cpu1:33242)base fs=0x0 gs=0x418040400000 Kgs=0x0

2014-08-27T13:48:01.042Z cpu1:33242)MC:PCPU0 B:0 S:0x9000004000010005 M:0x0 A:0x0 0

2014-08-27T13:48:01.043Z cpu1:33242)MC:PCPU1 B:8 S:0xbe2000000005110a M:0x9080000086 A:0x118be2600 5

MC:PCPU0: 1 hardware errors seen since boot (1 corrected by hardware)

MC:PCPU1: 1 hardware errors seen since boot (0 corrected by hardware)

2014-08-27T13:48:01.043Z cpu1:33242)PCPU fam:6 model:58 step:9 type:2 name:Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

2014-08-27T13:48:01.043Z cpu1:33242)vmkernel 0x0 .data 0x0 .bss 0x0

2014-08-27T13:48:01.043Z cpu1:33242)chardevs 0x418015171000 .data 0x417fc0000000 .bss 0x417fc0000400

2014-08-27T13:48:01.043Z cpu1:33242)user 0x418015178000 .data 0x417fc0400000 .bss 0x417fc040e180

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_mgmt 0x41801522b000 .data 0x417fc0800000 .bss 0x417fc0800140

2014-08-27T13:48:01.043Z cpu1:33242)vprobe 0x418015231000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_socket 0x41801526f000 .data 0x417fc1000000 .bss 0x417fc10005c0

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_vmkernel_shim 0x418015274000 .data 0x417fc1400000 .bss 0x417fc14080c0

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_vmkernel_shim 0x418015279000 .data 0x417fc1800000 .bss 0x417fc1808840

2014-08-27T13:48:01.043Z cpu1:33242)procfs 0x41801527e000 .data 0x417fc1c00000 .bss 0x417fc1c00240

2014-08-27T13:48:01.043Z cpu1:33242)vfat 0x418015281000 .data 0x417fc2000000 .bss 0x417fc2002600

2014-08-27T13:48:01.043Z cpu1:33242)procMisc 0x41801528b000 .data 0x417fc2400000 .bss 0x417fc2400000

2014-08-27T13:48:01.043Z cpu1:33242)vmci 0x41801528c000 .data 0x417fc2800000 .bss 0x417fc28057c0

2014-08-27T13:48:01.043Z cpu1:33242)iodm 0x4180152af000 .data 0x417fc2c00000 .bss 0x417fc2c00138

2014-08-27T13:48:01.043Z cpu1:33242)vmkplexer 0x4180152b3000 .data 0x417fc3000000 .bss 0x417fc3000260

2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9 0x4180152b7000 .data 0x417fc3400000 .bss 0x417fc3408e80

2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_0_0 0x41801533e000 .data 0x417fc3800000 .bss 0x417fc3807e84

2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_1_0 0x418015341000 .data 0x417fc3c00000 .bss 0x417fc3c07f98

2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_2_0 0x418015344000 .data 0x417fc4000000 .bss 0x417fc4008838

2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans 0x418015347000 .data 0x417fc4400000 .bss 0x417fc4401800

2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_compat_shim 0x418015352000 .data 0x417fc4800000 .bss 0x417fc480096c

2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_incompat_shim 0x418015353000 .data 0x417fc4c00000 .bss 0x417fc4c007e4

2014-08-27T13:48:01.043Z cpu1:33242)etherswitch 0x418015354000 .data 0x417fc5000000 .bss 0x417fc5013a00

2014-08-27T13:48:01.043Z cpu1:33242)netsched 0x418015389000 .data 0x417fc5400000 .bss 0x417fc5404800

2014-08-27T13:48:01.043Z cpu1:33242)cnic_register 0x41801539b000 .data 0x417fc5800000 .bss 0x417fc58001e0

2014-08-27T13:48:01.043Z cpu1:33242)e1000 0x41801539d000 .data 0x417fc5c00000 .bss 0x417fc5c01240

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_iscsi_shim 0x4180153c3000 .data 0x417fc6000000 .bss 0x417fc6000970

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_iscsi_shim 0x4180153c4000 .data 0x417fc6400000 .bss 0x417fc6400970

2014-08-27T13:48:01.043Z cpu1:33242)random 0x4180153c5000 .data 0x417fc6800000 .bss 0x417fc6800600

2014-08-27T13:48:01.043Z cpu1:33242)usb 0x4180153c9000 .data 0x417fc6c00000 .bss 0x417fc6c01660

2014-08-27T13:48:01.043Z cpu1:33242)ehci-hcd 0x4180153eb000 .data 0x417fc7000000 .bss 0x417fc70002a0

2014-08-27T13:48:01.043Z cpu1:33242)hid 0x4180153f6000 .data 0x417fc7400000 .bss 0x417fc74004e0

2014-08-27T13:48:01.043Z cpu1:33242)healthchk 0x4180153fb000 .data 0x417fc7800000 .bss 0x417fc7811e00

2014-08-27T13:48:01.043Z cpu1:33242)teamcheck 0x418015411000 .data 0x417fc7c00000 .bss 0x417fc7c12240

2014-08-27T13:48:01.043Z cpu1:33242)vlanmtucheck 0x418015424000 .data 0x417fc8000000 .bss 0x417fc8012000

2014-08-27T13:48:01.043Z cpu1:33242)heartbeat 0x418015439000 .data 0x417fc8400000 .bss 0x417fc8411f00

2014-08-27T13:48:01.043Z cpu1:33242)shaper 0x41801544a000 .data 0x417fc8800000 .bss 0x417fc8813e80

2014-08-27T13:48:01.043Z cpu1:33242)lldp 0x41801545d000 .data 0x417fc8c00000 .bss 0x417fc8c00040

2014-08-27T13:48:01.043Z cpu1:33242)cdp 0x418015462000 .data 0x417fc9000000 .bss 0x417fc9013400

2014-08-27T13:48:01.043Z cpu1:33242)ipfix 0x41801547e000 .data 0x417fc9400000 .bss 0x417fc9412540

2014-08-27T13:48:01.043Z cpu1:33242)tcpip4 0x418015492000 .data 0x417fc9800000 .bss 0x417fc9818180

2014-08-27T13:48:01.043Z cpu1:33242)dvsdev 0x418015598000 .data 0x417fc9c00000 .bss 0x417fc9c00030

2014-08-27T13:48:01.043Z cpu1:33242)dvfilter 0x41801559b000 .data 0x417fca000000 .bss 0x417fca000b00

2014-08-27T13:48:01.043Z cpu1:33242)lacp 0x4180155bd000 .data 0x417fca400000 .bss 0x417fca400160

2014-08-27T13:48:01.043Z cpu1:33242)hbr_filter 0x4180155c7000 .data 0x417fca800000 .bss 0x417fca800300

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_dvfilter_shim 0x4180155f3000 .data 0x417fcac00000 .bss 0x417fcac009b0

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_dvfilter_shim 0x4180155f4000 .data 0x417fcb000000 .bss 0x417fcb000930

2014-08-27T13:48:01.043Z cpu1:33242)dvfilter-generic-fastpath 0x4180155f5000 .data 0x417fcb400000 .bss 0x417fcb412380

2014-08-27T13:48:01.043Z cpu1:33242)vmkstatelogger 0x41801560f000 .data 0x417fcb800000 .bss 0x417fcb803a00

2014-08-27T13:48:01.043Z cpu1:33242)esxfw 0x418015633000 .data 0x417fcbc00000 .bss 0x417fcbc12d00

2014-08-27T13:48:01.043Z cpu1:33242)dm 0x418015648000 .data 0x417fcc000000 .bss 0x417fcc000000

2014-08-27T13:48:01.043Z cpu1:33242)nmp 0x41801564a000 .data 0x417fcc400000 .bss 0x417fcc403e50

2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_local 0x41801566d000 .data 0x417fcc800000 .bss 0x417fcc800028

2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_default_aa 0x41801566f000 .data 0x417fccc00000 .bss 0x417fccc00000

2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_lib 0x418015670000 .data 0x417fcd000000 .bss 0x417fcd000290

2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_fixed 0x418015672000 .data 0x417fcd400000 .bss 0x417fcd400000

2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_rr 0x418015674000 .data 0x417fcd800000 .bss 0x417fcd800068

2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_mru 0x418015677000 .data 0x417fcdc00000 .bss 0x417fcdc00000

2014-08-27T13:48:01.043Z cpu1:33242)libata_92 0x418015679000 .data 0x417fce000000 .bss 0x417fce002660

2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_0_0 0x41801569b000 .data 0x417fce400000 .bss 0x417fce401750

2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_1_0 0x41801569c000 .data 0x417fce800000 .bss 0x417fce801750

2014-08-27T13:48:01.043Z cpu1:33242)usb-storage 0x41801569d000 .data 0x417fcec00000 .bss 0x417fcec04780

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_nmp_shim 0x4180156a9000 .data 0x417fcf000000 .bss 0x417fcf000ca8

2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_nmp_shim 0x4180156aa000 .data 0x417fcf400000 .bss 0x417fcf400ca8

2014-08-27T13:48:01.043Z cpu1:33242)svmmirror 0x4180156ab000 .data 0x417fcf800000 .bss 0x417fcf8000c0

2014-08-27T13:48:01.043Z cpu1:33242)cbt 0x4180156b7000 .data 0x417fcfc00000 .bss 0x417fcfc00080

2014-08-27T13:48:01.043Z cpu1:33242)migrate 0x4180156bb000 .data 0x417fd0000000 .bss 0x417fd0004d40

2014-08-27T13:48:01.043Z cpu1:33242)libfc_92 0x41801571a000 .data 0x417fd0400000 .bss 0x417fd0400b80

2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_92 0x418015733000 .data 0x417fd0800000 .bss 0x417fd08001c0

2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_0_0 0x418015739000 .data 0x417fd0c00000 .bss 0x417fd0c00868

2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_0_0 0x41801573a000 .data 0x417fd1000000 .bss 0x417fd10001f4

2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_1_0 0x41801573b000 .data 0x417fd1400000 .bss 0x417fd1400868

2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_1_0 0x41801573c000 .data 0x417fd1800000 .bss 0x417fd18001f4

2014-08-27T13:48:01.043Z cpu1:33242)ahci 0x41801573d000 .data 0x417fd1c00000 .bss 0x417fd1c00420

2014-08-27T13:48:01.043Z cpu1:33242)sunrpc 0x418015744000 .data 0x417fd2000000 .bss 0x417fd2002b80

2014-08-27T13:48:01.043Z cpu1:33242)nfsclient 0x418015753000 .data 0x417fd2400000 .bss 0x417fd2403940

2014-08-27T13:48:01.043Z cpu1:33242)vmkibft 0x41801576c000 .data 0x417fd2800000 .bss 0x417fd28037c0

2014-08-27T13:48:01.043Z cpu1:33242)lvmdriver 0x41801576f000 .data 0x417fd2c00000 .bss 0x417fd2c03380

2014-08-27T13:48:01.043Z cpu1:33242)deltadisk 0x418015783000 .data 0x417fd3000000 .bss 0x417fd3005c00

2014-08-27T13:48:01.043Z cpu1:33242)tracing 0x4180157ae000 .data 0x417fd3400000 .bss 0x417fd3405b40

2014-08-27T13:48:01.043Z cpu1:33242)rdt 0x4180157b5000 .data 0x417fd3800000 .bss 0x417fd3804e00

2014-08-27T13:48:01.043Z cpu1:33242)vsanutil 0x4180157db000 .data 0x417fd3c00000 .bss 0x417fd3c069c0

2014-08-27T13:48:01.043Z cpu1:33242)lsomcommon 0x4180157fa000 .data 0x417fd4000000 .bss 0x417fd4001680

2014-08-27T13:48:01.043Z cpu1:33242)plog 0x41801582e000 .data 0x417fd4400000 .bss 0x417fd44056c0

2014-08-27T13:48:01.043Z cpu1:33242)vmfs3 0x418015871000 .data 0x417fd4800000 .bss 0x417fd4803840

2014-08-27T13:48:01.043Z cpu1:33242)dvfg-igmp 0x4180158d9000 .data 0x417fd5a00000 .bss 0x417fd5a00208

2014-08-27T13:48:01.043Z cpu1:33242)cmmds_net 0x4180158df000 .data 0x417fd5e00000 .bss 0x417fd5e02f40

2014-08-27T13:48:01.043Z cpu1:33242)cmmds 0x4180158ec000 .data 0x417fd6200000 .bss 0x417fd6204d80

2014-08-27T13:48:01.043Z cpu1:33242)cmmds_resolver 0x418015921000 .data 0x417fd6600000 .bss 0x417fd6600140

2014-08-27T13:48:01.043Z cpu1:33242)vsan 0x41801592d000 .data 0x417fd6a00000 .bss 0x417fd6a1c200

2014-08-27T13:48:01.043Z cpu1:33242)vmklink_mpi 0x418015a48000 .data 0x417fd6e00000 .bss 0x417fd6e02400

2014-08-27T13:48:01.043Z cpu1:33242)swapobj 0x418015a4d000 .data 0x417fd7200000 .bss 0x417fd7203010

2014-08-27T13:48:01.043Z cpu1:33242)osfs 0x418015a55000 .data 0x417fd7600000 .bss 0x417fd7603380

2014-08-27T13:48:01.043Z cpu1:33242)vflash 0x418015a63000 .data 0x417fd7a00000 .bss 0x417fd7a03540

2014-08-27T13:48:01.043Z cpu1:33242)vfc 0x418015a6e000 .data 0x417fd7e00000 .bss 0x417fd7e02ac0

Coredump to disk.

2014-08-27T13:48:01.093Z cpu1:33242)Slot 1 of 1.

2014-08-27T13:48:01.093Z cpu1:33242)Dump: 2212: Using dump slot size 2684354560.

Linjo · ‎08-28-2014

First thing to check if the hardware is on the HCL.

If it is I would recommend to upgrade the Bios and check the memory.

// Linjo

Best regards, Linjo Please follow me on twitter: @viewgeek If you find this information useful, please award points for "correct" or "helpful".

omidkosari · ‎08-28-2014

Thanks for reply,

The bios is latest version and memtest shows no error .

JarryG · ‎08-28-2014

It seems you had some parity-errors. While one on cpu0 was corrected, the other one on cpu1 not (probably too many bits to be corrected by ECC). And that caused kernel-panic. Not sure if you can play with HW, but if you could, I'd recommend to take cpu from one of working servers and put it in one of that crashing servers (if you dare to do it).

One more possibility (not so intrusive) is to download the latest cpu-microcode from Intel (or AMD) and upload it to ESXi (mobo-manufacturers are sometimes quite slow with updating bios). I once somewhere seen KB explaining how to load new cpu-microcode on every ESXi-server reboot, but I lost the link. IIRC it must be loaded to certain directory, and file must have some particular name, but I'm not sure...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! 😉

omidkosari · ‎08-28-2014

Thanks for helpful answer .

I,ve found what you mentioned at VMware Front Experience: FAQ: CPU microcode updates and VMware ESXi but unfortunately

File /etc/vmware/microcode/microcode-intel-20140624.dat does not contain a valid microcode update for any of the processors

admin · ‎08-29-2014

omidkosari wrote:

Thanks for helpful answer .

I,ve found what you mentioned at VMware Front Experience: FAQ: CPU microcode updates and VMware ESXi but unfortunately

File /etc/vmware/microcode/microcode-intel-20140624.dat does not contain a valid microcode update for any of the processors

Did you get this microcode.dat from the Intel web site? There should be microcode patch 0x1b for your processors in the 6/24/2014 data file.

omidkosari · ‎09-01-2014

After some investigation the microcodes updated . This time another PSOD occured.

2014-08-30T16:09:16.300Z cpu2:33404)World: 8773: PRDA 0x418040800000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b

2014-08-30T16:09:16.300Z cpu2:33404)World: 8775: TR 0x4020 GDT 0x412389f21000 (0x402f) IDT 0x4180214f3000 (0xfff)

2014-08-30T16:09:16.300Z cpu2:33404)World: 8776: CR0 0x80010031 CR3 0x11e74b000 CR4 0x42768

2014-08-30T16:09:16.305Z cpu2:33404)Backtrace for current CPU #2, worldID=33404, ebp=0x412389f1daf0

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1daf0:[0x4180217a43ba]Power_HaltPCPU@vmkernel#nover+0x1fe stack: 0x1596500, 0x41000670e000

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1db60:[0x41802164ebc9]CpuSchedIdleLoopInt@vmkernel#nover+0x4bd stack: 0x410800000002, 0x0,

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1dcc0:[0x418021654ca0]CpuSchedDispatch@vmkernel#nover+0x1630 stack: 0x410a2918f060, 0x410a

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1dd30:[0x418021655fd5]CpuSchedWait@vmkernel#nover+0x245 stack: 0x1, 0x827c, 0x412300002001

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1ddd0:[0x4180214dcfce]WorldWaitInt@vmkernel#nover+0x2c6 stack: 0x200, 0x25db1df304dc8, 0x3

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1de50:[0x418021980791]UserObj_Poll@<None>#<None>+0x195 stack: 0x410a29191e80, 0x412389f1df

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1deb0:[0x4180219a52b7]LinuxFileDesc_Poll@<None>#<None>+0xaf stack: 0x0, 0x4180219a5208, 0x

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1df00:[0x41802197d080]User_LinuxSyscallHandler@<None>#<None>+0x3f4 stack: 0x412389f1df20,

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1df10:[0x4180214aa67d]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0xffbcb1d8, 0xc8

2014-08-30T16:09:16.305Z cpu2:33404)0x412389f1df20:[0x4180214f1064]gate_entry@vmkernel#nover+0x64 stack: 0x0, 0x13b, 0x0, 0xa8, 0x2

2014-08-30T16:09:16.306Z cpu2:33404) [45m [33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64] [0m

Machine Check Exception: Fatal (unrecoverable) MCE on PCPU2 in world 33404:net-lacp

System has encountered a Hardware Error - Please contact the hardware vendor

2014-08-30T16:09:16.306Z cpu2:33404)cr0=0x80010031 cr2=0x11d56f0 cr3=0x11e74b000 cr4=0x42768

2014-08-30T16:09:16.306Z cpu2:33404)frame=0x4119c0023f40 ip=0x4180217a43ba err=18 rflags=0x202

2014-08-30T16:09:16.306Z cpu2:33404)rax=0x0 rbx=0x4180408001c0 rcx=0x0

2014-08-30T16:09:16.306Z cpu2:33404)rdx=0x0 rbp=0x412389f1daf0 rsi=0x3

2014-08-30T16:09:16.306Z cpu2:33404)rdi=0x25db1bcffebfc r8=0x1 r9=0xffffffffffffffff

2014-08-30T16:09:16.307Z cpu2:33404)r10=0x418021648ee0 r11=0x4108400c5580 r12=0x418040800000

2014-08-30T16:09:16.307Z cpu2:33404)r13=0x0 r14=0x1 r15=0x0

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:0 world:34360 name:"hostd-vix-poll" (U)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:1 world:32958 name:"memMap-1" (S)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:2 world:33404 name:"net-lacp" (U)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:3 world:33472 name:"mclk-sched-vmnic0" (S)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:4 world:32784 name:"netCoalesce2World" (S)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:5 world:36733 name:"vmm1:Analytics_VM" (V)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:6 world:33304 name:"tq:vmklinux" (S)

2014-08-30T16:09:16.307Z cpu2:33404)pcpu:7 world:33482 name:"tq:vsanutil" (S)

2014-08-30T16:09:16.307Z cpu2:33404)@BlueScreen: Machine Check Exception: Fatal (unrecoverable) MCE on PCPU2 in world 33404:net-lacp

System has encountered a Hardware Error - Please contact the hardware vendor

2014-08-30T16:09:16.307Z cpu2:33404)Code start: 0x418021400000 VMK uptime: 2:06:24:32.951

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1daf0:[0x4180217a43ba]Power_HaltPCPU@vmkernel#nover+0x1fe stack: 0x1596500

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1db60:[0x41802164ebc9]CpuSchedIdleLoopInt@vmkernel#nover+0x4bd stack: 0x410800000002

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1dcc0:[0x418021654ca0]CpuSchedDispatch@vmkernel#nover+0x1630 stack: 0x410a2918f060

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1dd30:[0x418021655fd5]CpuSchedWait@vmkernel#nover+0x245 stack: 0x1

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1ddd0:[0x4180214dcfce]WorldWaitInt@vmkernel#nover+0x2c6 stack: 0x200

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1de50:[0x418021980791]UserObj_Poll@<None>#<None>+0x195 stack: 0x410a29191e80

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1deb0:[0x4180219a52b7]LinuxFileDesc_Poll@<None>#<None>+0xaf stack: 0x0

2014-08-30T16:09:16.307Z cpu2:33404)0x412389f1df00:[0x41802197d080]User_LinuxSyscallHandler@<None>#<None>+0x3f4 stack: 0x412389f1df20

2014-08-30T16:09:16.308Z cpu2:33404)0x412389f1df10:[0x4180214aa67d]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0xffbcb1d8

2014-08-30T16:09:16.308Z cpu2:33404)0x412389f1df20:[0x4180214f1064]gate_entry@vmkernel#nover+0x64 stack: 0x0

2014-08-30T16:09:16.310Z cpu2:33404)base fs=0x0 gs=0x418040800000 Kgs=0x0

2014-08-30T16:09:16.310Z cpu2:33404)MC:PCPU2 B:8 S:0xbe2000000005110a M:0x5080000086 A:0x121cb7a80 5

MC:PCPU2: 1 hardware errors seen since boot (0 corrected by hardware)

2014-08-30T16:09:16.310Z cpu2:33404)PCPU fam:6 model:58 step:9 type:2 name:Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

2014-08-30T16:09:16.310Z cpu2:33404)vmkernel 0x0 .data 0x0 .bss 0x0

2014-08-30T16:09:16.310Z cpu2:33404)chardevs 0x418021971000 .data 0x417fc0000000 .bss 0x417fc0000400

2014-08-30T16:09:16.310Z cpu2:33404)user 0x418021978000 .data 0x417fc0400000 .bss 0x417fc040e180

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_mgmt 0x418021a2b000 .data 0x417fc0800000 .bss 0x417fc0800140

2014-08-30T16:09:16.310Z cpu2:33404)vprobe 0x418021a31000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_socket 0x418021a6f000 .data 0x417fc1000000 .bss 0x417fc10005c0

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_0_0_0_vmkernel_shim 0x418021a74000 .data 0x417fc1400000 .bss 0x417fc14080c0

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_1_0_0_vmkernel_shim 0x418021a79000 .data 0x417fc1800000 .bss 0x417fc1808840

2014-08-30T16:09:16.310Z cpu2:33404)procfs 0x418021a7e000 .data 0x417fc1c00000 .bss 0x417fc1c00240

2014-08-30T16:09:16.310Z cpu2:33404)vfat 0x418021a81000 .data 0x417fc2000000 .bss 0x417fc2002600

2014-08-30T16:09:16.310Z cpu2:33404)procMisc 0x418021a8b000 .data 0x417fc2400000 .bss 0x417fc2400000

2014-08-30T16:09:16.310Z cpu2:33404)vmci 0x418021a8c000 .data 0x417fc2800000 .bss 0x417fc28057c0

2014-08-30T16:09:16.310Z cpu2:33404)iodm 0x418021aaf000 .data 0x417fc2c00000 .bss 0x417fc2c00138

2014-08-30T16:09:16.310Z cpu2:33404)vmkplexer 0x418021ab3000 .data 0x417fc3000000 .bss 0x417fc3000260

2014-08-30T16:09:16.310Z cpu2:33404)vmklinux_9 0x418021ab7000 .data 0x417fc3400000 .bss 0x417fc3408e80

2014-08-30T16:09:16.310Z cpu2:33404)vmklinux_9_2_0_0 0x418021b3e000 .data 0x417fc3800000 .bss 0x417fc3807e84

2014-08-30T16:09:16.310Z cpu2:33404)vmklinux_9_2_1_0 0x418021b41000 .data 0x417fc3c00000 .bss 0x417fc3c07f98

2014-08-30T16:09:16.310Z cpu2:33404)vmklinux_9_2_2_0 0x418021b44000 .data 0x417fc4000000 .bss 0x417fc4008838

2014-08-30T16:09:16.310Z cpu2:33404)iscsi_trans 0x418021b47000 .data 0x417fc4400000 .bss 0x417fc4401800

2014-08-30T16:09:16.310Z cpu2:33404)iscsi_trans_compat_shim 0x418021b52000 .data 0x417fc4800000 .bss 0x417fc480096c

2014-08-30T16:09:16.310Z cpu2:33404)iscsi_trans_incompat_shim 0x418021b53000 .data 0x417fc4c00000 .bss 0x417fc4c007e4

2014-08-30T16:09:16.310Z cpu2:33404)etherswitch 0x418021b54000 .data 0x417fc5000000 .bss 0x417fc5013a00

2014-08-30T16:09:16.310Z cpu2:33404)netsched 0x418021b89000 .data 0x417fc5400000 .bss 0x417fc5404800

2014-08-30T16:09:16.310Z cpu2:33404)cnic_register 0x418021b9b000 .data 0x417fc5800000 .bss 0x417fc58001e0

2014-08-30T16:09:16.310Z cpu2:33404)e1000 0x418021b9d000 .data 0x417fc5c00000 .bss 0x417fc5c01240

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_1_0_0_iscsi_shim 0x418021bc3000 .data 0x417fc6000000 .bss 0x417fc6000970

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_0_0_0_iscsi_shim 0x418021bc4000 .data 0x417fc6400000 .bss 0x417fc6400970

2014-08-30T16:09:16.310Z cpu2:33404)random 0x418021bc5000 .data 0x417fc6800000 .bss 0x417fc6800600

2014-08-30T16:09:16.310Z cpu2:33404)usb 0x418021bc9000 .data 0x417fc6c00000 .bss 0x417fc6c01660

2014-08-30T16:09:16.310Z cpu2:33404)ehci-hcd 0x418021beb000 .data 0x417fc7000000 .bss 0x417fc70002a0

2014-08-30T16:09:16.310Z cpu2:33404)hid 0x418021bf6000 .data 0x417fc7400000 .bss 0x417fc74004e0

2014-08-30T16:09:16.310Z cpu2:33404)healthchk 0x418021bfb000 .data 0x417fc7800000 .bss 0x417fc7811e00

2014-08-30T16:09:16.310Z cpu2:33404)teamcheck 0x418021c11000 .data 0x417fc7c00000 .bss 0x417fc7c12240

2014-08-30T16:09:16.310Z cpu2:33404)vlanmtucheck 0x418021c24000 .data 0x417fc8000000 .bss 0x417fc8012000

2014-08-30T16:09:16.310Z cpu2:33404)heartbeat 0x418021c39000 .data 0x417fc8400000 .bss 0x417fc8411f00

2014-08-30T16:09:16.310Z cpu2:33404)shaper 0x418021c4a000 .data 0x417fc8800000 .bss 0x417fc8813e80

2014-08-30T16:09:16.310Z cpu2:33404)lldp 0x418021c5d000 .data 0x417fc8c00000 .bss 0x417fc8c00040

2014-08-30T16:09:16.310Z cpu2:33404)cdp 0x418021c62000 .data 0x417fc9000000 .bss 0x417fc9013400

2014-08-30T16:09:16.310Z cpu2:33404)ipfix 0x418021c7e000 .data 0x417fc9400000 .bss 0x417fc9412540

2014-08-30T16:09:16.310Z cpu2:33404)tcpip4 0x418021c92000 .data 0x417fc9800000 .bss 0x417fc9818180

2014-08-30T16:09:16.310Z cpu2:33404)dvsdev 0x418021d98000 .data 0x417fc9c00000 .bss 0x417fc9c00030

2014-08-30T16:09:16.310Z cpu2:33404)dvfilter 0x418021d9b000 .data 0x417fca000000 .bss 0x417fca000b00

2014-08-30T16:09:16.310Z cpu2:33404)lacp 0x418021dbd000 .data 0x417fca400000 .bss 0x417fca400160

2014-08-30T16:09:16.310Z cpu2:33404)hbr_filter 0x418021dc7000 .data 0x417fca800000 .bss 0x417fca800300

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_1_0_0_dvfilter_shim 0x418021df3000 .data 0x417fcac00000 .bss 0x417fcac009b0

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_0_0_0_dvfilter_shim 0x418021df4000 .data 0x417fcb000000 .bss 0x417fcb000930

2014-08-30T16:09:16.310Z cpu2:33404)dvfilter-generic-fastpath 0x418021df5000 .data 0x417fcb400000 .bss 0x417fcb412380

2014-08-30T16:09:16.310Z cpu2:33404)vmkstatelogger 0x418021e0f000 .data 0x417fcb800000 .bss 0x417fcb803a00

2014-08-30T16:09:16.310Z cpu2:33404)esxfw 0x418021e33000 .data 0x417fcbc00000 .bss 0x417fcbc12d00

2014-08-30T16:09:16.310Z cpu2:33404)dm 0x418021e48000 .data 0x417fcc000000 .bss 0x417fcc000000

2014-08-30T16:09:16.310Z cpu2:33404)nmp 0x418021e4a000 .data 0x417fcc400000 .bss 0x417fcc403e50

2014-08-30T16:09:16.310Z cpu2:33404)vmw_satp_local 0x418021e6d000 .data 0x417fcc800000 .bss 0x417fcc800028

2014-08-30T16:09:16.310Z cpu2:33404)vmw_satp_default_aa 0x418021e6f000 .data 0x417fccc00000 .bss 0x417fccc00000

2014-08-30T16:09:16.310Z cpu2:33404)vmw_psp_lib 0x418021e70000 .data 0x417fcd000000 .bss 0x417fcd000290

2014-08-30T16:09:16.310Z cpu2:33404)vmw_psp_fixed 0x418021e72000 .data 0x417fcd400000 .bss 0x417fcd400000

2014-08-30T16:09:16.310Z cpu2:33404)vmw_psp_rr 0x418021e74000 .data 0x417fcd800000 .bss 0x417fcd800068

2014-08-30T16:09:16.310Z cpu2:33404)vmw_psp_mru 0x418021e77000 .data 0x417fcdc00000 .bss 0x417fcdc00000

2014-08-30T16:09:16.310Z cpu2:33404)libata_92 0x418021e79000 .data 0x417fce000000 .bss 0x417fce002660

2014-08-30T16:09:16.310Z cpu2:33404)libata_9_2_0_0 0x418021e9b000 .data 0x417fce400000 .bss 0x417fce401750

2014-08-30T16:09:16.310Z cpu2:33404)libata_9_2_1_0 0x418021e9c000 .data 0x417fce800000 .bss 0x417fce801750

2014-08-30T16:09:16.310Z cpu2:33404)usb-storage 0x418021e9d000 .data 0x417fcec00000 .bss 0x417fcec04780

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_1_0_0_nmp_shim 0x418021ea9000 .data 0x417fcf000000 .bss 0x417fcf000ca8

2014-08-30T16:09:16.310Z cpu2:33404)vmkapi_v2_0_0_0_nmp_shim 0x418021eaa000 .data 0x417fcf400000 .bss 0x417fcf400ca8

2014-08-30T16:09:16.310Z cpu2:33404)svmmirror 0x418021eab000 .data 0x417fcf800000 .bss 0x417fcf8000c0

2014-08-30T16:09:16.310Z cpu2:33404)cbt 0x418021eb7000 .data 0x417fcfc00000 .bss 0x417fcfc00080

2014-08-30T16:09:16.310Z cpu2:33404)migrate 0x418021ebb000 .data 0x417fd0000000 .bss 0x417fd0004d40

2014-08-30T16:09:16.310Z cpu2:33404)libfc_92 0x418021f1a000 .data 0x417fd0400000 .bss 0x417fd0400b80

2014-08-30T16:09:16.310Z cpu2:33404)libfcoe_92 0x418021f33000 .data 0x417fd0800000 .bss 0x417fd08001c0

2014-08-30T16:09:16.310Z cpu2:33404)libfc_9_2_0_0 0x418021f39000 .data 0x417fd0c00000 .bss 0x417fd0c00868

2014-08-30T16:09:16.310Z cpu2:33404)libfcoe_9_2_0_0 0x418021f3a000 .data 0x417fd1000000 .bss 0x417fd10001f4

2014-08-30T16:09:16.310Z cpu2:33404)libfc_9_2_1_0 0x418021f3b000 .data 0x417fd1400000 .bss 0x417fd1400868

2014-08-30T16:09:16.310Z cpu2:33404)libfcoe_9_2_1_0 0x418021f3c000 .data 0x417fd1800000 .bss 0x417fd18001f4

2014-08-30T16:09:16.310Z cpu2:33404)ahci 0x418021f3d000 .data 0x417fd1c00000 .bss 0x417fd1c00420

2014-08-30T16:09:16.310Z cpu2:33404)sunrpc 0x418021f44000 .data 0x417fd2000000 .bss 0x417fd2002b80

2014-08-30T16:09:16.310Z cpu2:33404)nfsclient 0x418021f53000 .data 0x417fd2400000 .bss 0x417fd2403940

2014-08-30T16:09:16.310Z cpu2:33404)vmkibft 0x418021f6c000 .data 0x417fd2800000 .bss 0x417fd28037c0

2014-08-30T16:09:16.310Z cpu2:33404)lvmdriver 0x418021f6f000 .data 0x417fd2c00000 .bss 0x417fd2c03380

2014-08-30T16:09:16.310Z cpu2:33404)deltadisk 0x418021f83000 .data 0x417fd3000000 .bss 0x417fd3005c00

2014-08-30T16:09:16.310Z cpu2:33404)tracing 0x418021fae000 .data 0x417fd3400000 .bss 0x417fd3405b40

2014-08-30T16:09:16.310Z cpu2:33404)rdt 0x418021fb5000 .data 0x417fd3800000 .bss 0x417fd3804e00

2014-08-30T16:09:16.310Z cpu2:33404)vsanutil 0x418021fdb000 .data 0x417fd3c00000 .bss 0x417fd3c069c0

2014-08-30T16:09:16.310Z cpu2:33404)lsomcommon 0x418021ffa000 .data 0x417fd4000000 .bss 0x417fd4001680

2014-08-30T16:09:16.310Z cpu2:33404)plog 0x41802202e000 .data 0x417fd4400000 .bss 0x417fd44056c0

2014-08-30T16:09:16.310Z cpu2:33404)vmfs3 0x418022071000 .data 0x417fd4800000 .bss 0x417fd4803840

2014-08-30T16:09:16.310Z cpu2:33404)dvfg-igmp 0x4180220d9000 .data 0x417fd4c00000 .bss 0x417fd4c00208

2014-08-30T16:09:16.310Z cpu2:33404)cmmds_net 0x4180220df000 .data 0x417fd5000000 .bss 0x417fd5002f40

2014-08-30T16:09:16.310Z cpu2:33404)cmmds 0x4180220ec000 .data 0x417fd5400000 .bss 0x417fd5404d80

2014-08-30T16:09:16.310Z cpu2:33404)cmmds_resolver 0x418022121000 .data 0x417fd5800000 .bss 0x417fd5800140

2014-08-30T16:09:16.310Z cpu2:33404)vsan 0x41802212d000 .data 0x417fd5c00000 .bss 0x417fd5c1c200

2014-08-30T16:09:16.310Z cpu2:33404)vmklink_mpi 0x418022248000 .data 0x417fd6000000 .bss 0x417fd6002400

2014-08-30T16:09:16.310Z cpu2:33404)swapobj 0x41802224d000 .data 0x417fd6400000 .bss 0x417fd6403010

2014-08-30T16:09:16.310Z cpu2:33404)osfs 0x418022255000 .data 0x417fd6800000 .bss 0x417fd6803380

2014-08-30T16:09:16.310Z cpu2:33404)vflash 0x418022263000 .data 0x417fd6c00000 .bss 0x417fd6c03540

2014-08-30T16:09:16.310Z cpu2:33404)vfc 0x41802226e000 .data 0x417fd7000000 .bss 0x417fd7002ac0

Coredump to disk.

2014-08-30T16:09:16.360Z cpu2:33404)Slot 1 of 1.

2014-08-30T16:09:16.360Z cpu2:33404)Dump: 2212: Using dump slot size 2684354560.

Any workaround ? Disabling LACP service may help ? I did not any config about LACP .

admin · ‎09-01-2014

This is a hardware error. If I'm interpreting the machine check status correctly, there is a problem in the internal cache hierarchy. This CPU is defective and probably needs to be replaced. You should contact your hardware vendor, as indicated in the error message.

omidkosari · ‎12-15-2014

The CPUs have changed and they work for few weeks . Now one of those servers has crashed again . Is this crash related to my old problem ?

2014-09-21T11:25:10.191Z cpu3:34850)Backtrace for current CPU #3, worldID=34850, ebp=0x4123a08a7000

2014-09-21T11:25:10.191Z cpu3:34850)0x4123a08a7000:[0x410856418430]<no symbols>+0x56418430 stack: 0x0, 0x0, 0x0, 0x0, 0x0

2014-09-21T11:25:10.191Z cpu3:34850) [45m [33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64] [0m

#PF Exception 14 in world 34850:vpxa-worker IP 0x410856418430 addr 0x410856418430

PTEs:0x10041e023;0x108273063;0x8000000112ee6063;0x8000000112f0c063;

2014-09-21T11:25:10.192Z cpu3:34850)cr0=0x80010031 cr2=0x410856418430 cr3=0x24d228000 cr4=0x42768

2014-09-21T11:25:10.192Z cpu3:34850)frame=0x4123a089d860 ip=0x410856418430 err=17 rflags=0x10083

2014-09-21T11:25:10.192Z cpu3:34850)rax=0xef rbx=0x4 rcx=0x25

2014-09-21T11:25:10.192Z cpu3:34850)rdx=0x417fc14ef050 rbp=0x4123a08a7000 rsi=0x417fc1420fe0

2014-09-21T11:25:10.192Z cpu3:34850)rdi=0x4 r8=0x67c27e r9=0x1013766200000

2014-09-21T11:25:10.192Z cpu3:34850)r10=0x410856418890 r11=0x1 r12=0x3

2014-09-21T11:25:10.192Z cpu3:34850)r13=0x101376436c443 r14=0x0 r15=0x1

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:0 world:33466 name:"mclk-sched-vmnic0" (S)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:1 world:33476 name:"tq:vsanutil" (S)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:2 world:33360 name:"Tcpip4 wtask" (S)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:3 world:34850 name:"vpxa-worker" (U)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:4 world:34120 name:"hostd-worker" (U)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:5 world:33525 name:"FS3ResMgr" (S)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:6 world:33616 name:"clomd" (U)

2014-09-21T11:25:10.192Z cpu3:34850)pcpu:7 world:35603 name:"vmm0:CirrOS" (V)

2014-09-21T11:25:10.192Z cpu3:34850)@BlueScreen: #PF Exception 14 in world 34850:vpxa-worker IP 0x410856418430 addr 0x410856418430

PTEs:0x10041e023;0x108273063;0x8000000112ee6063;0x8000000112f0c063;

2014-09-21T11:25:10.192Z cpu3:34850)Code start: 0x418001400000 VMK uptime: 0:23:06:20.142

2014-09-21T11:25:10.192Z cpu3:34850)0x4123a08a7000:[0x410856418430]<no symbols>+0x56418430 stack: 0x0

2014-09-21T11:25:10.194Z cpu3:34850)base fs=0x0 gs=0x418040c00000 Kgs=0x0

2014-09-21T11:25:10.194Z cpu3:34850)vmkernel 0x0 .data 0x0 .bss 0x0

2014-09-21T11:25:10.194Z cpu3:34850)chardevs 0x418001971000 .data 0x417fc0000000 .bss 0x417fc0000400

2014-09-21T11:25:10.194Z cpu3:34850)user 0x418001978000 .data 0x417fc0400000 .bss 0x417fc040e180

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_mgmt 0x418001a2b000 .data 0x417fc0800000 .bss 0x417fc0800140

2014-09-21T11:25:10.194Z cpu3:34850)vprobe 0x418001a31000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_socket 0x418001a6f000 .data 0x417fc1000000 .bss 0x417fc10005c0

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_0_0_0_vmkernel_shim 0x418001a74000 .data 0x417fc2200000 .bss 0x417fc22080c0

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_1_0_0_vmkernel_shim 0x418001a79000 .data 0x417fc2600000 .bss 0x417fc2608840

2014-09-21T11:25:10.194Z cpu3:34850)procfs 0x418001a7e000 .data 0x417fc2a00000 .bss 0x417fc2a00240

2014-09-21T11:25:10.194Z cpu3:34850)vfat 0x418001a81000 .data 0x417fc2e00000 .bss 0x417fc2e02600

2014-09-21T11:25:10.194Z cpu3:34850)procMisc 0x418001a8b000 .data 0x417fc3200000 .bss 0x417fc3200000

2014-09-21T11:25:10.194Z cpu3:34850)vmci 0x418001a8c000 .data 0x417fc3600000 .bss 0x417fc36057c0

2014-09-21T11:25:10.194Z cpu3:34850)iodm 0x418001aaf000 .data 0x417fc3a00000 .bss 0x417fc3a00138

2014-09-21T11:25:10.194Z cpu3:34850)vmkplexer 0x418001ab3000 .data 0x417fc3e00000 .bss 0x417fc3e00260

2014-09-21T11:25:10.194Z cpu3:34850)vmklinux_9 0x418001ab7000 .data 0x417fc4200000 .bss 0x417fc4208e80

2014-09-21T11:25:10.194Z cpu3:34850)vmklinux_9_2_0_0 0x418001b3e000 .data 0x417fc4600000 .bss 0x417fc4607e84

2014-09-21T11:25:10.194Z cpu3:34850)vmklinux_9_2_1_0 0x418001b41000 .data 0x417fc4a00000 .bss 0x417fc4a07f98

2014-09-21T11:25:10.194Z cpu3:34850)vmklinux_9_2_2_0 0x418001b44000 .data 0x417fc4e00000 .bss 0x417fc4e08838

2014-09-21T11:25:10.194Z cpu3:34850)iscsi_trans 0x418001b47000 .data 0x417fc5200000 .bss 0x417fc5201800

2014-09-21T11:25:10.194Z cpu3:34850)iscsi_trans_compat_shim 0x418001b52000 .data 0x417fc5600000 .bss 0x417fc560096c

2014-09-21T11:25:10.194Z cpu3:34850)iscsi_trans_incompat_shim 0x418001b53000 .data 0x417fc5a00000 .bss 0x417fc5a007e4

2014-09-21T11:25:10.194Z cpu3:34850)etherswitch 0x418001b54000 .data 0x417fc5e00000 .bss 0x417fc5e13a00

2014-09-21T11:25:10.194Z cpu3:34850)netsched 0x418001b89000 .data 0x417fc6200000 .bss 0x417fc6204800

2014-09-21T11:25:10.194Z cpu3:34850)cnic_register 0x418001b9b000 .data 0x417fc6600000 .bss 0x417fc66001e0

2014-09-21T11:25:10.194Z cpu3:34850)e1000 0x418001b9d000 .data 0x417fc6a00000 .bss 0x417fc6a01240

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_1_0_0_iscsi_shim 0x418001bc3000 .data 0x417fc6e00000 .bss 0x417fc6e00970

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_0_0_0_iscsi_shim 0x418001bc4000 .data 0x417fc7200000 .bss 0x417fc7200970

2014-09-21T11:25:10.194Z cpu3:34850)random 0x418001bc5000 .data 0x417fc7600000 .bss 0x417fc7600600

2014-09-21T11:25:10.194Z cpu3:34850)usb 0x418001bc9000 .data 0x417fc7a00000 .bss 0x417fc7a01660

2014-09-21T11:25:10.194Z cpu3:34850)ehci-hcd 0x418001beb000 .data 0x417fc7e00000 .bss 0x417fc7e002a0

2014-09-21T11:25:10.194Z cpu3:34850)hid 0x418001bf6000 .data 0x417fc8200000 .bss 0x417fc82004e0

2014-09-21T11:25:10.194Z cpu3:34850)healthchk 0x418001bfb000 .data 0x417fc8600000 .bss 0x417fc8611e00

2014-09-21T11:25:10.194Z cpu3:34850)teamcheck 0x418001c11000 .data 0x417fc8a00000 .bss 0x417fc8a12240

2014-09-21T11:25:10.194Z cpu3:34850)vlanmtucheck 0x418001c24000 .data 0x417fc8e00000 .bss 0x417fc8e12000

2014-09-21T11:25:10.194Z cpu3:34850)heartbeat 0x418001c39000 .data 0x417fc9200000 .bss 0x417fc9211f00

2014-09-21T11:25:10.194Z cpu3:34850)shaper 0x418001c4a000 .data 0x417fc9600000 .bss 0x417fc9613e80

2014-09-21T11:25:10.194Z cpu3:34850)lldp 0x418001c5d000 .data 0x417fc9a00000 .bss 0x417fc9a00040

2014-09-21T11:25:10.194Z cpu3:34850)cdp 0x418001c62000 .data 0x417fc9e00000 .bss 0x417fc9e13400

2014-09-21T11:25:10.194Z cpu3:34850)ipfix 0x418001c7e000 .data 0x417fca200000 .bss 0x417fca212540

2014-09-21T11:25:10.194Z cpu3:34850)tcpip4 0x418001c92000 .data 0x417fca600000 .bss 0x417fca618180

2014-09-21T11:25:10.194Z cpu3:34850)dvsdev 0x418001d98000 .data 0x417fcaa00000 .bss 0x417fcaa00030

2014-09-21T11:25:10.194Z cpu3:34850)dvfilter 0x418001d9b000 .data 0x417fcae00000 .bss 0x417fcae00b00

2014-09-21T11:25:10.194Z cpu3:34850)lacp 0x418001dbd000 .data 0x417fcb200000 .bss 0x417fcb200160

2014-09-21T11:25:10.194Z cpu3:34850)hbr_filter 0x418001dc7000 .data 0x417fcb600000 .bss 0x417fcb600300

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_1_0_0_dvfilter_shim 0x418001df3000 .data 0x417fcba00000 .bss 0x417fcba009b0

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_0_0_0_dvfilter_shim 0x418001df4000 .data 0x417fcbe00000 .bss 0x417fcbe00930

2014-09-21T11:25:10.194Z cpu3:34850)dvfilter-generic-fastpath 0x418001df5000 .data 0x417fcc200000 .bss 0x417fcc212380

2014-09-21T11:25:10.194Z cpu3:34850)vmkstatelogger 0x418001e0f000 .data 0x417fcc600000 .bss 0x417fcc603a00

2014-09-21T11:25:10.194Z cpu3:34850)esxfw 0x418001e33000 .data 0x417fcca00000 .bss 0x417fcca12d00

2014-09-21T11:25:10.194Z cpu3:34850)dm 0x418001e48000 .data 0x417fcce00000 .bss 0x417fcce00000

2014-09-21T11:25:10.194Z cpu3:34850)nmp 0x418001e4a000 .data 0x417fcd200000 .bss 0x417fcd203e50

2014-09-21T11:25:10.194Z cpu3:34850)vmw_satp_local 0x418001e6d000 .data 0x417fcd600000 .bss 0x417fcd600028

2014-09-21T11:25:10.194Z cpu3:34850)vmw_satp_default_aa 0x418001e6f000 .data 0x417fcda00000 .bss 0x417fcda00000

2014-09-21T11:25:10.194Z cpu3:34850)vmw_psp_lib 0x418001e70000 .data 0x417fcde00000 .bss 0x417fcde00290

2014-09-21T11:25:10.194Z cpu3:34850)vmw_psp_fixed 0x418001e72000 .data 0x417fce200000 .bss 0x417fce200000

2014-09-21T11:25:10.194Z cpu3:34850)vmw_psp_rr 0x418001e74000 .data 0x417fce600000 .bss 0x417fce600068

2014-09-21T11:25:10.194Z cpu3:34850)vmw_psp_mru 0x418001e77000 .data 0x417fcea00000 .bss 0x417fcea00000

2014-09-21T11:25:10.194Z cpu3:34850)libata_92 0x418001e79000 .data 0x417fcee00000 .bss 0x417fcee02660

2014-09-21T11:25:10.194Z cpu3:34850)libata_9_2_0_0 0x418001e9b000 .data 0x417fcf200000 .bss 0x417fcf201750

2014-09-21T11:25:10.194Z cpu3:34850)libata_9_2_1_0 0x418001e9c000 .data 0x417fcf600000 .bss 0x417fcf601750

2014-09-21T11:25:10.194Z cpu3:34850)usb-storage 0x418001e9d000 .data 0x417fcfa00000 .bss 0x417fcfa04780

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_1_0_0_nmp_shim 0x418001ea9000 .data 0x417fcfe00000 .bss 0x417fcfe00ca8

2014-09-21T11:25:10.194Z cpu3:34850)vmkapi_v2_0_0_0_nmp_shim 0x418001eaa000 .data 0x417fd0200000 .bss 0x417fd0200ca8

2014-09-21T11:25:10.194Z cpu3:34850)svmmirror 0x418001eab000 .data 0x417fd0600000 .bss 0x417fd06000c0

2014-09-21T11:25:10.194Z cpu3:34850)cbt 0x418001eb7000 .data 0x417fd0a00000 .bss 0x417fd0a00080

2014-09-21T11:25:10.194Z cpu3:34850)migrate 0x418001ebb000 .data 0x417fd0e00000 .bss 0x417fd0e04d40

2014-09-21T11:25:10.194Z cpu3:34850)libfc_92 0x418001f1a000 .data 0x417fd1200000 .bss 0x417fd1200b80

2014-09-21T11:25:10.194Z cpu3:34850)libfcoe_92 0x418001f33000 .data 0x417fd1600000 .bss 0x417fd16001c0

2014-09-21T11:25:10.194Z cpu3:34850)libfc_9_2_0_0 0x418001f39000 .data 0x417fd1a00000 .bss 0x417fd1a00868

2014-09-21T11:25:10.194Z cpu3:34850)libfcoe_9_2_0_0 0x418001f3a000 .data 0x417fd1e00000 .bss 0x417fd1e001f4

2014-09-21T11:25:10.194Z cpu3:34850)libfc_9_2_1_0 0x418001f3b000 .data 0x417fd2200000 .bss 0x417fd2200868

2014-09-21T11:25:10.194Z cpu3:34850)libfcoe_9_2_1_0 0x418001f3c000 .data 0x417fd2600000 .bss 0x417fd26001f4

2014-09-21T11:25:10.194Z cpu3:34850)ahci 0x418001f3d000 .data 0x417fd2a00000 .bss 0x417fd2a00420

2014-09-21T11:25:10.194Z cpu3:34850)sunrpc 0x418001f44000 .data 0x417fd2e00000 .bss 0x417fd2e02b80

2014-09-21T11:25:10.194Z cpu3:34850)nfsclient 0x418001f53000 .data 0x417fd3200000 .bss 0x417fd3203940

2014-09-21T11:25:10.194Z cpu3:34850)vmkibft 0x418001f6c000 .data 0x417fd3600000 .bss 0x417fd36037c0

2014-09-21T11:25:10.194Z cpu3:34850)lvmdriver 0x418001f6f000 .data 0x417fd3a00000 .bss 0x417fd3a03380

2014-09-21T11:25:10.194Z cpu3:34850)deltadisk 0x418001f83000 .data 0x417fd3e00000 .bss 0x417fd3e05c00

2014-09-21T11:25:10.194Z cpu3:34850)tracing 0x418001fae000 .data 0x417fd4200000 .bss 0x417fd4205b40

2014-09-21T11:25:10.194Z cpu3:34850)rdt 0x418001fb5000 .data 0x417fd4600000 .bss 0x417fd4604e00

2014-09-21T11:25:10.194Z cpu3:34850)vsanutil 0x418001fdb000 .data 0x417fd4a00000 .bss 0x417fd4a069c0

2014-09-21T11:25:10.194Z cpu3:34850)lsomcommon 0x418001ffa000 .data 0x417fd4e00000 .bss 0x417fd4e01680

2014-09-21T11:25:10.194Z cpu3:34850)plog 0x41800202e000 .data 0x417fd5200000 .bss 0x417fd52056c0

2014-09-21T11:25:10.194Z cpu3:34850)vmfs3 0x418002071000 .data 0x417fd5600000 .bss 0x417fd5603840

2014-09-21T11:25:10.194Z cpu3:34850)dvfg-igmp 0x4180020d9000 .data 0x417fd5a00000 .bss 0x417fd5a00208

2014-09-21T11:25:10.194Z cpu3:34850)cmmds_net 0x4180020df000 .data 0x417fd5e00000 .bss 0x417fd5e02f40

2014-09-21T11:25:10.194Z cpu3:34850)cmmds 0x4180020ec000 .data 0x417fd6200000 .bss 0x417fd6204d80

2014-09-21T11:25:10.194Z cpu3:34850)cmmds_resolver 0x418002121000 .data 0x417fd6600000 .bss 0x417fd6600140

2014-09-21T11:25:10.194Z cpu3:34850)vsan 0x41800212d000 .data 0x417fd6a00000 .bss 0x417fd6a1c200

2014-09-21T11:25:10.194Z cpu3:34850)vmklink_mpi 0x418002248000 .data 0x417fd6e00000 .bss 0x417fd6e02400

2014-09-21T11:25:10.194Z cpu3:34850)swapobj 0x41800224d000 .data 0x417fd7200000 .bss 0x417fd7203010

2014-09-21T11:25:10.194Z cpu3:34850)osfs 0x418002255000 .data 0x417fd7600000 .bss 0x417fd7603380

2014-09-21T11:25:10.194Z cpu3:34850)vflash 0x418002263000 .data 0x417fd7a00000 .bss 0x417fd7a03540

2014-09-21T11:25:10.194Z cpu3:34850)vfc 0x41800226e000 .data 0x417fd7e00000 .bss 0x417fd7e02ac0

JarryG · ‎12-15-2014

This is probably hardware-error too, but of different kind. "Exception 14" has something to do with memory-paging:

VMware KB: Understanding Exception 13 and Exception 14 purple diagnostic screen events in ESX 3.x/4....

What HW are you using? Your servers look like "monday's products" (the worst quality of the whole week's production batch)...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! 😉

Alistar · ‎12-15-2014

Hi, this might indicate a problem with memory or the motherboard itself - if you use Windows in your environment, check out my articles Stress Testing an ESXi Host with Windows Server VMs | VMXP and Debugging Machine Check Errors (MCEs) | VMXP. If your ESXi gets filled with MCEs in vmkernel log during the stress test and crashes again, you can decode what the MCE meant and have the faulty parts replaced.

Good luck!

Stop by my blog if you'd like 🙂 I dabble in vSphere troubleshooting, PowerCLI scripting and NetApp storage - and I share my journeys at http://vmxp.wordpress.com/

All

5 ESXi same hardware . 2 of them crash