Kernel panic in OSX guest on Fusion 10.0.1 when configured for 3 or more CPU's

[raiford]

I have had a lot of kernel panics when booting OSX guests if I configure 3 or more cpus.  It never panics with 1 or 2 cpus, rarely happens with 3 cpus, and almost always happens with 4 or more cpus.  Stragely it does eventually boot after many failed attempts.

I don't see the problem on an ESXi server so I'm guessing it may point to a hardware problem on my laptop (Late 2016 15" MBP w/ touchbar), but the machine passes all Apple hardware diagnostics.  Is it possible this a known problem with Fusion?  Is it possible to set cpu affinity to prevent Fusion from using certain cpu cores?  I believe I have a directory full of boot failure logs but I forget where they are written.  Should I attach a bunch of them?

Thanks in advance!

Edit:  The same behavior occurs on Fusion 10.1.0 as well as 10.1.1.  It appears that several other people experience the same issue, so it may indeed be a bug in Fusion.

[alexlissy]

Of course, you are right. But running into kernel panic with one single VM configured with 4 CPUs, 8 GB RAM on a i7 7920HQ with 16 GB RAM shows there is something bad going on.

[Mikero]

I agree, it should just perform more slowly in the case of over provisioning/resourcing, not panic.

[ColoradoMarmot]

Yeah, very strange bug.  I wonder if Apple's hard coded something in there that expects specific physical hardware.

I wish they'd spend at least a bit of the repatriated funds on some enhanced QA!

[raiford]

Mikero​, can you tell us if there has been progress with this?  Have you been able to identify why it is happening?  Thanks.

[Mikero]

We've filed a radar with Apple, but I can't comment on their progress. It's a high priority, we don't like crashes at the kernel level... because ESX

[kevm77]

Having numerous kernel panics while restarting...

Running latest sierra as guest on latest sierra running latest vmware fusion

With 4 processor cores in iMac Pro 8 core/ 32GB machine.  Seems more stable with 2 cores allocated

Kevin

[alexlissy]

Little ping. While you might not be allowed to share the whole status, would it be possible for you to advise if I can safely buy new (more powerful) hardware, with the expectation that a fix / workaround (other than limiting to 2 cores, of course) will be available sooner rather than later ? Let's say on a two to three months timeframe.

[bbastion]

Fusion 10.1.1 (7520154) - SAME PROBLEM. Running 10.12.6 Sierra GUEST on a 10.12.6 Sierra HOST. Got it with 4 processors. Turned it down to 2 processors and it booted, but this is really problematic.

Definitely a VMware problem. PANIC BELOW:

*** Panic Report ***

panic(cpu 3 caller 0xffffff801b5362f8): "a freed zone element has been modified in zone kalloc.48: expected 0xc0ffeed4a7f26ce2 but found 0, bits changed 0xc0ffeed4a7f26ce2, at offset 0 of 48 in element 0xffffff8023cbb990, cookies 0x3f0011548439da72 0x53521720a438595"@/Library/Caches/com.apple.xbs/Sources/xnu/xnu-3789.70.16/osfmk/kern/zalloc.c:650

Backtrace (CPU 3), Frame : Return Address

0xffffff887bd33500 : 0xffffff801b4e953c

0xffffff887bd33580 : 0xffffff801b5362f8

0xffffff887bd335f0 : 0xffffff801b535549

0xffffff887bd33640 : 0xffffff801b533f84

0xffffff887bd33790 : 0xffffff801ba35de7

0xffffff887bd337c0 : 0xffffff801ba527ba

0xffffff887bd337f0 : 0xffffff801ba88fba

0xffffff887bd33820 : 0xffffff7f9c157661

0xffffff887bd33860 : 0xffffff7f9c156e99

0xffffff887bd339f0 : 0xffffff7f9c168d8c

0xffffff887bd33ac0 : 0xffffff801babd76a

0xffffff887bd33b30 : 0xffffff7f9c167b46

0xffffff887bd33b60 : 0xffffff7f9cbf4e9c

0xffffff887bd33c50 : 0xffffff7f9cbf07e8

0xffffff887bd33c80 : 0xffffff7f9cbe4542

0xffffff887bd33ca0 : 0xffffff801babd76a

0xffffff887bd33d10 : 0xffffff7f9cbe44f8

0xffffff887bd33d20 : 0xffffff7f9caf173a

0xffffff887bd33e90 : 0xffffff7f9caf10e8

0xffffff887bd33f30 : 0xffffff7f9cb2121f

0xffffff887bd33f50 : 0xffffff7f9cb211ce

0xffffff887bd33f70 : 0xffffff7f9cb21244

0xffffff887bd33f90 : 0xffffff7f9cb21346

0xffffff887bd33fb0 : 0xffffff801b4978f7

      Kernel Extensions in backtrace:

         com.apple.iokit.IOUSBHostFamily(1.1)[896273C6-36AE-30AC-9274-1BCB7C9A4062]@0xffffff7f9c13b000->0xffffff7f9c1a3fff

            dependency: com.apple.driver.AppleBusPowerController(1.0)[DB526B45-1A45-3A81-A0C1-57F826CADEDF]@0xffffff7f9c12e000

         com.apple.iokit.IOBluetoothFamily(5.0.5f1)[5AA0EAB8-6128-3B0C-B8CE-293BDB9D21A8]@0xffffff7f9caeb000->0xffffff7f9cbcefff

            dependency: com.apple.iokit.IOReportFamily(31)[B14DC3D3-7250-3DA3-BF50-C666EBEDAF4C]@0xffffff7f9c915000

         com.apple.iokit.IOBluetoothHostControllerTransport(5.0.5f1)[DAC3389D-A9B3-3E1E-8D1E-8D72CE96BBC1]@0xffffff7f9cbe3000->0xffffff7f9cbedfff

            dependency: com.apple.iokit.IOBluetoothFamily(5.0.5f1)[5AA0EAB8-6128-3B0C-B8CE-293BDB9D21A8]@0xffffff7f9caeb000

            dependency: com.apple.iokit.IOACPIFamily(1.4)[4F7FB6AD-2498-3F71-827C-ED7AA4BF2511]@0xffffff7f9c125000

            dependency: com.apple.iokit.IOPCIFamily(2.9)[3E00E7D2-E569-341D-9BE0-34D5DE491825]@0xffffff7f9bd32000

            dependency: com.apple.iokit.IOUSBHostFamily(1.1)[896273C6-36AE-30AC-9274-1BCB7C9A4062]@0xffffff7f9c13b000

         com.apple.iokit.IOBluetoothHostControllerUSBTransport(5.0.5f1)[98323CF2-1639-3312-A334-0E82F99359D3]@0xffffff7f9cbee000->0xffffff7f9cc10fff

            dependency: com.apple.iokit.IOBluetoothFamily(5.0.5f1)[5AA0EAB8-6128-3B0C-B8CE-293BDB9D21A8]@0xffffff7f9caeb000

            dependency: com.apple.iokit.IOACPIFamily(1.4)[4F7FB6AD-2498-3F71-827C-ED7AA4BF2511]@0xffffff7f9c125000

            dependency: com.apple.iokit.IOPCIFamily(2.9)[3E00E7D2-E569-341D-9BE0-34D5DE491825]@0xffffff7f9bd32000

            dependency: com.apple.iokit.IOBluetoothHostControllerTransport(5.0.5f1)[DAC3389D-A9B3-3E1E-8D1E-8D72CE96BBC1]@0xffffff7f9cbe3000

            dependency: com.apple.iokit.IOUSBHostFamily(1.1)[896273C6-36AE-30AC-9274-1BCB7C9A4062]@0xffffff7f9c13b000

BSD process name corresponding to current thread: kernel_task

Mac OS version:

16G29

Kernel version:

Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64

Kernel UUID: D3314D98-5D40-3CD8-98A4-F1DD46C20E03

Kernel slide:     0x000000001b200000

Kernel text base: 0xffffff801b400000

__HIB  text base: 0xffffff801b300000

System model name: VMware7,1 (Mac-27ADBB7B4CEE8E61)

System uptime in nanoseconds: 4012925077

last loaded kext at 4004288890: com.apple.driver.pmtelemetry 1 (addr 0xffffff7f9c104000, size 90112)

loaded kexts:

com.apple.driver.pmtelemetry 1

com.apple.iokit.IOUserEthernet 1.0.1

com.apple.iokit.IOBluetoothSerialManager 5.0.5f1

com.apple.Dont_Steal_Mac_OS_X 7.0.0

com.apple.driver.AppleHV 1

com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0

com.apple.driver.AppleOSXWatchdog 1

com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0

com.apple.driver.AppleFIVRDriver 4.1.0

com.apple.iokit.SCSITaskUserClient 394.50.1

com.apple.driver.AppleFileSystemDriver 3.0.1

com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1

com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0

com.apple.BootCache 40

com.apple.filesystems.hfs.kext 366.70.1

com.apple.iokit.IOAHCIBlockStorage 295.20.1

com.apple.driver.AppleAHCIPort 326.60.1

com.apple.driver.Intel82574LEthernet 2.7.2

com.apple.driver.AppleIntelPIIXATA 2.5.1

com.apple.driver.AppleACPIButtons 5.0

com.apple.driver.AppleHPET 1.8

com.apple.driver.AppleRTC 2.0

com.apple.driver.AppleSMBIOS 2.1

com.apple.driver.AppleAPIC 1.7

com.apple.nke.applicationfirewall 172

com.apple.security.quarantine 3

com.apple.security.TMSafetyNet 8

com.apple.iokit.IOSurface 159.9

com.apple.iokit.IOSerialFamily 11

com.apple.driver.AppleSSE 1.0

com.apple.driver.IOPlatformPluginLegacy 1.0.0

com.apple.driver.IOPlatformPluginFamily 6.0.0d8

com.apple.iokit.IOBluetoothHostControllerUSBTransport 5.0.5f1

com.apple.iokit.IOBluetoothHostControllerTransport 5.0.5f1

com.apple.iokit.IOBluetoothFamily 5.0.5f1

com.apple.driver.AppleHDAController 279.48

com.apple.iokit.IOGraphicsFamily 515.3

com.apple.iokit.IOHDAFamily 279.48

com.apple.iokit.IOAudioFamily 205.15

com.apple.vecLib.kext 1.2.0

com.apple.driver.AppleSMC 3.1.9

com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0

com.apple.driver.usb.AppleUSBHub 1.1

com.apple.driver.usb.IOUSBHostHIDDevice 1.1

com.apple.driver.AppleUSBHostMergeProperties 1.1

com.apple.driver.usb.cdc 5.0.0

com.apple.driver.usb.networking 5.0.0

com.apple.driver.usb.AppleUSBHostCompositeDevice 1.1

com.apple.iokit.IOSCSIMultimediaCommandsDevice 394.50.1

com.apple.iokit.IOBDStorageFamily 1.8

com.apple.iokit.IODVDStorageFamily 1.8

com.apple.iokit.IOCDStorageFamily 1.8

com.apple.filesystems.hfs.encodings.kext 1

com.apple.driver.AppleXsanScheme 3

com.apple.driver.usb.AppleUSBHostPacketFilter 1.0

com.apple.iokit.IOUSBFamily 900.4.1

com.apple.iokit.IOAHCISerialATAPI 266.50.1

com.apple.iokit.IOSCSIArchitectureModelFamily 394.50.1

com.apple.driver.usb.AppleUSBXHCIPCI 1.1

com.apple.driver.usb.AppleUSBXHCI 1.1

com.apple.driver.usb.AppleUSBUHCIPCI 1.1

com.apple.driver.usb.AppleUSBUHCI 1.1

com.apple.iokit.IOAHCIFamily 288

com.apple.driver.usb.AppleUSBEHCIPCI 1.1

com.apple.driver.usb.AppleUSBEHCI 1.1

com.apple.iokit.IONetworkingFamily 3.2

com.apple.iokit.IOATAFamily 2.5.3

com.apple.driver.AppleEFINVRAM 2.1

com.apple.driver.AppleEFIRuntime 2.1

com.apple.iokit.IOHIDFamily 2.0.0

com.apple.security.sandbox 300.0

com.apple.kext.AppleMatch 1.0.0d1

com.apple.driver.AppleKeyStore 2

com.apple.driver.AppleMobileFileIntegrity 1.0.5

com.apple.driver.AppleCredentialManager 1.0

com.apple.driver.KernelRelayHost 1

com.apple.iokit.IOUSBHostFamily 1.1

com.apple.driver.AppleBusPowerController 1.0

com.apple.driver.DiskImages 444.50.16

com.apple.iokit.IOStorageFamily 2.1

com.apple.iokit.IOReportFamily 31

com.apple.driver.AppleFDEKeyStore 28.30

com.apple.driver.AppleACPIPlatform 5.0

com.apple.iokit.IOPCIFamily 2.9

com.apple.iokit.IOACPIFamily 1.4

com.apple.kec.Libm 1

com.apple.kec.pthread 1

com.apple.kec.corecrypto 1.0

[ColoradoMarmot]

It sounds like it's an Apple bug, so VMWare's at their mercy.

[Mikero]

Yah, it's one of those 'do we try to fix it (and then possibly have to un-fix it later) or wait for them to fix it?'

For now, if this is the same underlying cause as the RADAR we've filed, the issue can be mitigated by enabling 'code profiling' in the VM.

• Power off the virtual machine
• go into VM > Settings... > "Processors & Memory"
• Expand "Advanced options"
• Select "Enable code profiling applications in this virtual machine".
• Close the Settings window and power on the virtual machine.

The failure should no longer occur.

edit:
This is not the same issue, my bad!

The issue corrected with the fix I describe is for 10.14 beta crashing when doing a sysctl -a.  (same boat, Apple changed a thing, it breaks us)

[alexlissy]

Sure, but at some point, i'm not the one in touch with Apple, so it's my only way to: get status update, get potential workaround informations. I'm being blocked on that for going forward on some work, for weeks now.

[alexlissy]

One more month, no news. Either VMware or Apple does not care at all.

[ColoradoMarmot]

Once again, this is an Apple bug.  Apple doesn't comment on, nor release expected fix dates.

[raiford]

@Mikero,  since we are approaching 6 months without a fix from Apple are we nearing that threshold where VMWare chooses to go down the path of 'do we try to fix it (and then possibly have to un-fix it later)'?  It does appear that this is a hardware related issue since some machines don't show the problem.  Hopefully it is something that could be mitigated from your side?  Oddly enough, once one of these VM's manages to start it seems rock solid so I imagine the kludge would only need to be in affect during initial start up.

[Mikero · Accepted solution]

Without going into much detail, we think this is fixed in 10.14.

We're keeping an eye on the situation, but we haven't been seeing this with our 10.14 testing. (tho it's early and we have plenty more testing to do)

We would be very interested in hearing if folks still have the same behavior with the new beta.

[alexlissy]

This upgrade to 10.14 is needed on the host side ? Or on the VM side ? Or both ?

[alexlissy]

Mikero ping?

[raiford]

Mikero​, I have some interesting news to report.  I just upgraded to Fusion 10.1.3 and installed a fresh install of MacOS 10.13 (High Sierra) + the current updates.  I have been able to consistently boot this VM with no kernel panics with 4 CPU's configured.  I haven't tried other MacOS versions, but this is very encouraging.  This is the same host computer that I was using when I first started this thread, so the hardware hasn't changed.

[alexlissy]

I've just upgraded to VMware Fuison 10.1.3 but kept the Sierra and not High Sierra host, set the number of VCPUs to 4, and it is still panicking. Either you are more lucky than I am, or there's some undocumented hotfix in High Sierra with all the updates and/or Fusion 10.1.3 is required as well. But just the VMware upgrade is not enough :/.

[raiford]

For what it's worth, my host is running macOS Sierra (10.12.6), not High Sierra.  It is the VM that I set up that is High Sierra.  My guess is that Apple corrected the issue in High Sierra so that it doesn't panic any more when it boots.  We likely have to wait for a future version of Fusion to fix the issue for other guest OS's?