Guest machine crashing during ChefBuild

[BrokenMouse]

Hey everyone.

Currently I'm running a VM that has Mac OSX 10.13.4 and Chef 13.8

It's running 8 Cores with 8gbs of RAM and 70gb HDD.

I'm not sure why, but whenever I run a zero instance of Chef the machine just crashes.  I've attached the logs to help better understand what's going on. I can run the chef commands on the host machine just fine.

If you need more info let me know,

Thank you.

[Mikero]

If you reduce the number of cores in the VM to 2 or less, does it crash?

[BrokenMouse]

Thanks Mikero,

EDIT: After another crash I was able to run it, however it the build failed due to a bad piece of code. Once I fixed it, the machine crashed again..

It did crash, but I was able to see at least the Chef 13.8.5 message before it did. I also lowered the ram from 8gbs to 2.

Anonymous UUID:       B21644EA-D72B-F07B-0101-914C5CA43195

Thu Apr 12 15:27:06 2018

*** Panic Report ***

panic(cpu 1 caller 0xffffff800b388c2d): Kernel trap at 0xffffff800b34ef2c, type 13=general protection, registers:

CR0: 0x0000000080010033, CR2: 0x000000010ea23000, CR3: 0x000000004484e00c, CR4: 0x00000000003606e0

RAX: 0x0000000000000000, RBX: 0x0000000000000001, RCX: 0x000000000000038d, RDX: 0x0000000000000000

RSP: 0xffffff806ae93930, RBP: 0xffffff806ae93950, RSI: 0xffffff806ae93970, RDI: 0x0000000000000001

R8:  0x0000000000000000, R9:  0x0000000000000000, R10: 0xffffff806ae93b90, R11: 0xffffff7f8c2fa800

R12: 0x0000000000000000, R13: 0x0000000000000001, R14: 0xffffff806ae93970, R15: 0x0000000000000000

RFL: 0x0000000000010202, RIP: 0xffffff800b34ef2c, CS:  0x0000000000000008, SS:  0x0000000000000010

Fault CR2: 0x000000010ea23000, Error code: 0x0000000000000000, Fault CPU: 0x1 VMM, PL: 0, VF: 0

Backtrace (CPU 1), Frame : Return Address

0xffffff800b14ca70 : 0xffffff800b26e166

0xffffff800b14cac0 : 0xffffff800b396714

0xffffff800b14cb00 : 0xffffff800b388a00

0xffffff800b14cb80 : 0xffffff800b220180

0xffffff800b14cba0 : 0xffffff800b26dbdc

0xffffff800b14ccd0 : 0xffffff800b26d99c

0xffffff800b14cd30 : 0xffffff800b388c2d

0xffffff800b14ceb0 : 0xffffff800b220180

0xffffff800b14ced0 : 0xffffff800b34ef2c

0xffffff806ae93950 : 0xffffff800b7af453

0xffffff806ae93d90 : 0xffffff800b734cb6

0xffffff806ae93e10 : 0xffffff800b734ee1

0xffffff806ae93f40 : 0xffffff800b804418

0xffffff806ae93fa0 : 0xffffff800b220966

BSD process name corresponding to current thread: sysctl

Mac OS version:

17E199

Kernel version:

Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST 2018; root:xnu-4570.51.1~1/RELEASE_X86_64

Kernel UUID: 1B55340B-0B14-3026-8A47-1E139DB63DA3

Kernel slide:     0x000000000b000000

Kernel text base: 0xffffff800b200000

__HIB  text base: 0xffffff800b100000

System model name: MacBookPro13,3 (440BX Desktop Reference Platform)

System uptime in nanoseconds: 176039429891

last loaded kext at 19641910618: org.virtualbox.kext.VBoxNetAdp 5.2.8 (addr 0xffffff7f8d68e000, size 28672)

loaded kexts:

org.virtualbox.kext.VBoxNetAdp 5.2.8

org.virtualbox.kext.VBoxNetFlt 5.2.8

com.carbonblack.CbOsxSensorNetmon 1801.24.61fc3

com.vmware.kext.vmhgfs 0725.33.23

org.virtualbox.kext.VBoxUSB 5.2.8

com.vmware.kext.vmmemctl 0725.33.23

org.virtualbox.kext.VBoxDrv 5.2.8

com.carbonblack.CbOsxSensorProcmon 1801.24.61fc3

com.bit9.cbsystemproxy 1801.24.61fc3

com.vmware.kext.VMwareGfx 0725.33.23

com.apple.filesystems.autofs 3.0

com.apple.driver.AppleTyMCEDriver 1.0.3d2

com.apple.driver.AppleGraphicsDevicePolicy 3.18.48

com.apple.AGDCPluginDisplayMetrics 3.18.48

com.apple.driver.AppleHV 1

com.apple.iokit.IOUserEthernet 1.0.1

com.apple.iokit.IOBluetoothSerialManager 6.0.5f3

com.apple.driver.AppleUpstreamUserClient 3.6.5

com.apple.driver.AppleMCCSControl 1.5.4

com.apple.driver.pmtelemetry 1

com.apple.Dont_Steal_Mac_OS_X 7.0.0

com.apple.driver.AppleOSXWatchdog 1

com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0

com.apple.driver.AppleFIVRDriver 4.1.0

com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0

com.apple.driver.AppleFileSystemDriver 3.0.1

com.apple.filesystems.hfs.kext 407.50.6

com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1

com.apple.BootCache 40

com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0

com.apple.iokit.SCSITaskUserClient 404.30.2

com.apple.filesystems.apfs 748.51.0

com.apple.driver.AppleAHCIPort 329.50.2

com.apple.driver.Intel82574LEthernet 2.7.2

com.apple.driver.AppleACPIButtons 6.1

com.apple.driver.AppleHPET 1.8

com.apple.driver.AppleRTC 2.0

com.apple.driver.AppleSMBIOS 2.1

com.apple.driver.AppleAPIC 1.7

com.apple.nke.applicationfirewall 183

com.apple.security.TMSafetyNet 8

com.apple.security.quarantine 3

com.apple.iokit.IOUSBUserClient 900.4.1

com.apple.driver.AppleUSBHostMergeProperties 1.2

com.apple.kext.triggers 1.0

com.apple.driver.AppleGraphicsControl 3.18.48

com.apple.iokit.IOAVBFamily 675.6

com.apple.plugin.IOgPTPPlugin 675.12

com.apple.iokit.IOEthernetAVBController 1.1.0

com.apple.driver.AppleSSE 1.0

com.apple.iokit.IOSurface 211.12

com.apple.iokit.IOSerialFamily 11

com.apple.driver.AppleSMBusController 1.0.18d1

com.apple.iokit.IOSMBusFamily 1.1

com.apple.iokit.IOBluetoothFamily 6.0.5f3

com.apple.AppleGPUWrangler 3.18.48

com.apple.AppleGraphicsDeviceControl 3.18.48

com.apple.driver.IOPlatformPluginLegacy 1.0.0

com.apple.driver.IOPlatformPluginFamily 6.0.0d8

com.apple.iokit.IONDRVSupport 519.15

com.apple.iokit.IOGraphicsFamily 519.15

com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0

com.apple.driver.usb.IOUSBHostHIDDevice 1.2

com.apple.driver.usb.cdc 5.0.0

com.apple.driver.usb.networking 5.0.0

com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2

com.apple.driver.usb.AppleUSBHub 1.2

com.apple.filesystems.hfs.encodings.kext 1

com.apple.driver.usb.AppleUSBHostPacketFilter 1.0

com.apple.iokit.IOUSBFamily 900.4.1

com.apple.iokit.IOSCSIMultimediaCommandsDevice 404.30.2

com.apple.iokit.IOBDStorageFamily 1.8

com.apple.iokit.IODVDStorageFamily 1.8

com.apple.iokit.IOCDStorageFamily 1.8

com.apple.driver.AppleXsanScheme 3

com.apple.iokit.IOAHCISerialATAPI 267.50.1

com.apple.iokit.IOAHCIBlockStorage 301.40.2

com.apple.driver.usb.AppleUSBUHCIPCI 1.2

com.apple.driver.usb.AppleUSBUHCI 1.2

com.apple.iokit.IOAHCIFamily 288

com.apple.driver.usb.AppleUSBEHCIPCI 1.2

com.apple.driver.usb.AppleUSBEHCI 1.2

com.apple.driver.AppleEFINVRAM 2.1

com.apple.driver.AppleEFIRuntime 2.1

com.apple.iokit.IOHIDFamily 2.0.0

com.apple.security.sandbox 300.0

com.apple.kext.AppleMatch 1.0.0d1

com.apple.driver.DiskImages 480.50.10

com.apple.driver.AppleFDEKeyStore 28.30

com.apple.driver.AppleEffaceableStorage 1.0

com.apple.driver.AppleKeyStore 2

com.apple.driver.AppleUSBTDM 439.50.6

com.apple.driver.AppleMobileFileIntegrity 1.0.5

com.apple.iokit.IOUSBMassStorageDriver 140.50.3

com.apple.iokit.IOSCSIBlockCommandsDevice 404.30.2

com.apple.iokit.IOSCSIArchitectureModelFamily 404.30.2

com.apple.iokit.IOStorageFamily 2.1

com.apple.driver.AppleCredentialManager 1.0

com.apple.driver.KernelRelayHost 1

com.apple.iokit.IOUSBHostFamily 1.2

com.apple.driver.usb.AppleUSBCommon 1.0

com.apple.driver.AppleBusPowerController 1.0

com.apple.driver.AppleSEPManager 1.0.1

com.apple.driver.IOSlaveProcessor 1

com.apple.iokit.IOReportFamily 31

com.apple.iokit.IOTimeSyncFamily 675.12

com.apple.iokit.IONetworkingFamily 3.4

com.apple.driver.AppleACPIPlatform 6.1

com.apple.driver.AppleSMC 3.1.9

com.apple.iokit.IOPCIFamily 2.9

com.apple.iokit.IOACPIFamily 1.4

com.apple.kec.pthread 1

com.apple.kec.Libm 1

com.apple.kec.corecrypto 1.0

EOF

Model: MacBookPro13,3, BootROM VMW71.00V.6997262.B64.1710270607, 2 processors, 2.77 GHz, 2 GB, SMC 2.8f0

Graphics: Display, 128 MB

Memory Module: RAM slot #0/RAM slot #0, 2 GB, DRAM, 0 MHz, VMware Virtual RAM, VMW-2048MB

Network Service: Ethernet, Ethernet, en0

Serial ATA Device: VMware Virtual SATA Hard Drive, 75.16 GB

Serial ATA Device: VMware Virtual SATA CDRW Drive

USB Device: USB Bus

USB Device: VMware Virtual USB Hub

USB Device: VMware Virtual USB Keyboard

USB Device: VMware Virtual USB Mouse

USB Device: USB 2.0 Bus

Thunderbolt Bus:

[Mikero]

hmmm, I see the virtual box kext is loaded, try unloading them and seeing if we can still crash?

There is an open issue with Apple (Radar bug filed) regarding kernel panics in macOS Guests with 3 or more CPU cores allocated, so my first thought was that it could be related, but I do know that virtualbox doesn't unload it's kexts when it exits and that has caused problems before, but when working at the UI and we handle that a little more gracefully.

I could see it crashing 'less-gracefully' if we're automating things with chef and there isn't as much UI involvement.

[BrokenMouse]

I unloaded the Kexts from VirtualBox and I'm still getting the same error.

[Mikero]

Hmmm... dariusd​, any thoughts here?

[dariusd · Accepted solution]

Ewwww... It looks like Apple has made some unsafe assumptions about the capabilities of the virtualized processor.  The guest OS is attempting to query the processor's Performance Monitoring Counter (PMC) support, but the VM's current configuration does not support PMCs, and as far as I can tell the guest OS is failing to handle the situation correctly.

Let's see if we can work around it.  Ensure that the guest OS is fully shut down, then follow the instructions in this documentation page (Set Advanced Processor Options) and select the option to Enable code profiling applications in this virtual machine, then try again.  I hope it will work, but it is possible that it will still fail...  Please let us know how it goes.

Cheers,

--

Darius

[BrokenMouse]

Sorry for the late reply, but that worked like a charm.

Thanks!