VMware Cloud Community
ViennaAustria
Contributor
Contributor

ESXi 5.x on new Apple Mac Mini 6,2 Late 2012 *NOT* working

Hello!

I bought one of the new MacMinis, which were presented yesteraday, plugged two 8GiB DD3-1600 modules in (OsX reported 16GiB of memory, as the models before) and tried to install ESXi 5.1 Build 799733. It started promising:

Initializing chipset...

Initializing timing...

Initializing scheduler...

Initializing processors...

Initializing ACPI...

*BANG*

Pink screen! Smiley Sad

NOT_REACHED bora/vmkernel/hardware/intel/vtd.c:3638

cr0=0x8001003d cr2=0x0 cr3=0x449ad500 cr4=0x12c

*PCPU0:4096/bootstrap

PCPU  0: SISISISISISISIS

Code start: 0x41800c600000 VMK uptime: 0:00:00:04.301

and a stackdump.

macmini.png

I tried instalkling 5.0 and 5.0 U1 also, but with the same result.

It is the regular MacMini with an i7 4C 2.3GHz and one 1TB SATA drive (http://store.apple.com/us/configure/MD388LL/A with no extras), not the MacMini Server.

Any suggestions? Thanks!

PS1: I was unable to boot the ESXi setup from USB key! Since other USB keys (OsX setup for example) didn't work either, I suspect that there's some EFI boot problem from the new USB3 ports. Booting from CD worked, however.

PS2: on the earlies models we had the problem, that the Alt key wasn't recognized after powering on, when there is an USB hub between the keyboard and the Mac (a KVM switch for example). That seems to be fixed now: whenever I press the Alt key at the powerup chime, I get the Apple "boot menu", even with our KVM switch in between.

338 Replies
ViennaAustria
Contributor
Contributor

PS3: the 8GiB DDR3-1600 SO-DIMM modules I tried were

Kingston KVR16S11/8

Corsair Vengance CMSX8GX3M1A1600C10

Both are working.

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

I tried with the original 2x2GiB memory modules. As expected, it made no difference: pink screen.

I also tried to pre-install ESXi 5.1 on USB key on another (older) MacMini and boot the new MacMini from that USB key. It booted up, but gave me the very same pink screen. That also was to be expected, but I wanted to make sure.

I hope that VMware is quickly taking on that case, because I suspect the same problems with all new Apple machines (iMac and MacPro). If that will be the case, there is no hardware for the VMware Macintosh support any more.

I smile about the fact, that VMware officially still only supports the Apple Xserve - a hardware, that's not produced since almost two years! So there's a great product - we have lots of customers using virtualized OsX - but no hardware to run it. :smileyplain:

Reply
0 Kudos
saltmaster
Contributor
Contributor

I just received my MacMini as well (the same model as above) and tried to install the hypervisor (5.1) on it and received exactly the same kernel panic.

http://imageshack.us/a/img21/9052/img0206ce.jpg

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Hi ViennaAustria and saltmaster,

Thanks for letting us know about this problem – in pink text no less!  Apologies for the trouble... There is obviously something about your new Apple hardware that we're not handling.  I've filed an internal bug report so that our ESXi hardware folks will investigate the problem.

Cheers,

--

Darius

Reply
0 Kudos
oturn
Enthusiast
Enthusiast

If anyone following this thread gets ESXi installed on the 2012 Mac Mini, please also report whether VT-d / VMDirectPath I/O is functional.

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Here's a possible workaround...

Very early in the boot process, a prompt appears in the bottom-right corner of the screen to press Shift+O to edit boot options.  At that time, press Shift+O and you'll get an editable line of text along the bottom of the screen with a cursor at the end.  Add a single space followed by:

   iovDisableIR=TRUE

Then press the Enter key and the system will attempt boot.

I don't have access to the relevant hardware right away, so I haven't got the opportunity to test out the above... I thought it would be worthwhile to pass the information along promptly and hope that someone could give it a try and see if that is all that's needed to get ESXi to boot on this hardware.

Cheers,

--

Darius

saltmaster
Contributor
Contributor

Hi,

I can confirm adding the boot option iovDisableIR=TRUE does do the trick. After adding and modifying the most recent tg3 drivers for the NIC I was able to get it up and running (have to add in the PCI device ID, looking at the drivers source code there is support for the specific revision it just isn't included on the PCI device mapping tables).

Thanks for the speedy help !!

Cheers

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Thank you for trying that out for us all, saltmaster, and reporting back so quickly with your detailed findings!

We'll keep investigating how to solve the problem properly here.  Depending on the outcome, affected hosts will either need an Apple firmware update or an ESXi patch update to get things going smoothly.

Cheers,

--

Darius

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

@dariusd: I can also confirm that the boot option prevents ESXi 5.1 from crashing. Thank you very much! How to permanently add this boot option?

@saltmaster: can you please help me how to insert the new PCI device id into the device mapping tables? I use a customized image with the latest version of the net-tg3 offline bundle. That works with all the previous MacMini models, but not on the new 6,1.

@all: the Apple Tunderbolt Gigabit Ethernet adapter is NOT working. It worked on the previous MacMini model. What does "iovDisableIR=TRUE" do? Could that maybe affect the Thunderbolt Interface? It was an easy option to add a second network adapter to a MacMini (pitty, that there are no Thunderbolt hubs yet).

As I booted from the pre-installed USB key I saw a message "loading vmfs3 failed" on the yellow/grey ESXi screen. That could be, because I booted from USB and have no VMFS datastore on the Mac yet (the harddisk still contains OsX). Hopefully it's not another adaptation that is necessary for the new MacMini. Saltmaster, did you experience problems with VMFS?

Reply
0 Kudos
saltmaster
Contributor
Contributor

On the installation media there is a file called: NET-TG3.V00

You will need to extract this and edit two files that relate to pci id mapping. The new MacMini's use a BROADCOM BCM57766 which have a device ID of 0x1686. If you copy the entries for say device ID 0x1688 and modify them to reflect the 0x1686 that will get it working.

The V00 file can be extracted using vmtar, followed by a normal tar. The easist way to go about this is to install the hypervisor in VMware Fusion (or if you have it running on another machine already use that). Activate SSH on the Hypervisor, remote in (copying NET-TG3.V00 over), extract the files, modify them and then perform the inverse of the extraction to recreate the NET-TG3.V00 file.

From memory it looked something like:

# vmtar -x NET-TG3.V00 -o NET-TG3.tar

# tar xvf NET-TG3.tar

edit the files

# rm NET-TG3.V00

# rm NET-TG3.tar

# tar cvf NET-TG3.tar etc usr

# vmtar -c NET-TG3.tar -o NET-TG3.V00

Copy the NET-TG3.V00 file onto the installation media removing the older version.

Once you have it installed on the MacMini you can use the following command on the host to have the iovDisableIR settings stick (enable SSH just to perform the following):

# esxcli system settings kernel set --setting=iovDisableIR -v TRUE

saltmaster
Contributor
Contributor

One thing, you may want to take a disk image of the installed version of MacOS X ML before playing around too much. I thought it would be easy to reinstall 10.8.2 on the machine after some testing, but that doesn't seem to be the case. I've tried installing from USB/SD card/NetBoot all of which fail (creating the USB/SD card using both DiskUtility/CCC). Internet Recovery also fails.

To that end could someone running Mac OS X on a MacMini 6,2 run the following command to get the Board ID (may just be a simple case of needing to add it into the installation media's allowed machines):

ioreg -lp IOService | grep board-id

Reply
0 Kudos
RiteshA
Contributor
Contributor

Here you go.. The result of the grep commmand::

"board-id" = <"Mac-F65AE981FFA204ED">

Hoping we get ESXi working on 6,2 soon.

Thanks

Reply
0 Kudos
saltmaster
Contributor
Contributor

Thanks. Installing MacOS X 10.8.2 using target disk mode on another Mac and then modifying "/System/Library/CoreServices/PlatformSupport.plist" got it booting again. After I got it booting I was able to apply: http://support.apple.com/kb/DL1603

Much appreciated.

Reply
0 Kudos
willplaice
Contributor
Contributor

my custom built mini is on it's way from China, glad to know ESXi will run, even if it needs some tweaks,

i wondered what storage options people where using, i need some sort of DAS rather than a NAS, and wondered what worked, i don't see any support for thunderbolt raid ?

also is DirectIO or TH supported ? (i guess i'll find out later in the week)

I was also wondering, if i boot from SD, can i create a datastore of the OSX disk, and access that as a vm - writing directly to the real disk ? - other vms would me installed on an external disk.

many thanks, Will

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

@saltmaster: thank you for the foolproof description. It worked perfectly!

It is interesting, that the Apple Thunderbolt NIC is not recognized during setup, but when the machine is up, it becomes vmnic32 (why vmnic32 and not vmnic1?):

tbnic.png

So the Thunderbolt Gigabit Ethernet adapter is still useable with the new MacMini hardware.

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

@willplaice: the SATA controllers/drives in the MacMinis are being recognized in ESXi. That is also true for the new Late 2012 series. You can use them as local storage and/or to install ESXi onto. Howerver, the MacMinis don't have an internal RAID controller and ESXi does not support softRAID (softRAID0 support would'n hurt, by the way). So you have no redundancy on DAS in a MacMini.

I never tried installing ESXi on SD card. But I guess it would work. On ESXi hosts with DAS we simply install ESXi onto it. Diskless "workers" in a cluster are being booted from USB key. You may PXE boot them also, but not with Mac, because Apple uses netboot instead of PXE and I believe there's no VMware support for that yet.

For NAS/SAN you can use the biuiltin Gigabit NIC and/or attach several Thunderbolt devices, such as the Apple Gigabit Ethernet Adapter, which will give you NIC failover for just 25 bucks, the Sonnet 10GbE adapter or several FC HBAs. I only have experience with the Apple GbE and it works well. We do larger Mac installations with MacPro and regular Intel 10GbE NICs.

Keep in mind, that the MacMini only has ONE Thunderbolt port. It is also shared with the Display Port, so you can't attach the cheap miniDP-to-VGA adapter and have to use the HDMI port. That often is an issue with KVM switches in a rack or server farm. However, there are HDMI-to-VGA adapters available for +50 bucks.

For HW-RAID you can use any Thundebolt-to-PCIe Adapter (Sonnet would be my first choice) with any supported RAID controller (I would pick LSI here). But I would not recomend that design, because you end up with a dozen of external boxes and any cabeling problem will crash your system severely. If you add up all the price tags, you almost end up at the level of a MacPro, where you can easily put all those cards into its PCIe slots.

But beware with DAS/RAID on MacPro or Xserve: the Apple RAID controller is not supported with ESXi (any news there?) and for other RAID controllers you have to solve the problem on how to connect the built-in harddisks, that use a proprietary connector! There are kits available on eBay, but I have no experience with them.

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

I could also sum up all my writing to: Thunderbolt transparently bridges 4 PCI-e lanes to any external hardware. So whatever PCIe device is supported by ESXi should also work at the end of a Thunderbolt bridge. Maximum throughput is PCI-e x4 (4x2.5=10Gbps), so 2 FC4 ports or 1 FC8 is fully supported, whereas 10GbE might be a bit underfead. But I saw no saturated 10GbE yet - except maybe for 10Gb crosslinks between heavily loaded network switches.

Reply
0 Kudos
ViennaAustria
Contributor
Contributor

@oturn: apparently it is working:

vtd.png

Reply
0 Kudos
oturn
Enthusiast
Enthusiast

@ViennaAustria: Thank you for confirming. Which processor do you have? If my research is correct, this should be accurate:

2012 Mac Mini:
ProcessorSpeedVT-d Support
i5-3210M2.5Ghz Dual Core No
i7-3615QM2.3GHz Quad CoreYes
i7-3720QM2.6GHz Quad CoreYes
Reply
0 Kudos