VMware Cloud Community
ViennaAustria
Contributor
Contributor

ESXi 5.x on new Apple Mac Mini 6,2 Late 2012 *NOT* working

Hello!

I bought one of the new MacMinis, which were presented yesteraday, plugged two 8GiB DD3-1600 modules in (OsX reported 16GiB of memory, as the models before) and tried to install ESXi 5.1 Build 799733. It started promising:

Initializing chipset...

Initializing timing...

Initializing scheduler...

Initializing processors...

Initializing ACPI...

*BANG*

Pink screen! Smiley Sad

NOT_REACHED bora/vmkernel/hardware/intel/vtd.c:3638

cr0=0x8001003d cr2=0x0 cr3=0x449ad500 cr4=0x12c

*PCPU0:4096/bootstrap

PCPU  0: SISISISISISISIS

Code start: 0x41800c600000 VMK uptime: 0:00:00:04.301

and a stackdump.

macmini.png

I tried instalkling 5.0 and 5.0 U1 also, but with the same result.

It is the regular MacMini with an i7 4C 2.3GHz and one 1TB SATA drive (http://store.apple.com/us/configure/MD388LL/A with no extras), not the MacMini Server.

Any suggestions? Thanks!

PS1: I was unable to boot the ESXi setup from USB key! Since other USB keys (OsX setup for example) didn't work either, I suspect that there's some EFI boot problem from the new USB3 ports. Booting from CD worked, however.

PS2: on the earlies models we had the problem, that the Alt key wasn't recognized after powering on, when there is an USB hub between the keyboard and the Mac (a KVM switch for example). That seems to be fixed now: whenever I press the Alt key at the powerup chime, I get the Apple "boot menu", even with our KVM switch in between.

338 Replies
HPReg
VMware Employee
VMware Employee

> I tried to make a snapshot of the installation in ESXi, but the snapshot menue is greyed out while the OS X VM is running. I can make snapshots of all my other VMs (Linux-based and Windows 7) while the VMs are running.

Mmm, that is a bug. Can you file a Service Request and mention my name (hpreg@YouKnowTheCompany.com) in it, so the corresponding bug will be routed to me?

> it took about 3 seconds for the entire snapshot. Therefore, I'm not sure the snapshot was properly made.

Snapshots of powered off VMs are instant, so that is normal.

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

HPReg: I'm using a free version of ESXi. So, I had to file this under "Licensing" as a Support Request. I have mentioned you in "Additional Notes".

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Hi telecastle,

Is this the VM for which you've enabled VM DirectPath I/O?  That feature is incompatible with snapshot/suspend/resume of a running VM, since we have no way to capture or restore the state of the passed-through hardware.  Allowing the VM to be snapshotted while a device is passed-through would lead to catastrophic failure of the VM (and/or the host) when the device is next used.  You should be able to snapshot your Mac OS VM once you remove all the VM DirectPath I/O devices from it.

Cheers,

--

Darius

Message was edited by: dariusd: Clarified that snapshots are OK for a powered-off VM; It's only a "live" VM that won't snapshot with passed-through devices.

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

Darius,

Thanks for this clarification. Yes, this is the OS X VM that I enabled for Direct Path I/O. While on this topic, can you elaborate on the two USB controllers that show up in ESXi Host > Configuration > Advanced Settings > Edit? The two USB controllers are:

00:1a.0 | Intel Corporation Panther Point USB Enhanced Host Controller #2

00:1d.0 | Intel Corporation Panther Point USB Enhanced Host Controller #1

Do you know what each of these USB controllers is responsible for? There are four USB ports on the back of the Mac Mini, so why are there two USB controllers that ESXi see? Also, does USB work for for Direct Path I/O in ESXi 5.1? I recall someone mention that USB was broken in ESXi 5.1. Is it true?

Thanks!

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Hi telecastle,

I've read a few reports that USB passthrough is broken on ESXi 5.1, but I have not personally investigated that.  I believe that passing through USB controllers is not something we officially support (to the best of my knowledge, which I readily admit is not necessarily complete), and passing through motherboard-integrated devices is fraught with peril, so I would not be surprised if it didn't work...

Regarding the number of ports and controllers: In general, the designers of the hardware platform can do whatever they want in terms of mapping physical ports or functions to USB controllers.  There are no definite rules, and often it is not documented anywhere, and you just have to figure it out for yourself.

In the case of my Macmini6,2 here, some experimentation with System Profiler tells me that:

00:1a.0 is the controller managing the four rear USB ports when USB 2.0 devices are attached.

00:1d.0 is the controller managing the internal USB 2.0 devices: an IR Receiver and a Bluetooth USB Host Controller.

System Profiler also reports an xHCI (USB 3) controller at 00:14.0, which also manages the four rear USB ports when USB 1.x or USB 3.0 devices are attached.

In other words, if you attach a USB 1.x or USB 3.0 device to any rear port, it'll appear on 00:14.0; If you attach a USB 2.0 device to the very same port, it'll appear on 00:1a.0.  I don't think you can attach any external device to 00:1d.0 without opening the Mac Mini and perhaps doing some soldering or rewiring.  :smileysilly:

Cheers,

--

Darius

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

Darius,

This is very valuable information. Thank you very much.

I'll try to enable Direct Path I/O for 00:1d.0 - Bluetooth and IR Receiver. ESXi doesn't have drivers for these devices anyway, so I will pass them to the OS X VM. This way, if I decide to use this VM as my desktop with the monitor attached (and Direct Path I/O enabled for video), I can use my Bluetooth keyboard and Apple Magic Trackpad to control the OS X VM.

You are a treasure troth of knowledge.

Thanks again!

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

Success. I've enabled Direct Path I/O for 00:1d.0 and passed both IR Receiver and Bluetooth (under USB Device) to the OS X VM. I can now see Bluetooth in OS X.

Edit: When I tried to pair the Bluetooth keyboard to the OS X VM, I got to the point that the VM found the Bluetooth Keyboard, then it asked me to enter the code followed by Return, and finally it said that the Keyboard could not be identified (which is normal - it's a new Logitech K811), and prompted me to press a key to the right of the left Shift key. When I did so, ESXi crashed. My vSphere Client lost connection to the ESXi host, and I could no longer ping it. I have to go and hard reset the ESXi host now.

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

After multiple times of trying to power on the OS X VM, each resulting in the ESXi host's crash, I decided to remove ALLdevices enabled in ESXi for Direct Path I/O and only after I restarted ESXi 5.1, was I able to power up the OS X VM without it crashing the ESXi host running on Mac Mini 6,2.

However, the farthest I could get was the login prompt in OS X (via ESXi's VM console). Once I supplied the password for the account, the OS X showed the spinning wheel and the spinning beach ball. I left it like this for 1.5 hours, and finally decided to repair the OS X installation virtual disk using Disk Utilities from the Installation DVD image.

In order to boot from the image of the installation DVD (InstallESD.dmg), I had to boot the VM into EFI and change the boot order. After booted from InstallESD.dmg, I ran the check on the OS X virtual volume, but found no errors. Then I ran the permission repair, and many permissions were out of whack. After that repair completed, I decided to go ahead and reinstall OS X Mountain Lion from the DVD image. This was not a clean install because I did not reformat the virtual volume on which OS X Mountain Lion (not working) was installed - my hope was to preserve all of the user data on this volume. After the reinstallation of the Mountain Lion completed, I was able to log in to the Mountain Lion (finally!) and then enabled Desktop Sharing via the ESXi's VM console (which had been enabled before the first ESXi crash, but was disabled now). At this point, I was able to remote into the VM, using Screen Sharing.

I found out that the Open Directory master that I had enabled (before my fiasco with Direct Path I/O for Bluetooth) was no longer enabled, and I had to recreate the master. It failed after the first attempt, but worked on the second attempt.

Even though file sharing was already enabled, I had to disable file sharing over AFP and SMB and then re-enable them in order for my clients to be able to mount shares on the OS X Mountain Lion running in the VM on the Mac Mini 6,2. At this point, I was able to gain access to my files, which seem to be all intact.

So, this was a very educational experience with the following conclusions:

1. Always back up your files before attempting anything crazy like using non-working features on unsupported hardware, which is exactly what it was with using a USB Controller Direct Path I/O passed through to an OS X Mountain Lion VM running on Mac Mini 6,2 under ESXi 5.1.

2. When people report that a feature is broken, believe them. I was apprehensive about enabling one (or both) USB Controller for Direct Path I/O, but for some reason, when I learned that one of these controllers was responsible for Bluetooth, I got so excited that I forgot about the warning.

3. Before enabling a major feature, such as Direct Path I/O, always take a snapshot of the VM. In this case, I would have had to power down the VM and take a snapshot (with all of my valuable files already having been copied to this VM), but I failed to do so. Had I done so, I would have been able to restore this VM in a few minutes instead of 7+ hours I spent on this.

4. Think logically, go slow, and do not give up.

Now, I have a couple questions for the VMware guys reading this.

1. It's obvious now that USB controllers are not reliable (at least in ESXi 5.1) when enabled for Direct Path I/O. Is this something that's going to be corrected in the next release of ESXi 5.1?

2. After I tried to pair my OS X VM with a Bluetooth keyboard, which crashed my ESXi, the USB controller responsible for Bluetooth and IR Receiver was no longer enabled for Direct Path I/O upon the hard restart of the ESXi host. However, the VM kept crashing the ESXi host when I tried to power the ESXi host. When I tried to re-enable this USB controller for Direct Path I/O and restart the ESXi, Direct Path I/O would become disabled again upon the ESXi host's graceful reboot. Question, why was Direct Path I/O no longer working for this USB controller, and why did the OS X VM continue to crash the ESXi host?

3. The only way I was able to stop the OS X VM from crashing the ESXi host was by disabling Direct Path I/O on ALL other devices that had been working prior to the Bluetooth fiasco. These devices are: Audio, Wi-Fi, and FireWire. Question, why was this step necessary? Is there a bug with other devices (besides USB) being enabled for Direct Path I/O in ESXi 5.1 running on Mac Mini 6,2?

4. I'm not going to attempt to enable any devices for Direct Path I/O until the next release of ESXi 5.1. So, are you guys aware of the issues that I encountered and are the fixes being worked on? Obviously, without using Direct Path I/O, OS X running in a VM can only be used as a headless server. However, with being able to enable video, audio, wi-fi, FireWire, Bluetooth, and IR Receiver for Direct Path I/O, an OS X VM could be used as a client while being able to run other servers concurrently on the same hardware. So, is this something that's being given proper attention to make sure these physical devices can be passed through to an OS X VM reliably and not result in the ESXi host's crashes like what I've experienced?

5. Will the next release of ESXi 5.1 incoporate the SMC fix so that it can be run on the Mac Mini 6,2 without any non-official modifications of the installation image?

Thank you everyone for all of your work!

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

Hi telecastle,

The intended use of PCI Passthrough is for users with high-performance storage or network controllers to connect those controllers directly to the VM which needs it at peak performance.  It is not intended for turning a VM into a fully-fledged "client" using the host's hardware.  The many and varied quirks of PCI[e] devices mean that we can only support passing through well-behaved devices.  The many-more varied quirks of everyday motherboard-integrated or southbridge PCI[e] devices mean that we will most likely never be able to support passing-through such devices.  At least standalone/plug-in PCI[e] devices are designed to operate as standalone devices, and we know that they won't have stealthy interactions with other devices in the host, or with the host firmware... we can't say the same for integrated devices.

As a concrete example: If you pass through only your host USB 2.0 controller (00:1a.0) and start connecting/disconnecting USB devices, we can not guarantee that there won't be unexpected interaction with the xHCI controller (00:14.0) which shares the same physical ports.  We have no way to characterize what could possibly happen in that configuration, so there is basically no way we can support it.

I've been amazed by some of the reports here of what people have managed to get working by passing-through PCI[e] devices, but I have equally been terrified to hear some of those stories, which are often way beyond the bounds of what we support!

The SMC fix is queued up for delivery in a future ESXi 5.1 update... I can't say precisely when.

Cheers,

--

Darius

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

Darius,

Thanks for this detailed response. I did not realize that passing video card, USB (besides the fact it's broken in ESXi 5.1), and other PCI devices was "experimental" so to speak. So, thank you for this clarification.

You mentioned that the "intended use of PCI Passthrough is for users with high-performance storage or network controllers to connect those controllers directly to the VM which needs it at peak performance". This is of great interest to me. If so, does ESXi itself have drivers to support Thunderbolt so that a Thunderbolt storage device could be used as an external datastore?

If ESXi itself has no support for Thunderbolt-based datastores, can you comment on whether the Thunderbolt port pass-through has been tested and is something you would consider "supported" so that an OS X VM could use a Thunderbolt storage device as its DAS?

If the answer is yes to either (or both) of the above questions, do you know if the Drobo's Thunderbolt storage devices (Drobo 5D or Drobo Mini) would work in either of these two scenarios?

Finally, I came across a strange error message in my OS X VM running under ESXi 5 when using the ls -l command on the /Volumes directory of the OS X VM:

ls: VMware Shared Folders: Input/output error
total 8
lrwxr-xr-x  1 root  admin    1 Feb  8 13:02 Macintosh HD -> /
drwxrwxr-x  9 root  wheel  374 Feb  8 10:08 Recovery HD

I tried to google this error, and the only thing I can find is a reference to VMware Fusion and a bug in VMware Utilities. I actually installed VMware Utilities from the vSphere's Client into this OS X VM. So, it appears this actually could be the same issue, but my question is about the message itself. There's no concept of "Shared Folders" in ESXi because there's no host OS, correct? So, why is this error even displayed? Is this an example of VMware Utilities for OS X being ported from Fusion to ESXi?

Thanks!

Reply
0 Kudos
HPReg
VMware Employee
VMware Employee

> ls: VMware Shared Folders: Input/output error

> I tried to google this error, and the only thing I can find is a reference to VMware Fusion and a bug in VMware Utilities. I actually installed VMware Utilities from the vSphere's Client into this OS X VM. So, it appears this actually could be the same issue, but my question is about the message itself. There's no concept of "Shared Folders" in ESXi because there's no host OS, correct? So, why is this error even displayed? Is this an example of VMware Utilities for OS X being ported from Fusion to ESXi?

There is no such thing as "VMware Tools for Fusion" or "VMware Tools for ESXi". You should think of the VMware Tools as a seperate product (i.e. seperate from Fusion or ESXi), that you just happen to always install inside a VM instead of outside on the physical box.

So what happens here is that on ESXi, because as you explained they make no sense, Shared Folders are disabled. So you get this error inside the VM when you try to browse Shared Folders. You would get exactly the same error in Fusion if you disabled Share Folders in the Fusion UI.

Now IIRC this particular issue has been fixed in a later version of the VMware Tools for Mac OS X. For example, when you enable/disable Shared Folders on the host, the new version of VMware Tools automatically make the icon appear/disappear on your desktop. To get a more recent version of VMware Tools for Mac OS X, I recommend you use the VMware Tools for Mac OS X that are shipped with VMware Workstation 9.x or VMware Fusion 5.0.2.

Reply
0 Kudos
andrew_mackeret
Contributor
Contributor

HPReg,

That is a neat trick, VMware Tools for OS X does not ship with Workstation 9.x, you probably meant ESXi 5.1, just a typo, LoL, sorry couldn't resist.

A link to the repository for VMware Tools see below:

http://softwareupdate.vmware.com/cds/vmw-desktop/

Reply
0 Kudos
AK_____28
Contributor
Contributor

Can anyone help with this error I'm experiencing on my Mac Mini late 2012 6,2?

I'm getting the following message when trying to use a USB device;

"WARNING: LinuxDMACheckContraints:138:Cannot map machine address = 0xfffffffffff, length = 6 for device 0000:00:1d.0; reason = address exceeds dma_mask (0xffffffff))"

Any help is immensely appreciated.

Thanks in advance! Have a wonderful day!

Reply
0 Kudos
glenndm
Contributor
Contributor

Hi,

First: Many thanks to this thread and its contributors: the mac mini/esxi combination is great !

Question:

What options exist to add external storage to the esxi datastore  for the mac mini late 2012 model?

And which allow for removable storage?

I would like to use usb3 1 TB disks, but I understand vmfs is not possible on USB ?

I now take offline VM full backups by exporting to OVF template, but this is manual and very slow.

best regards

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

I second the request for clarification on the redundant storage solution to work with a Mac Mini (with Thunderbolt and USB3) support.

Currently I'm using an external NAS that hosts ESXi datastores via iSCSI. This works fairly well for a home lab environment. I'm currently running 6 VMs on the Mac Mini 6,2 with five of them being hosted on the iSCSI datastore on the RAID NAS. However, I'm not too pleased with the vendor of my NAS, and I'm looking to migrate my RAID based datastore to another platform.

My primary alternatives at this time is either Synology DS713+ or a Drobo Mini/Drobo 5D. I am now running OS X 10.8.2 VM with the Server App, and I believe that I can provide all the file serving, media serving, FTP/SFTP, WebDAV, etc., via this OS X Server VM. So, the RAID storage device I'm looking for does not have to be a NAS (but it can if it also supports iSCSI). I would really prefer to use a DAS like a Drobo Mini/Drobo 5D solution.

I've asked this question many times, but I have not yet heard anyone answer it. Has the Thunderbolt Drobo devices (Drobo Mini/Drobo 5D) or any other Thunderbolt based RAID devices been tested with ESXi 5.x? My first preference would be to use in such a capacity that ESXi can use it as an external datastore and so that Drobo itself manages the RAID. If this is not possible, I could consider using the Thunderbolt based Drobo as a directly attached volume for the OS X VM running on the Mac Mini 6.2. This would only be possible if I could enable Thunderbolt for Direct Path I/O (passthrough) in ESXi 5.x, but this would have to be a reliable passthrough (not experimental support like I've learned that other types of Direct Path I/O passthrough are - especially USB passthrough).

I would really appreciate if someone who has had any experience using ESXi 5.x with Thunderbolt devices could contribute to this discussion.

Thanks!

Reply
0 Kudos
vSean1
Contributor
Contributor

I haven't seen any reports of it working, but the thunderbolt is just a PCIe bus as far as ESX is concerned.  You would need a thunderbolt attached array that uses a raid controller that has a driver that works under esx.  As cool as the drobo mini looks I really don't think it'll work direct attached to the esx box because there is nothing close to it on the hcl. You might have better luck with a promise based solution, but you'll still need to ID the controller used in the raid box and find a working driver for esx based on its controller device-id and its closest living relatives on the hcl.

Reply
0 Kudos
telecastle
Enthusiast
Enthusiast

This is good information - thanks!

There's also a Drobo B800i, which is an iSCSI-only device with two Gigabit ports (that can only do iSCSI). The benefit of this box is that it can directly communicate with two discrete iSCSI initiators, which can concurrently access LUNs homed to either Gigabit port. This device would allow the ESXi host to maintain its own datastore on the B800i, while letting the OS X VM use a separate LUN via iSCSI for its own storage, utilizing the other B800i's port, and both the ESXi host and the OS X VM could use the same B800i box concurrently. Incidentally, Drobo includes an iSCSI initiator for OS X with the B800i's dashboard. The downside of this solution is the price, which is $2,500 USD for the diskless device.

So, it appears that the Synology is the way to go for now - it can be had diskless for $530 (with only two bays, though).

Reply
0 Kudos
ganaoil
Contributor
Contributor

Hi.  Thanks for all the instructions and helpful links I was able to get my macmini working.  Unfortunately I cant get any virtual installed.  I keep getting - "DHCP in the console followed by PXE-E53: No boot filename received then PXE-M0F: Exiting Intel PXE ROM.  Operating System not found".  Can anybody help?

Reply
0 Kudos
AK_____28
Contributor
Contributor

Ganaoil,

     When you get to the point of no operating system found, connect the cd device, then send the guest ctrl+alt+del... Do not hard power off or the cd drive will unattached again.

Reply
0 Kudos
ganaoil
Contributor
Contributor

Thank you! Newbie here Smiley Happy

Reply
0 Kudos