VMware Communities
SvenGus
Expert
Expert

M1, M2 and nested virtualization

Until now, I had always thought that the lack of a decent nested virtualization in the current Fusion, etc. was due only to its incomplete implementation by Apple: i.e., only a software issue; until I read this, among others:

https://www.reddit.com/r/apple/comments/qz0ill/nested_virtualization_on_m1_is_this_a_hardware

So, is it also a hardware issue (which would make nested VMs impossible on the M1)…? If that is the case, those who need nested virtualization would necessarily have to wait for M2 or M3 MacBook Pros, or similar…

Reply
0 Kudos
15 Replies
ColoradoMarmot
Champion
Champion

Mx virtualization is early days, so limitations/futures are really foggy.

What's the use case for nested virtualization on M1's?  You can't do it to run Intel guests if that's what the thought is.

Reply
0 Kudos
SvenGus
Expert
Expert

For example, WSL 2 inside a Windows 11 ARM guest, on M1: that doesn’t seem to be possible, now. What isn’t so clear is if this is only a software issue, which could eventually be superseded by a more evolved Apple virtualization framework; or if it is also a limitation of the M1 at the hardware level, as some people say: who knows - difficult to say…

Reply
0 Kudos
ColoradoMarmot
Champion
Champion

Well, we'd need windows support first 😉

 

Seriously, that's a really good question.  Apple's sort of half-supporting/tolerating virtualization apps these days, but I suspect it was below the line on the silicon design team's list.  Be interesting to find out.

Reply
0 Kudos
bluefirestorm
Champion
Champion

It would appear ARM v8.4 supports nested virtualisation.
https://developer.arm.com/documentation/102142/0100/Nested-virtualization?lang=en

Isn't M1 based on ARM v8.5? So, it is likely an Apple hypervisor framework limitation rather than an ARM limitation (unless somehow Apple chose not to implement them in their own M series of CPUs).

 

Reply
0 Kudos
SvenGus
Expert
Expert

Sadly, here the developers of Asahi Linux (who should know the M1 well) say that only the M2 probably will support nested virtualization:

https://mobile.twitter.com/AsahiLinux/status/1513037664379113481

In addition, Apple would have to support it in software, too…

Reply
0 Kudos
bluefirestorm
Champion
Champion

It would be interesting if the virtualisation/nested virtualisation is already inside the silicon but just not enabled. I remember when VMware Workstation started using Intel VT-x, I had a Dell laptop and it simply required a BIOS update for the Intel VT-x to be enabled; so the Intel VT-x feature was already sitting inside the silicon.

Also wonder if Apple would make a distinction of virtualisation/nested virtualisation features (or any other features) between the normal M1 from M1 Pro, M1 Max (or for any other future M series CPUs). Surely, it can't be just core count that makes for difference in pricing/performance. I have no idea what the licensing/royalty fee agreement is like between ARM and Apple but it is entirely possible that adding in certain CPU feature/designs might require extra royalty payments. But Apple being Apple it is quite unlikely they will let end-users control the CPU features just like what PC world does in the UEFI/BIOS.

SvenGus
Expert
Expert

Certainly, it would be cool if features could simply be unlocked: but Apple should probably first show much more interest in bringing a complete virtualisation solution to the new Mx Macs (one would indeed expect full feature parity with Intel-on-Intel virtualisation, for ARM-on-ARM virtualisation, eventually)…

Reply
0 Kudos
bluefirestorm
Champion
Champion

My guess is Apple wants to control all virtualisation through their Hypervisor Framework APIs; and that's what probably make it look like Apple is not fully embracing virtualisation. If they don't get this control right from the get-go, it will be difficult to stop somebody creating an iOS or iPadOS VM running on Apple Mx Macs.

One of the use cases would be running a macOS VM with Xcode is to develop for iOS/iPadOS and not using a simulator (like what is done now with Xcode on Intel Macs) but run these iOS/iPadOS as VMs instead (or nested virtualisation if the Xcode is running inside a macOS VM).

Anyway, looks like there are beta API and sample code for creating macOS VM running on Apple Silicon.
https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple...

Reply
0 Kudos
Technogeezer
Immortal
Immortal

Tend to agree. With the deprecation and eventual removal of "legacy" kernel extensions, the only way that virtualization is going to work is via ULM's and macOS APIs such as the Hypervisor Framework. They have to get the APIs right and fully featured enough for the whole thing to work.

- Paul (Technogeezer)
Editor of the Unofficial Fusion Companion Guides
Reply
0 Kudos
ColoradoMarmot
Champion
Champion

I do worry about that - Apple's been half-hearted with it so far.  Hopefully they'll at least be enough for someone like VMWare to build a real product on top of them.

Reply
0 Kudos
qualeos
Contributor
Contributor

Another more niche use case is network virtualisation tools such as GNS3 and EVE-NG.  They use a Linux VM to host and network together many different network device VMs (routers, switches, firewalls, load balancers etc).  They use the outer VM as it makes it much easier to control the environment and allows for the network device VMs to be connected together in a consistent fashion.  Currently, the majority of the network device VMs are x86 based, so we are a way off this being useful, even if Mx supported nested, but I can dream.

Reply
0 Kudos
Technogeezer
Immortal
Immortal


@ColoradoMarmot wrote:

I do worry about that - Apple's been half-hearted with it so far.  Hopefully they'll at least be enough for someone like VMWare to build a real product on top of them.


I wouldn't say half-hearted given what they've been enhancing in Ventura. However, Apple is going about it in their own way.

The issue as I see it is not "enough for someone like VMware to build a real product" - but the willingness of VMware to adapt to what Apple is providing. That could mean not re-using technology they use in Workstation and ESXi to maintain compatibility with their other platforms. I believe that VMware could have a lot of the virtualization goodies that Apple announced for Ventura... if they decided to start supporting "Linux de-facto standard" virtio drivers for virtual devices. I don't see that happening any time soon.

 @Mikero has hinted, for example, that supporting macOS virtualization on Apple Silicon will mean they have have to do things "differently" than they do for other platforms. VMware is targeting for an ARM SystemReady VM virtual machine template. macOS is not adhering to this, so VMware has tough decisions to make.

 

 

 

- Paul (Technogeezer)
Editor of the Unofficial Fusion Companion Guides
Reply
0 Kudos
salmanHKhan
Contributor
Contributor

Have you looked at UTM? It can do x86-based CPU emulation on M2. I was able to run x86-based eve-ng with UTM on M2 but the virtual router did not work inside eve-ng due to no nested virtualization support on M2.

Reply
0 Kudos
Technogeezer
Immortal
Immortal

 

Unfortunately eve-ng is one of those things that you are going to have to run on a PC or Intel Mac. They do not provide a workable ARM port and the gotchas for trying to use it under emulation are show-stopping. Couple that with lack-luster performance of x86/x64 chip emulation that QEMU does (not one-time static translation like what  Rosetta does) and you find that an Apple Silicon Mac is a poor plarform choice for running eve-ng.  

- Paul (Technogeezer)
Editor of the Unofficial Fusion Companion Guides
Reply
0 Kudos
SvenGus
Expert
Expert

Now that the macOS 14 Sonoma developer beta is out… any news for nested virtualisation in Sonoma…? Probably not, but who knows…

Reply
0 Kudos