Good morning to everybody,
I have this big problem with Fusion and Big Sur. The scenario is the following.
VMware Fusion 12.1.0
MacOS Big Sur 11.0.1
Network relevant details:
en0 -> untagged interface adapter on a private network
vlan0 -> tagged interface adapter on a public network
OS: Ubuntu 20.04.1
Network: public ip on a network adapter bridged on vlan0
The problem is that the guest is unable to ping outside the host (guest <-> host is ok).
In other words, the default gateway for the vlan0 doesn’t respond to the guest (in any case both firewalls are off and forwarding is enabled).
I also used tcpdump and I have verified that packets go out from guest and reach the host, but they do not come back.
I already tried many of your dns or nat tested solutions to the related issues, but the problem persists.
What is interesting is if I move the guest bridge on the en0 (of the host), reconfiguring it with a public ip, all works (as expected).
This for saying that the problem should be strictly related to VLAN.
Note: before updating (both to Fusion 12 and Big Sur) all was working between host and guest (both having a public IP address) and routing to Internet.
Any help or suggestions is very appreciated.
I have been working on this issue for a while and could not get it figured out. Then with one click I was back on my VPN. In the Mac settings go to Security and Privacy and go to Firewall and turn on Firewall options. Then turn off Stealth mode. With that one click I was back up and using the VPN.
Unfortunately, this does not apply to my case. I'm talking about VLAN and not VPN. Despite this, I checked the stealth mode and it is already disabled (btw it is only related to icmp applications like ping and related). Thank you in any case.
I encountered the exact same problem after updating to Big Sur.
Using a similar scenario, en0 for untagged traffic and vlan0 for tagged, both connected to my guest VMs in bridge mode.
In the untagged VLAN the guest VMs can communicate with any network device. In the tagged VLAN the guest VMs can only communicate with the macOS host.
I have an enterprise-grade switch, so I was able to do some port-level packet capture.
- packets sent from the guest VM are correctly forwarded out of vlan0 (tagged) and are obviously received by devices in this network segment.
- L2 broadcast packets (ARP) sent by other network devices in VLAN 16 are received by the VM - confirmed with Wireshark capture
- L2 unicast packets sent by other network devices to the VM are not received on the guest interface. I confirmed with a packet capture that the switch interface is sending them to the Mac, but it's as if macOS discards them, so they never reach the guest.
This leads me to believe the problem is with macOS, and nothing I will do on the guests or switch can fix it.
If I were to take an educated guess, I would say Big Sur has a different behavior on virtual interfaces, where it checks the destination Mac address against its own, instead of forwarding packets to the bridged interfaces. The bridge configuration seems identical.
Is there any way to check the Mac table on a virtual bridge?
bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=3<RXCSUM,TXCSUM> ether f2:18:98:1f:a4:64 Configuration: id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0 maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200 root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0 ipfilter disabled flags 0x0 member: vlan0 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 15 priority 0 path cost 0 member: en8 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 18 priority 0 path cost 0 member: en10 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 22 priority 0 path cost 0 media: autoselect status: active bridge101: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=3<RXCSUM,TXCSUM> ether f2:18:98:1f:a4:65 Configuration: id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0 maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200 root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0 ipfilter disabled flags 0x0 member: en0 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 4 priority 0 path cost 0 member: en9 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 20 priority 0 path cost 0 member: en11 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 23 priority 0 path cost 0 media: autoselect status: active
I answered my own question. Seems guest VM MAC addresses are learned on the bridge, so in theory macOS should forward packets to the guests. Probably the frames are dropped at the vlan0 interface
~ % ifconfig bridge100 addr 30:d1:7e:e6:e7:ca Vlan1 vlan0 635 flags=0<> c4:47:3f:ee:3e:6c Vlan1 vlan0 555 flags=0<> 0:c:29:6:38:fb Vlan1 en10 1193 flags=0<> 0:c:29:57:21:e9 Vlan1 en8 555 flags=0<>
My next question would be what are the differences en0 and vlan0.
Does anyone know what's the purpose of the SMART flag, and the AV and CHANNEL_IO options?
~ % ifconfig en0 link en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=50b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV,CHANNEL_IO> ether f0:18:98:f1:f6:cb ~ % ifconfig vlan0 link vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=3<RXCSUM,TXCSUM> ether f0:18:98:f1:f6:cb
Thank you for your interest in this.
I don't know the meaning of those options even if I remember SMART is deprecated.
Another strange thing in my case:
# ifconfig bridge100
member: vlan0 flags=3<LEARNING,DISCOVER> ----> OK see later, id the vlan public interface
member: en4 flags=3<LEARNING,DISCOVER> ----> NOTOK? this new interface has appeared under net prefs as VM NETWORK INTERFACE (!?). But I expected en0 (physical interface untagged)
# ifconfig vlan0 (OK)
vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
inet <MYPUBLICIP> netmask 0xffffff00 broadcast <BRD_ADDRESS>
# ifconfig en4
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
inet 169.254.71.128 netmask 0xffff0000 broadcast 169.254.255.255 ----> a probably unwanted DHCP configuration
# ifconfig en0
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::1010:6aed:c94e:4553%en0 prefixlen 64 secured scopeid 0x4
inet <MY_UNTAGGED_PRIVATEIP_NET> netmask 0xffffff00 broadcast <MY_UNTAGGED_PRIVATEIP_BRD>
media: autoselect (100baseTX <full-duplex,flow-control>)
At this point I cannot even find a way to set those options to test them.
For bridge100 I think your configuration is ok.
Each bridge interface should have one member interface from the host and one member interface from Fusion going to the guest. So bridge100 having vlan0 and en4 is normal, it means the two are bridged.
Definitely vlan0 and en0 should not be bridged.
# ifconfig en4 en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=400<CHANNEL_IO> ether 2a:7c:52:a1:4a:d7 inet 169.254.71.128 netmask 0xffff0000 broadcast 169.254.255.255 ----> a probably unwanted DHCP configuration
This is not normal. In bridge mode, the Fusion virtual interface should not have an IP address. I don't think it's related to your problem, but I suggest either way to double-check your VM network configuration.
You can think about it like this: bridge100 is like a virtual switch, and en4 is like a virtual port on this switch, created by Fusion. The VM network interface then connects to this port, in the same way a physical network interface connects to a physical switch.
Thank you for the explanations, you're right.
For the rest, I'm reading something about the network packet filter (pf) that maybe does not create the needed rules opportunely.
But this was related to NAT, I don't know if this can also suit in some way to our case.
I'm starting to read up on this, but it seems PF doesn't operate at layer 2.
If my assumptions are correct, the packets are dropped based on the destination MAC address.
You can test this by starting a packet capture on en4, or even in the VM itself, then look for ARP packets. These are L2 broadcast packets, and in my case I can see them reach the VM.
Just ping from a device outside the host to any IP address in your VLAN subnet, that isn't assigned. This way you should see continuous ARP packets for easier confirmation.
You can even ping the VM itself, but you may only see one ARP packet, because the VM will reply to it and then ARP is no longer required.
But you won't see any unicast L2 packets, i.e. packets where the destination MAC address is the VM. This leads me to think Big Sur is dropping them for some reason. Either due to the vlan0 interface options, or some bug or "feature".
I've filed this bug with Apple on Dec 21, 2020 and they seemed to have acknowledged it. The Feedback ID for this issue is FB8948116.
To speed up the process, it might help if you file a report as well under https://feedbackassistant.apple.com and refer to the above mentioned Feedback ID.
Also I've just verified that the bug is still there in the latest version of Big Sur beta 11.2.
Hi, I just disovered the same issue and added myself to the Apple Feedback thread. Hopefully this is fixed soon. I cannot believe nobody from Apple or VMware discovered this during testing, as it is essential functionality in my eyes. Thanks a lot for pointing this out. Best regards, Matthias.
So can you be a bit more specific? What exactly does not work? Using tagged VLANs with Fusion 12.1.1 in 11.3 works fine since last week. Let's see if it still does after the 11.3.1 update... 😉