oriolo76
Contributor
Contributor

Big issue between VMware Fusion 12 and Big Sur on VLAN

Good morning to everybody,

I have this big problem with Fusion and Big Sur. The scenario is the following.

Host Machine:

VMware Fusion 12.1.0

MacOS Big Sur 11.0.1

Network relevant details:

en0 -> untagged interface adapter on a private network

vlan0 -> tagged interface adapter on a public network

Guest Machine

OS: Ubuntu 20.04.1

Network: public ip on a network adapter bridged on vlan0

The problem is that the guest is unable to ping outside the host (guest <-> host is ok).

In other words, the default gateway for the vlan0 doesn’t respond to the guest (in any case both firewalls are off and forwarding is enabled).

I also used tcpdump and I have verified that packets go out from guest and reach the host, but they do not come back.

I already tried many of your dns or nat tested solutions to the related issues, but the problem persists.

What is interesting is if I move the guest bridge on the en0 (of the host), reconfiguring it with a public ip, all works (as expected).

This for saying that the problem should be strictly related to VLAN.

Note: before updating (both to Fusion 12 and Big Sur) all was working between host and guest (both having a public IP address) and routing to Internet.

Any help or suggestions is very appreciated.

Thank you.

Cheers,

—Carlo

0 Kudos
20 Replies
JoshCastro
Contributor
Contributor

I have been working on this issue for a while and could not get it figured out. Then with one click I was back on my VPN. In the Mac settings go to Security and Privacy and go to Firewall and turn on Firewall options. Then turn off Stealth mode. With that one click I was back up and using the VPN.

0 Kudos
oriolo76
Contributor
Contributor

Unfortunately, this does not apply to my case. I'm talking about VLAN and not VPN. Despite this, I checked the stealth mode and it is already disabled (btw it is only related to icmp applications like ping and related). Thank you in any case.

 

0 Kudos
JoshCastro
Contributor
Contributor

Sorry about that, I totally read VLAN as VPN. 

0 Kudos
carlto76
Contributor
Contributor

Thank you in any case.

Anyone else is using tagged/untagged VLAN with Fusion 12 and Big Sur?

0 Kudos
raducosma
Contributor
Contributor

I encountered the exact same problem after updating to Big Sur.

Using a similar scenario, en0 for untagged traffic and vlan0 for tagged, both connected to my guest VMs in bridge mode.

In the untagged VLAN the guest VMs can communicate with any network device. In the tagged VLAN the guest VMs can only communicate with the macOS host.

I have an enterprise-grade switch, so I was able to do some port-level packet capture.

- packets sent from the guest VM are correctly forwarded out of vlan0 (tagged) and are obviously received by devices in this network segment.

- L2 broadcast packets (ARP) sent by other network devices in VLAN 16 are received by the VM - confirmed with Wireshark capture

- L2 unicast packets sent by other network devices to the VM are not received on the guest interface. I confirmed with a packet capture that the switch interface is sending them to the Mac, but it's as if macOS discards them, so they never reach the guest.

 

This leads me to believe the problem is with macOS, and nothing I will do on the guests or switch can fix it.

If I were to take an educated guess, I would say Big Sur has a different behavior on virtual interfaces, where it checks the destination Mac address against its own, instead of forwarding packets to the bridged interfaces. The bridge configuration seems identical.

Is there any way to check the Mac table on a virtual bridge?

bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether f2:18:98:1f:a4:64 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: vlan0 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 15 priority 0 path cost 0
	member: en8 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 18 priority 0 path cost 0
	member: en10 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 22 priority 0 path cost 0
	media: autoselect
	status: active
bridge101: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether f2:18:98:1f:a4:65 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en0 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 4 priority 0 path cost 0
	member: en9 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 20 priority 0 path cost 0
	member: en11 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 23 priority 0 path cost 0
	media: autoselect
	status: active

 

raducosma
Contributor
Contributor

I answered my own question. Seems guest VM MAC addresses are learned on the bridge, so in theory macOS should forward packets to the guests. Probably the frames are dropped at the vlan0 interface

~ % ifconfig bridge100 addr 
30:d1:7e:e6:e7:ca Vlan1 vlan0 635 flags=0<>
c4:47:3f:ee:3e:6c Vlan1 vlan0 555 flags=0<>
0:c:29:6:38:fb Vlan1 en10 1193 flags=0<>
0:c:29:57:21:e9 Vlan1 en8 555 flags=0<>

 

My next question would be what are the differences en0 and vlan0.

Does anyone know what's the purpose of the SMART flag, and the AV and CHANNEL_IO options?

~ % ifconfig en0 link  
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=50b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV,CHANNEL_IO>
	ether f0:18:98:f1:f6:cb 
~ % ifconfig vlan0 link
vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether f0:18:98:f1:f6:cb 

 

oriolo76
Contributor
Contributor

Thank you for your interest in this.

I don't know the meaning of those options even if I remember SMART is deprecated.

Another strange thing in my case:

# ifconfig bridge100

...

member: vlan0 flags=3<LEARNING,DISCOVER>  ----> OK see later, id the vlan public interface

member: en4 flags=3<LEARNING,DISCOVER> ----> NOTOK? this new interface has appeared under net prefs as VM NETWORK INTERFACE (!?). But I expected en0 (physical interface untagged)

...

# ifconfig vlan0 (OK)

vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=3<RXCSUM,TXCSUM>

ether 38:c9:86:3c:a7:11 

inet <MYPUBLICIP> netmask 0xffffff00 broadcast <BRD_ADDRESS>

# ifconfig en4  

en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=400<CHANNEL_IO>

ether 2a:7c:52:a1:4a:d7 

inet 169.254.71.128 netmask 0xffff0000 broadcast 169.254.255.255  ----> a probably unwanted DHCP configuration

# ifconfig en0

en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=50b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV,CHANNEL_IO>

ether 38:c9:86:3c:a7:11 

inet6 fe80::1010:6aed:c94e:4553%en0 prefixlen 64 secured scopeid 0x4 

inet <MY_UNTAGGED_PRIVATEIP_NET> netmask 0xffffff00 broadcast <MY_UNTAGGED_PRIVATEIP_BRD>

nd6 options=201<PERFORMNUD,DAD>

media: autoselect (100baseTX <full-duplex,flow-control>)

status: active

0 Kudos
raducosma
Contributor
Contributor

At this point I cannot even find a way to set those options to test them. 

 

For bridge100 I think your configuration is ok.

Each bridge interface should have one member interface from the host and one member interface from Fusion going to the guest. So bridge100 having vlan0 and en4 is normal, it means the two are bridged.

Definitely vlan0 and en0 should not be bridged.

 

# ifconfig en4  
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 2a:7c:52:a1:4a:d7 
inet 169.254.71.128 netmask 0xffff0000 broadcast 169.254.255.255  ----> a probably unwanted DHCP configuration

This is not normal. In bridge mode, the Fusion virtual interface should not have an IP address. I don't think it's related to your problem, but I suggest either way to double-check your VM network configuration.

You can think about it like this: bridge100 is like a virtual switch, and en4 is like a virtual port on this switch, created by Fusion. The VM network interface then connects to this port, in the same way a physical network interface connects to a physical switch.

0 Kudos
oriolo76
Contributor
Contributor

Thank you for the explanations, you're right.

For the rest, I'm reading something about the network packet filter (pf) that maybe does not create the needed rules opportunely.

But this was related to NAT, I don't know if this can also suit in some way to our case.

 

 

 

0 Kudos
raducosma
Contributor
Contributor

I'm starting to read up on this, but it seems PF doesn't operate at layer 2.

If my assumptions are correct, the packets are dropped based on the destination MAC address.

You can test this by starting a packet capture on en4, or even in the VM itself, then look for ARP packets. These are L2 broadcast packets, and in my case I can see them reach the VM.

Just ping from a device outside the host to any IP address in your VLAN subnet, that isn't assigned. This way you should see continuous ARP packets for easier confirmation.

You can even ping the VM itself, but you may only see one ARP packet, because the VM will reply to it and then ARP is no longer required.

 

But you won't see any unicast L2 packets, i.e. packets where the destination MAC address is the VM. This leads me to think Big Sur is dropping them for some reason. Either due to the vlan0 interface options, or some bug or "feature".

0 Kudos
carlto76
Contributor
Contributor

Yes you have figured out the problem.

I'm not able to setup different options for vlan0, if you get news please update me!

thank you very much.

0 Kudos
fhibler86
Contributor
Contributor

Hey,

I've filed this bug with Apple on Dec 21, 2020 and they seemed to have acknowledged it. The Feedback ID for this issue is FB8948116.

To speed up the process, it might help if you file a report as well under https://feedbackassistant.apple.com and refer to the above mentioned Feedback ID.

Also I've just verified that the bug is still there in the latest version of Big Sur beta 11.2.

Best regards,
Florian

Screen Shot 2021-01-25 at 12.34.33.png

0 Kudos
carlto76
Contributor
Contributor

dit it. My number is FB8980734.

0 Kudos
matthiaslink77
Contributor
Contributor

Hi, I just disovered the same issue and added myself to the Apple Feedback thread. Hopefully this is fixed soon. I cannot believe nobody from Apple or VMware discovered this during testing, as it is essential functionality in my eyes. Thanks a lot for pointing this out. Best regards, Matthias.

0 Kudos
carlto76
Contributor
Contributor

Yes that’s unacceptable. I discovered and alerted about this bug from the beginning of BigSur, but nothing has been done by Apple nor VMWare in the subsequent updates.

0 Kudos
carlto76
Contributor
Contributor

Habemus fix. Solved with BigSur 11.3.

0 Kudos
matthiaslink77
Contributor
Contributor

Yes - I just checked and it's working now. Finally 🙂

0 Kudos
hirochan_hh
Contributor
Contributor

I faced the same problem and haven't solved it yet.

VMware Fusion 12.1.1

MacOS Big Sur 11.3

0 Kudos
matthiaslink77
Contributor
Contributor

So can you be a bit more specific? What exactly does not work? Using tagged VLANs with Fusion 12.1.1 in 11.3 works fine since last week. Let's see if it still does after the 11.3.1 update... 😉

0 Kudos