VMware Communities
gringley
Hot Shot
Hot Shot

Fusion Networking and WiFi

Tonight I checked my iMac Pro, and found that if I connect VMs to WiFi they had poor network connectivity, evidenced by an inability of the Microsoft Store to update apps and mediocre Speedtest results.  Ethernet is fine with Speedtest results in the guests generally the same as the host.  WiFi has performed poorly in Fusion 12, and I am wondering if there is any effort around resolving the issues?  The only networking I ever use is bridged networking. 

Reply
0 Kudos
21 Replies
gringley
Hot Shot
Hot Shot

Checked again after 11.4 upgrade and still broken.  Playing around a bit QUIC seems to work so Google sites and others using UDP 443 work OK. SO why is TCP via Wi-Fi in bridged networking broken?

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

OK so now I know why WiFI is broken in Fusion 12 bridged networking, but I have no idea how to work around it.  When Fusion builds the bridge interface between the guest and the WiFi Fusion configures "MACNAT" and then the bridge interface is mostly useless.  The ifconfig in Big Sur is still using man pages from 2008 so no idea how I could reconfigured the bridge interface after it is built?  Is there a way to see the scripts Fusion uses to build the bridge interfaces?

 

In this example below I have two guests running.  The first guest on en11 was connected to WiFi and got MACNATted.  The second guest on en10 was set to Autodetect, did not get MACNATted and works fine.

en11: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

ether a2:5c:45:1c:41:74 

media: autoselect

status: active

bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=3<RXCSUM,TXCSUM>

ether d2:81:7a:6d:1d:64 

Configuration:

id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0

maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200

root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0

ipfilter disabled flags 0x0

member: en1 flags=8003<LEARNING,DISCOVER,MACNAT>

        ifmaxaddr 0 port 7 priority 0 path cost 0

member: en11 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 22 priority 0 path cost 0

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

en10: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

ether 3e:a5:25:f5:21:d5 

media: autoselect

status: active

bridge101: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=3<RXCSUM,TXCSUM>

ether d2:81:7a:6d:1d:65 

Configuration:

id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0

maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200

root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0

ipfilter disabled flags 0x0

member: en0 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 4 priority 0 path cost 0

member: en10 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 24 priority 0 path cost 0

media: autoselect
status: active

 

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

To go a step further I can do this and see the NATs, but Internet Sharing is not enabled?

~ % ifconfig bridge100

bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=3<RXCSUM,TXCSUM>

ether d2:81:7a:6d:1d:64 

Configuration:

id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0

maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200

root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0

ipfilter disabled flags 0x0

member: en1 flags=8003<LEARNING,DISCOVER,MACNAT>

        ifmaxaddr 0 port 7 priority 0 path cost 0

member: en11 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 22 priority 0 path cost 0

Address cache:

6c:70:9f:d7:6f:8a Vlan1 en1 974 flags=0<>

14:7d:da:8a:87:c0 Vlan1 en1 1187 flags=0<>

3c:7d:a:4:e0:63 Vlan1 en1 1091 flags=0<>

9c:e6:5e:d9:b9:68 Vlan1 en1 1147 flags=0<>

c8:69:cd:31:45:31 Vlan1 en1 1116 flags=0<>

bc:ec:5d:b:a7:8c Vlan1 en1 969 flags=0<>

dc:52:85:35:a4:e7 Vlan1 en1 1183 flags=0<>

dc:52:85:d1:58:17 Vlan1 en1 1166 flags=0<>

88:6a:e3:a2:44:f2 Vlan1 en1 1168 flags=0<>

40:cb:c0:ad:1e:1 Vlan1 en1 1091 flags=0<>

c8:8:e9:54:85:1f Vlan1 en1 1185 flags=0<>

98:18:88:ce:4d:5 Vlan1 en1 1127 flags=0<>

0:c:29:c4:77:ec Vlan1 en11 1195 flags=0<>

dc:52:85:32:b4:c8 Vlan1 en1 1064 flags=0<>

d4:90:9c:cf:c3:f Vlan1 en1 1192 flags=0<>

14:7d:da:c1:c5:b3 Vlan1 en1 1192 flags=0<>

3c:7d:a:a:93:e0 Vlan1 en1 1192 flags=0<>

b6:35:3c:92:71:2b Vlan1 en1 1191 flags=0<>

dc:52:85:d1:9f:79 Vlan1 en1 1122 flags=0<>

d0:81:7a:d6:89:85 Vlan1 en1 1191 flags=0<>

d4:90:9c:cf:f:fc Vlan1 en1 1192 flags=0<>

58:d3:49:d8:ae:f6 Vlan1 en1 1194 flags=0<>

dc:a4:ca:a2:4b:27 Vlan1 en1 1196 flags=0<>

MAC NAT list:

en11 192.168.128.208 0:c:29:c4:77:ec 1063

en1 192.168.128.205 38:f9:d3:cf:52:62 1194

en11 fd9f:44f2:913e:3:20c:29ff:fec4:77ec 0:c:29:c4:77:ec 821

en11 fe80::20c:29ff:fec4:77ec 0:c:29:c4:77:ec 1195

en1 fd9f:44f2:913e:3:189f:9883:34ab:e84a 38:f9:d3:cf:52:62 797

en1 fe80::14d1:6fa5:6ee9:200c 38:f9:d3:cf:52:62 1191

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

The MACNAT for WiFi issue was not fixed in 12.2.0

Reply
0 Kudos
ivivanov
VMware Employee
VMware Employee

Hello,

To give some background: in macOS Big Sur Apple have disabled loading of 3rd-party kernel extensions. Fusion Networking was implemented this way, so we had to refactor it and use the new vmnet API provided by Apple for implementing virtual networking. At this point most of the internal implementations details have moved from our (VMware) code to macOS. The bridge interfaces created are entirely managed by macOS and are a part of their implementation approach. The macOS API we are calling looks like "start a new virtual interface in bridged mode using this physical adapter as a carrier". All details after that are handled by macOS, so technically the question for the MACNAT flag should be targeted to Apple.

A wild guess (based on our legacy networking implementation using a kernel extension - I could not find any meaningful information about the MACNAT flag on the internet): it is hard to implement bridged-mode with WiFi. Based on some WiFi router configurations it is even impossible. An Ethernet adapter can be configured to send "raw" packets, that means the client software (guest OS in our case) had created an entire ethernet frame with filled source and destination MAC addresses, that is ready to be sent on the wire. We use the ethernet NIC only as a media. For WiFi the story is different, we cannot send a packet putting a MAC address of the sender different than the physical address, because the packet would be dropped by the router. To overcome this limitation the macOS has to implement something similar to NAT service on MAC level - i. e. inspect the packets coming from the guest OS, replace the source MAC address while keeping track of the destination, later when a reply packet comes it changes the destination MAC address to match the one of the VM NIC and routes the packet to the corresponding virtual interface. Again, this is entirely implemented by Apple on Big Sur and above, so it is beyond the control of VMware Fusion.

__________
It is worse!
gringley
Hot Shot
Hot Shot

You are conceding that every Fusion customer with a MacBook can no longer use bridge mode networking with Fusion unless they buy an Ethernet dongle or adapter. I do not have a developer relationship with Apple so I cannot complain to them.  I would think VMware has that relationship and would want their product to keep working?

Agreed on that MACNAT thing.  These discussions now appear in the search results for MACNAT at least.

Reply
0 Kudos
ivivanov
VMware Employee
VMware Employee

> You are conceding that every Fusion customer with a MacBook can no longer use bridge mode networking with Fusion unless they buy an Ethernet dongle or adapter. 

I could not speak on behalf of every Fusion customer with a MacBook. As I said, it is hard to implement bridged-mode with Wi-Fi. We (Fusion) had issues with that before in kernel-extesion mode, now that the implementation is moved to macOS with the Apple vmnet API we see kind of similar issues. It depends mostly on the environment - the router and the Wi-Fi security configuration. There is no good way to implement it and the whole approach is "best effort". Just to mention it does not work at all in our corporate Wi-Fi network (VM is connected to the network in bridged mode, but there is no traffic at all), but the same MacBook with the same Fusion installation is working OK in my home environment with my home Wi-Fi. 

> I do not have a developer relationship with Apple so I cannot complain to them.

I think everyone with Apple ID can file requests to Apple through https://feedbackassistant.apple.com/

> I would think VMware has that relationship and would want their product to keep working?

I hope so as well :-). The initial version of macOS vmnet framework did not provide bridge mode at all - only NAT and host-only. We have requested support for bridged mode and it was added in macOS Catalina, however it supported only Ethernet, but no Wi-Fi physical adapters. We have explicitly asked for Wi-Fi bridging as well and Apple provided it as a second step. Again, for bridged mode over Wi-Fi the problem is not trivial and heavily depends on the environment. Your specific question (about the MACNAT flag) is entirely an implementation detail of Apple vmnet framework, so I cannot answer that question - Fusion is only invoking the API for creating a VM virtual network interface in bridged mode passing the Wi-Fi adapter as a physical NIC. I don't think either it is appropriate for us to be a proxy to Apple, because most likely the issue is not in the product, but in the environment.

__________
It is worse!
Reply
0 Kudos
gringley
Hot Shot
Hot Shot

Hopefully you can see from rest of the posts here is that there is quite a bit of demand for bridged networking and it is in VMware's interest to push Apple on this as us pebbles do not really get much of a vote.  It is your product that suffers as a normal Macintosh use case would never invoke a bridge mode from what I have seen.  You are not a proxy you are a direct customer of Apple's operating system APIs and if they goof on it you should be calling them out.

I realized from your description and another post that maybe Apple was adding a tag or something in the bridge.  I found tonight my Ubuntu guests could apt on Wi-Fi if I set the guest MTU to 1499 - which is REALLY ANNOYING that we have had all this difficulty for one byte.  So I think there is a workaround now but agree its quite onerous to have to remember to change the MTU on every new guest one deploys if the Mac only has a WiFi adapter.

Reply
0 Kudos
ivivanov
VMware Employee
VMware Employee

>I found tonight my Ubuntu guests could apt on Wi-Fi if I set the guest MTU to 1499 - which is REALLY ANNOYING that we have had all this difficulty for one byte.  So I think there is a workaround now but agree its quite onerous to have to remember to change the MTU on every new guest one deploys if the Mac only has a WiFi adapter.

I am glad that you were able to resolve the issue you were having. Thanks for letting us know, it is good to know this, however the MTU size is not the main/only reason for issues in all of the cases. I Just tried ping ping -M do -s 1472 (for a total of packet size of 1500) and MTU size set to 1500 in the guest and on the virtual adapter through in bridged mode through Wi-Fi and it worked for me (showing the MTU size of 1500 was OK). In your case it was 1499, in other it might be 1498, or 1400. There is no reasonable default to set (OK, the "reasonable" default is the industry-standard default of 1500 bytes). Besides as you said it has to be configured in the guest as well, not (just) in the virtual NIC.

> You are not a proxy you are a direct customer of Apple's operating system APIs.

I get your point. I agree in general but again - this specific issue is highly dependent on the environment. When it didn't work in our corporate network, I filed a report to Apple about it, this is what they replied after a few rounds of providing additional information:

Wi-Fi bridging can’t work on corporate Wi-Fi networks that only allow a single IP address (DHCP lease, effectively).

If I am about to file a bug report to Apple about it, I need to provide host support bundle, packet captures, etc. Given the issue is not reproduced locally, I cannot give them the information they need, so it is not likely they make any progress. So I need to reach out to the end customers to provide all that information and relay it to Apple. In many cases this information is not available, as the customers become unresponsive, there are legal concerns here as well. So in the ultimately I can see this going to a dead end, until reproduced locally, but for this we need to have at least a vague idea what to look for (apart from the MTU size and the single-IP Wi-Fi policy, that we know about). I cannot think of any changes we could make to our code (or ask Apple to do in macOS) to overcome these issues, as they are related to the specific environment.

__________
It is worse!
Reply
0 Kudos
gringley
Hot Shot
Hot Shot

OK - I did the ping command like you did by the way and it worked even though TCP traffic did not.  

This I do not really get "Wi-Fi bridging can’t work on corporate Wi-Fi networks that only allow a single IP address (DHCP lease, effectively)." The problem existed when I used Apple's Airport Express so it does happen on "non-corporate" Wi-Fi networks.  Today I use a Meraki MR-52 so my home network qualifies as "corporate" I guess?!? I then connected my Wi-Fi to the AT&T U-Verse modem's Wi-Fi so as to be on "home" network and reproduced the problem.  So the conditions for reproduction are:

Ubuntu server guest.  20.04.3 LTS so we can show we are using a supported OS.

Issue terminal command 'sudo apt update' - On a "bad" day you will see failures here but often this command completes.  

Issue terminal command 'sudo apt full-upgrade' - When the condition exists some of the packages will fail.

Issue the command 'sudo ip link set dev ens160 mtu 1499' then rerun the failed command(s) above and see them complete normally.

Testing using apt automates file checksumming over a reusable variety of file sizes.  The defect causes hash failures which in turn show as errors in the update and upgrade process.

I realize I do have a channel with Apple in that I have the public betas as guests.  I need to figure out a reproducible condition in macOS and try feedback there I guess?

At this point also it appears the defect only manifests in TCP sessions also, but I am not sure on that.  QUIC is UDP and ping is ICMP and those seem to have worked when TCP does not.

Reply
0 Kudos
ivivanov
VMware Employee
VMware Employee

Thanks for the update. After following your steps I was able to reproduce the problem locally. It was resolved if I set the MTU in the guest OS to 1499 and also it was not reproduced with NAT mode with MTU 1500.

I tried to get a packet capture on the bridgeXXX interface, however for some reason only the outgoing traffic was recorded, but not the inbound (where the large packets would be expected). Now that I have a local reproduction I can do additional investigation and file a report to Apple about this issue. Thanks for sharing the exact steps.

__________
It is worse!
Reply
0 Kudos
gringley
Hot Shot
Hot Shot

I tried a Monterey Beta 10 guest by enabling Remote Login then using Transmit on the Big Sur host to open an SFTP session and move files back and forth that way.  From host to guest (Download) I could get a fail on the first attempt but each subsequent attempt with the same file worked.  This tells me something else is going on and I do not have a valid test.  From guest to host I found it was running at bps speeds and discovered I had done the vmxnet3 workaround to get my Monterey Beta running initially.  I switched the guest back to e1000e and then I could transfer from guest to host (upload) error free at MTU 1500.

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

You know what...I am on Monterey 12.0.1 and I cannot reproduce this problem anymore.  Perhaps Apple figured something out?

Reply
0 Kudos
ivivanov
VMware Employee
VMware Employee

Hello,

Thanks for the update.

Yes, this is one option (let's hope this is the case). Another one is that the problem is a race condition and it is not reproducible each time, so something in the timing changed on the latest Monterey (I am just speculating here). Unfortunately I did not have time to experiment further and file a bug to Apple.

Regards,
Ivan

__________
It is worse!
Reply
0 Kudos
gringley
Hot Shot
Hot Shot

I think it is resolved. I created a Ubuntu 18.04 server from an old ISO. I was able to fully update it and then do-release-upgrade to 20.04 over the WI-FI adapter with no errors. 
I realize that as I skipped macOS 12.6.1 I cannot say if it is fixed as well?

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

Well honeymoon over as they say.  None of the Windows guests I have tried tonight can get a DHCP address if bridged to Wi-Fi.

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

Today I checked my guests and my Mac, Windows and Ubuntu guests were all normal when bridged to my WiFi adapter.  I am on macOS Monterey 12.3.1 and Fusion 12.2.3.  I have both IPv4 and IPv6 enabled as well.  

rfevrier
Contributor
Contributor

Greetings,  I get these VMs with prebuilt applications and I am still have in the same issue with both Big Sur and Monterey, I am not able to get an IP address for the bridged network.  However, I noticed an older version of the VM that was released I’m Oct 2019 (older version of Linux) seems to work fine, but newer releases don’t work.  For the newer version, I have modified .vmx file to ethernet0.virtualDev = "vmxnet3" but I still can’t get it to work(won’t assign and IP).  I was wondering whether there is something you would recommend I do in Linux that can fix this issue?

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

You can try lowering the MTU in the VM guest.  Looking back over my notes lowering the MTU from 1500 to 1499 fixed me initially.  I am assuming that these "pre-built" VMs should not be updated in any way thus updating the vmtools would not be an option.?

Reply
0 Kudos