Is VMware causing nentwork dropouts?

morgsdaly · ‎08-18-2013

Hello,

We have been having random network dropouts and every time it happens VMware seems to feature a lot in the logs. It might be symptomatic of something else, or maybe VMware Fusion is actually the root of the problem. I am hoping you guys and girls can help me work it out?

These are the first 100 log lines from the time the drop out starts on our OS X 10.8.4 Server:

19/08/13 12:07:56.084 PM vmnet-bridge[602]: Dynamic store changed

19/08/13 12:07:56.084 PM vmnet-bridge[602]: Adapter: en0, Inactive

19/08/13 12:07:56.000 PM kernel[0]: Intel82574L::setLinkStatus - not active 1face000, 0

19/08/13 12:07:56.000 PM kernel[0]: vmnet: bridge-en0: interface en is going DOWN

19/08/13 12:07:56.000 PM kernel[0]: vmnet: VMNetDisconnect called for port 0xffffff80365aa000

19/08/13 12:07:56.000 PM kernel[0]: en0: promiscuous mode disable succeeded

19/08/13 12:07:56.000 PM kernel[0]: vmnet: bridge-en0: disabled promiscuous mode

19/08/13 12:07:56.000 PM kernel[0]: vmnet: bridge-en0: filter detached

19/08/13 12:07:56.000 PM kernel[0]: vmnet: bridge-en0: down

19/08/13 12:07:56.000 PM kernel[0]: vmnet: bridge-en0: detached

19/08/13 12:07:57.085 PM pppd[73135]: l2tp_wait_input: Address deleted. previous interface setting (name: en0, address: 192.168.29.4), deleted interface setting (name: en0, family: Ether, address: 192.168.29.4, subnet: 255.255.255.0, destination: 192.168.29.255).

19/08/13 12:07:57.134 PM configd[20]: network changed: v4(ppp0:192.168.29.4, en0-:192.168.29.4) DNS- Proxy- SMB-

19/08/13 12:07:57.135 PM vmnet-bridge[602]: Dynamic store changed

19/08/13 12:07:57.136 PM vmnet-bridge[602]: Could not retrieve media for interface ppp0: Operation not supported on socket.

19/08/13 12:07:57.000 PM kernel[0]: vmnet: VMNetDisconnect called for port 0xffffff80361d7400

19/08/13 12:07:57.000 PM kernel[0]: vmnet: bridge-en0: filter detached

19/08/13 12:07:57.000 PM kernel[0]: vmnet: bridge-en0: down

19/08/13 12:07:57.000 PM kernel[0]: vmnet: bridge-en0: detached

19/08/13 12:07:57.000 PM kernel[0]: vmnet: Freeing hub at 0xffffff8035474000.

19/08/13 12:07:57.139 PM pppd[73135]: starting wait-interface timer for l2tp: 20 secs

19/08/13 12:07:57.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:57.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state DOWN

19/08/13 12:07:57.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state UP

19/08/13 12:07:57.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state DOWN

19/08/13 12:07:57.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state UP

19/08/13 12:07:57.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:58.000 PM kernel[0]: 22

19/08/13 12:07:58.000 PM kernel[0]: ip4_output (ipsec): error code 22

19/08/13 12:07:58.085 PM vmnet-bridge[602]: Dynamic store changed

19/08/13 12:07:58.000 PM kernel[0]: Ethernet [Intel82574L]: Link up on en0, 1-Gigabit, Full-duplex, Symmetric flow-control, Debug [796d,af48,0de1,0200,cde1,38ff]

19/08/13 12:07:58.000 PM kernel[0]: Intel82574L::setLinkStatus - active 1face000, b

19/08/13 12:07:58.000 PM kernel[0]: ip4_output (ipsec): error code 22

19/08/13 12:07:58.000 PM kernel[0]: vmnet: VNetUserIf_Create: created userIf at 0xffffff8031dad400.

19/08/13 12:07:58.000 PM kernel[0]: vmnet: VMNetConnect: returning port 0xffffff8031dad400

19/08/13 12:07:58.000 PM kernel[0]: vmnet: VMNET_SO_BINDTOHUB: port: paddr 00:50:56:e0:f7:b6

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Hub 2

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Port 0

19/08/13 12:07:58.085 PM vmnet-bridge[602]: Started bridge for 2, en0

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Port 1

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Port 2

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Port 5

19/08/13 12:07:58.000 PM kernel[0]: vmnet: Port 6

19/08/13 12:07:58.000 PM kernel[0]: vmnet: bridge-en0: media 20 dev 0xffffff8033a41428 family 2

19/08/13 12:07:58.000 PM kernel[0]: ip_output: can't update route after IPsec processing

19/08/13 12:07:58.000 PM kernel[0]: vmnet: bridge-en0: up

19/08/13 12:07:58.000 PM kernel[0]: vmnet: bridge-en0: attached

19/08/13 12:07:58.000 PM kernel[0]: en0: promiscuous mode enable succeeded

19/08/13 12:07:58.000 PM kernel[0]: vmnet: bridge-en0: enabled promiscuous mode

19/08/13 12:07:58.000 PM kernel[0]: vmnet: VNetUserIfFree: freeing userIf at 0xffffff8031dad400.

19/08/13 12:07:58.395 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 0000000000000009 vmnet1/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 100

19/08/13 12:07:58.395 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 0000000000000009 vmnet1/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 200

19/08/13 12:07:58.396 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 000000000000000A vmnet8/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 300

19/08/13 12:07:58.396 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 000000000000000A vmnet8/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 400

19/08/13 12:07:58.396 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(9) failed to send packet on InterfaceID 0000000000000001 lo0/6 to FF02:0000:0000:0000:0000:0000:0000:00FB:5353 skt 9 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 500

19/08/13 12:07:58.396 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(9) failed to send packet on InterfaceID 0000000000000001 lo0/6 to FF02:0000:0000:0000:0000:0000:0000:00FB:5353 skt 9 error -1 errno 55 (No buffer space available) 6348501084 MessageCount is 600

19/08/13 12:07:58.000 PM kernel[0]: code 22

19/08/13 12:07:58.000 PM kernel[0]: ip4_output (ipsec): error code 22

...

19/08/13 12:07:58.646 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 000000000000000A vmnet8/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 8028798216342761815 MessageCount is 700

19/08/13 12:07:58.897 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 0000000000000009 vmnet1/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 10643468882 MessageCount is 800

19/08/13 12:07:59.000 PM kernel[0]: 22

19/08/13 12:07:59.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:59.142 PM mDNSResponderHelper[76029]: do_mDNSInterfaceAdvtIoctl: ioctl call SIOCGIFINFO_IN6 failed - error (22) Invalid argument

19/08/13 12:07:59.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:59.479 PM mDNSResponder[100]: mDNSPlatformSendUDP: sendto(8) failed to send packet on InterfaceID 0000000000000009 vmnet1/4 to 224.0.0.251:5353 skt 8 error -1 errno 55 (No buffer space available) 10643469463 MessageCount is 900

19/08/13 12:07:59.000 PM kernel[0]: 22

19/08/13 12:07:59.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:59.690 PM pppd[73135]: l2tp_wait_input: Address added. previous interface setting (name: en0, address: 192.168.29.4), current interface setting (name: en0, family: Ether, address: 192.168.29.4, subnet: 255.255.255.0, destination: 192.168.29.255).

19/08/13 12:07:59.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:07:59.000 PM kernel[0]: ip_output: can't update route after IPsec processing

19/08/13 12:07:59.705 PM racoon[265]: IKE Packet: transmit success. (Information message).

19/08/13 12:07:59.705 PM racoon[265]: IKEv1 Information-Notice: transmit success. (Delete IPSEC-SA).

19/08/13 12:07:59.705 PM racoon[265]: IKE Packet: transmit success. (Information message).

19/08/13 12:07:59.706 PM racoon[265]: IKEv1 Information-Notice: transmit success. (Delete ISAKMP-SA).

19/08/13 12:07:59.706 PM racoon[265]: Connecting.

19/08/13 12:07:59.706 PM racoon[265]: IPSec Phase1 started (Initiated by me).

19/08/13 12:07:59.706 PM racoon[265]: IKE Packet: transmit success. (Initiator, Main-Mode message 1).

19/08/13 12:08:00.000 PM kernel[0]: ip4_output (ipsec): error code 22

…

19/08/13 12:08:00.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state DOWN

19/08/13 12:08:00.000 PM kernel[0]: vmnet: VMNetSetopt: Set link state UP

19/08/13 12:08:02.114 PM racoon[265]: IKE Packet: transmit success. (Phase1 Retransmit).

19/08/13 12:08:02.252 PM digest-service[76044]: label: default

19/08/13 12:08:02.252 PM digest-service[76044]: dbname: od:/Local/Default

Thanks

morgsdaly · ‎08-22-2013

Bump.

Maybe this will help. We isolated the network interface that VMware is using to the en1 interface so as what ever is happening would ot keep bringing the en0 interface down. This seems to have helped big time and now neither interface is being brought down, which is some kind of progress. But the logs are still filled with the following:

23/08/13 3:25:08.000 PM kernel[0]: vmnet: mbuf_dup() failed: 12.

23/08/13 3:25:08.000 PM kernel[0]: vmnet: Failed to deep copy mbuf: 12.

Just over and over again.

Any ideas?

Thanks

sr105 · ‎09-04-2013

I didn't look too closely at your logs, but I wonder if this is related to the bug I noticed? Whenever the link status changes on any external network interface to the host, Fusion restarts the NAT guest network. I'm not sure what it does to bridged or internal-only networking.

For example:

Ubuntu 12.04 guest
MacBook Pro with OSX 10.8.4
VMware Fusion 5 or 6
Wireless is default and active connection for host
guest is using NAT
Plug in thunderbolt ethernet adapter.

Now, dis-/connect the ethernet adapter to a hub or anything else that will cause a link status change. You'll see the network go down for the guest and resume within 15-60 seconds. I open an ssh connection to the guest from the host and run "watch -n 1 vmstat", but any continuous, regular output over the network will do.

Note: you don't need two external links to do this. Simply turning the wireless on/off above will achieve the same effect. The above scenario is simply a common one for me since I'm directly ethernet connected to a device I'm constantly rebooting for testing. Every time I reboot, my VM networking stalls. I've bought a small portable hub, to get around the problem.

sr105 · ‎09-04-2013

I may have misread your post. You're saying Fusion brings down ALL networking on the host?

morgsdaly · ‎09-10-2013

Indeed... It seemed that VMware was having an issue and this was bringing down the Network interface.

We couldn't work it out and in the end had to move it to a different machine.

nancyz · ‎09-16-2013

Hi sr105 This was resolved in your post https://communities.vmware.com/thread/453617

nancyz · ‎09-16-2013

Hi morgsdaly,

Are you using pptp VPN and bridged your VM to that adapter?

morgsdaly · ‎09-16-2013

I am pretty sure that there was no VPN involved.

It was just a basic bridged setup.

nancyz · ‎09-16-2013

Hi morgsdaly,

morgsdaly wrote:

I am pretty sure that there was no VPN involved.

It was just a basic bridged setup.

What do you mean by basic bridged setup? Did you use Fusion bridged VM or you setup bridge on your Mac os X server? Is there other network setup on your Mac or on your VM?

All

Is VMware causing nentwork dropouts?