This issue is certainly not directly firmware related or broadcom related, I have Intel Network Adapters in my servers and experience the same issues.

Also I do not believe that neccesarily all "disconnect" issues are directly the same issue we are experiencing here, though most have this identical issue. Though some have a very simular issue and resolve it via some means, after all disconnects and network issues do exist with drivers, firmware, ... Often though to make out what is this bug or what is a random issue.

I've been experiencing the same problem for several weeks now, same story, x64 Guest OS, Dell 2950 server and Broadcom NetXtreme II NIC. I'm suprised that no one else has not seen (or at least not mentioned) delayed write errors which are reported in the event log on my Guest OS every time the problem surfaces, see below. Unforunately, the information revealed by this error has not gotten me any closer to a solution.

Event Type: Warning
Event Source: MRxSmb
Event Category: None
Event ID: 50
Date: 4/11/2008
Time: 12:09:22 AM
User: N/A
Computer: EXCHANGE
Description:
{Delayed Write Failed} Windows was unable to save all the data for the file \Device\LanmanRedirector. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00040004 00560002 00000000 80040032
0010: 00000000 c000020c 00000000 00000000
0020: 00000000 00000000 c000020c

Well, there's a core group of people that are clearly experiencing the exact same problem. Ignore the obvious differences and it's pretty clear.

The core as I see it:

• windows 64-bit host
• broadcom NIC
• usually a 64-bit guest, but not always
• typically a multiprocessor guest

---

As a side-note, I switched my problem guest from dual processor to single a couple weeks ago. During that time, I didn't have a single network drop. I switched it back to dual processor and had two drops in four days. This is not conclusive though, since my load level was also low during that time. I'm slammed at work and don't have time to mess with it anymore.

This server role is becoming more critical to production, so I moved it to a physical server. I had $20K in the budget this quarter to start the VI3 transition. I'm not really happy about giving that money to vmware at this point, but there's nothing comparable at the moment.

i didn`t see the network issue for a long time, but a customer had it today and i had a chance to analyze.

one VM he was using for admin purpose (cisco tools) lost it`s BRIDGED network connection.

i could not make it work again, reboot of VM didn`t help.

i switched vm from bridged network to host-only-network and set a different IP inside VM, but that didn`t help either.
i also changed the VMs nic from AMD PCNet to e1000 , but that also didn`t help.

here are some more details:

ping
from host (192.168.109.1)
to VM (192.168.109.10)
connected via vmnet1

ping request times out

C:\Programme\VMware\VMware Server>vnetsniffer /e vmnet1
len 74 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 IP src 192.168.109.1 dst 192.168.109.10 ICMP ping request
len 74 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 IP src 192.168.109.1 dst 192.168.109.10 ICMP ping request
len 74 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 IP src 192.168.109.1 dst 192.168.109.10 ICMP ping request

as we can see, the packets from the host appear on vmnet1 - but no response from VM.

now - vice versa

ping
from VM (192.168.109.10)
to host (192.168.109.1)
but ping in VM tells that "destination host unreachable"

let`s take a look:

C:\Programme\VMware\VMware Server>vnetsniffer /e vmnet1
len 42 src 00:0c:29:07:db:b4 dst ff:ff:ff:ff:ff:ff ARP sender 00:0c:29:07:db:b4 192.168.109.10 target 00:00:00:00:00:00 192.168.109.1 ARP request
len 42 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 ARP sender 00:50:56:c0:00:01 192.168.109.1 target 00:0c:29:07:db:b4 192.168.109.10 ARP reply
len 42 src 00:0c:29:07:db:b4 dst ff:ff:ff:ff:ff:ff ARP sender 00:0c:29:07:db:b4 192.168.109.10 target 00:00:00:00:00:00 192.168.109.1 ARP request
len 42 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 ARP sender 00:50:56:c0:00:01 192.168.109.1 target 00:0c:29:07:db:b4 192.168.109.10 ARP reply
len 42 src 00:0c:29:07:db:b4 dst ff:ff:ff:ff:ff:ff ARP sender 00:0c:29:07:db:b4 192.168.109.10 target 00:00:00:00:00:00 192.168.109.1 ARP request
len 42 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 ARP sender 00:50:56:c0:00:01 192.168.109.1 target 00:0c:29:07:db:b4 192.168.109.10 ARP reply
len 42 src 00:0c:29:07:db:b4 dst ff:ff:ff:ff:ff:ff ARP sender 00:0c:29:07:db:b4 192.168.109.10 target 00:00:00:00:00:00 192.168.109.1 ARP request
len 42 src 00:50:56:c0:00:01 dst 00:0c:29:07:db:b4 ARP sender 00:50:56:c0:00:01 192.168.109.1 target 00:0c:29:07:db:b4 192.168.109.10 ARP reply

as we can see, VM sends ARP to vmnet1, packets pass vmnet1, reaching vmnet1 virtual host interface and host is giving arp reply.

i can see, that host has learned correct MAC/IP of VM (arp -a) , but it seems that VM never receives those arp replies.

VM doesn`t receive ANY packet, as the interface statistics tell.

that may explain, why VM is sending arp request again and again.

now comes the best:

If i assign the VM`s network identity (ethernet0.generatedAddress) to a different VM on same host (e.g. linux vm instead of windows), the problem remains and the VM inherits the network issue - the linux vm has now the same symptom as the windows VM.

BUT - if i change the VMs mac adress to a different one, the problem goes away.

If i revert the mac, the problem re-appears.

it seems, that the vmware virtual networking/switch has got "stuck" with that specific mac adress and doesn`t forward any packets into a VM anymore.

so we have:

packet from guest-os -> vmnic -> vmnet1 -> vmnet1-host-nic -> host-os --->OK!
packet from host-os -> vmnet1-host-nic -> vmnet1 --|here must be a problem| -> vmnic ->guest-os --->NotOK!!!

this really looks like an issue with vmware virtual networking, i.e. the virtual hub/switch implementation.

vmware, will you start help finding the root cause of this this serious problem, please ?

regards
roland

OK, so we took the following steps and the server has not had delayed write errors nor the bridged networking failed:

Recap:

We are using Dell 1950s & 2950s with dual onboard Broadcomm Nic's.

64x 2003 SP2 R2 with all patches as the host

64x 2003 SP2 R2 with all the patches as the guests

I have edited both guest servers with the following settings:

This seems to have solved the problem (knock on wood) I have been running our coprorate servers (SQL) with these settings and they have not dropped off yet (it used to happen once every 3-4 days),

Question: has any turned off VM Bridge Protocol from the server's primary interface and still had this problem?

While this may be re-hashing old suggestions, our workaround (other than install Linux like we did) was to turn off the VMBridge Protocol from the Windows server's interface, then turn off any MS file sharing & MS Client bindings on the remaining interfaces and leave the VMBridge Protocol on those same remaining interfaces.

This means, of course, that you'll have to change your default VM bridge, but it seems to work nicely w/o any problems in our environment which is mostly Dell boxes.

This is what I did. My guest server network card was failing EVERY night during the backup. Since going to 1 proc, has not failed yet - it's been over 55 days! I think that's the proof in the pudding right there! However, the real test would be to go back to 2 procs and retest. Haven't done that since I don't want my Exchange server going down again!

Glad to see some activity on this thread again although it's a shame that VMware remains silent on the matter... I recently put new Intel PRO 1000/PT Dual Port adapter in one of my servers that was experiencing this problem and it's been running stable for the past 48 hours or so, that isn't enough time to get excited though. The 1 processor solution seems to be promising but I can't afford to lose processing power on my guest OS's.