VMware Cloud Community
Jocelyn_Viau
Contributor
Contributor

Intermitent network disconnect in VM using E1000E NIC

Hi,

We just installed three new Windows Server 2012 and one new Windows 8 VMs on ESXi 5.1. On those VMs, we get a lot of errors in the System Event log about network disconnects. According to the logs, the network disconnects for about one second as we get another event saying that a network connection was established at 1 Gbps just after. Just on the day of January 28, this problem occured 85 times on a single server VM.

Those VMs are using the recommended E1000E NIC with vmx-09 hardware version. The Tools are installed and are current. All VMs are connected to the same vSwitch which has a dedicated physical port to the gigabit network switch (no NIC teaming and no VLAN tagging). The VMs are running from the same datastore that uses local disks setup in RAID 10 with a hardware RAID controller (LSI 2108 chipset).

Of course, when the alledged disconnect occurs, Windows shuts down momentarily network bound services (such as NetBIOS Helper service) and brings it back right away when the reconnection occurs. This annoying problem also causes interruptions in file transfers on network shares, so it is not just a harmless error that can be ignored.

Here is an example of the disconnect event:

Log Name:      System
Source:        e1iexpress
Date:          2013-01-29 10:53:13
Event ID:      27
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      --- censored by me! ---
Description:
Intel(R) 82574L Gigabit Network Connection
Network link is disconnected.

Here is the reconnection 1 second later:

Log Name:      System
Source:        e1iexpress
Date:          2013-01-29 10:53:14
Event ID:      32
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      --- censored by me! ---
Description:
Intel(R) 82574L Gigabit Network Connection
Network link has been established at 1Gbps full duplex.

Here are some interesting facts about this problem:

  • We checked the physical link to the host and everything is fine. We don't have any disconnection reported by the host or the switch.
  • The problem occurs in all VM which have E1000E NIC, but not at the same time.
  • Changing the VM NIC type to E1000 solves the problem: no more network disconnection reported by the VM and file transfers are not interrupted anymore.
  • The disconnects are not evenly spaced in time. It may occur many times in the same minute and then nothing for more than an hour.

Has anyone had this kind of problem before? Any clue about what may be the cause?

We can still use the workaround we found and change the NICs to E1000 but it is not the one recommended for Windows 8 and Windows Server 2012.

Thanks for your help!

42 Replies
auburntigerrich
Contributor
Contributor

I was able to resolve our issue by switching the vNIC to "e1000E" type via vSphere Client.  My issue was with ESXi, not Workstation, so I'm afraid I can't speak to that...

Reply
0 Kudos
Akelsey
Contributor
Contributor

Hi everyone, i faced with problem as well.

I have a two network card "Intel Corporation 82574L Gigabit Network Connection", two guest OSes, Win2008 sp1 r2, and Windows 7 SP1, all VM are installed in ESXi 5.5. In VMs the nics are VMXNET 3 (i have already faced the problems with vpn connection on e1000e, and all new VMs are working with VMXNET3)

nic0 - vmkernel 192.168.50.30

When i am copying a big files inside VMs from physical comps - i lost connection to vmkernel. It restores after copying is completed. I cannot find any KB with solution. I tried to change to 100mbs, nothing changes.

Reply
0 Kudos
Akelsey
Contributor
Contributor

Hi again, i solved my issue today, key reason of issue was not esxi, it was an switch, i forgot the name, i replaced it via cisco, and now it is ok.

Reply
0 Kudos
junkoo
Contributor
Contributor

Hi,

I found Intel(R)  82574L Auto-Negotiation Link Setup  as below,

Intel® 82574 GbE Controller Family3.2.3 Auto-Negotiation & Link Setup Feature (Document)

http://www.versalogic.com/support/Downloads/PDF/Intel_82574L_Datasheet.pdf

p.43

3.2.3 Auto-Negotiation & Link Setup Feature

This event that "Network link is disconnected." comes from default setting as Auto-Negotiation "OFF"

that means before finished auto-negotiation,  this driver checking link condition and give back this message.

If appeared "Link up" sign  after this message(Link has been disconnected),  we can ignore this event.

But you wont show this event, you can remove with setting as Auto-Negotiation "OFF" on the device manager.

I hope this helped.

Reply
0 Kudos
Jocelyn_Viau
Contributor
Contributor

This information is interresting but it does not explain why the VM needs to re-negociate the network link and, when it occurs, all network connections are dropped. The problem is not just an annoying message. If it was, I would just have "muted" the alert in our monitoring system. The problem is the frequently dropping network connections and, when it occurs, any running transfer (ex: file copy) is aborted.

I posted the first article more than a year and a half ago and the problem is still unsolved even after several VMware patches. The problem seem to also have been ported to ESXi 5.5. At the moment, I have replaced all the network adapters of my VMs to VMXNET3. It is the only workaround that works for now.

Reply
0 Kudos
junkoo
Contributor
Contributor

Hi Leyuka,

Thank you for your reply and sorry such us delayed.

It seems  vmxnet 3 is  right solution as you said and I found article as below;

Choosing a network adapter for your virtual machine

http://kb.vmware.com/kb/1001805

-----

VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems:

32- and 64-bit versions of Microsoft Windows 7, 8, XP, 2003, 2003 R2, 2008, 2008 R2, Server 2012 and Server 2012 R2.

-----

Many thanks!



Reply
0 Kudos
JimAllen104
Contributor
Contributor

Greetings,

Ditto.  After switching VNIC from E1000 and E1000E on two different VMs (Windows Server 2008 R2) from E1000, to VMXNET2 (Enhanced).  The random 'disconnects' reported by the guest OS have stopped.

ESXi hosts 5.1.0 1483097

VM Version 7 (both guests)

VMWare Tools - current.

Reply
0 Kudos
jjharrison81
Contributor
Contributor

Has this been addressed by VMWare yet in any way?  If so, I'm having some trouble finding the right keywords to locate it.

I'm configuring several new 2012 VMs and before they are put into production I would like to know what the best practice is for a Server 2012 R2 NIC right now.

VMWare's KB on choosing a NIC is vague at best and completely ignores this problem with some NICs not actually working on Windows Server 2012 R2 in certain scenarios.  I am also seeing a bunch of articles on the VMXNET3 causing PSOD issues on Windows Server 2012 and that seems to be the only alternative to E1000 or E1000E right now.

Reply
0 Kudos
Corpus_no
Contributor
Contributor

Time goes and no solution?

We still have this issue in a fully patched 5.5 environment and it doesn't seem to be any final solution to this problem yet.

Of cause, there is always the option to change to VMXNET3 for the affected VM's, but I would really like to have the option to choose if it's supposed to work.

--

Øystein.

Reply
0 Kudos
TedBWork
Contributor
Contributor

I really dont think we are going to see anything until VMware 6.  There seems to be other threads that have similar problems and no results.  I was told that we can expect fixes with 2 major issues that will be resolved in 6.  However that doesn't mean that they wont fix this but I cant get any more info

Reply
0 Kudos
JSpoor
Contributor
Contributor

Does anyone have a solution for this?

I have a number of Win 2012 guests, like Exchange and SQL, both clustered.

"swapping" to VMXNET 3 is not an easy task...

I also noticed that the drivers are not proeprly setup as all the descriptions are unknown. This is seen on ALL my Windows 2012 R2 servers...

The description for Event ID 27 from source e1iexpress cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If I could at least solve this would at least be a little step forward.


Thnx

Reply
0 Kudos
MatteoMarchetti
Contributor
Contributor

Same problem here, Windows 2012 VM disconnects from the network.

Vmware support said "I have checked and I found that the VM named "XXX" is using E1000E network adapter. There is an open issue with this driver where it is breaking the connectivity occasionally for the VMs in unpredicted intervals. No actual solution yet, but as a workaround can you please re-configure the VM to use E1000 or VMXNET3 vNIC adapter within this VM?" without quoting any KB about this known bug.


Does anyone have some more info?

Reply
0 Kudos
DATECProGmbH
Contributor
Contributor

Same problem here. Happens under 2012 R2 VMs with E1000 Network cards. Problem occurs also under 5.5  ESX Servers.

Switching to VMXNET3 is not an acceptable option as all VMs need E1000 cards in order to properly function with our Server Deployment Tool. For the moment this issue happened 2 or 3 times in 2 months. Although this isn't much, having a bug in our environment which could impact on critical systems is very bad.


Can't understand why this thread is open for now 2 years without any solution or even answer from Vmware when so much people have this problem...

Reply
0 Kudos
shaikhdaniaal
Contributor
Contributor

Some of the VMs were having issues on this and I have tried to apply your solution Leyuka, will monitor for about a week to see how stable it is (on E1000E the disconnections occur 2-3 times per day on a VM).

Update: The VMs with the VMXNET3 NICs tend to hang and will not be powered on until changed back to E1000E. Not a fix. Message was edited by: shaikhdaniaal

Reply
0 Kudos
Delor
Contributor
Contributor

Hello,

to keep this thread up I´m also raising my that my Windows 2012 R2 machines have the same Problem (only 2012 R2, the 2008 R2 does not have the issue!)

EventI27 e1iexpress with the Network Adapter E1000e.

The VMware version is 6.0 an the issue is still there.

I tried to switch the network adapter from a E1000e to a VMXNET3 in an windows 8 machine, the errors are now gone but now I got an other problem.

Now the machine does not lose the connection for a second but forever. When i connect via remote console the network of the machine works again and after about one hour of doing nothing the network of the machine is gone.

So switching to VMXNET3 is not a real option. Anyone got new info on how to solve this problem?

Greetings

B. Abraham

Reply
0 Kudos
Shottah
Contributor
Contributor

I am seeing the same e1iexpress warnings in the system logs of my Win 2012 R2 x64 VMs running with e1000e Nics as others have stated.

VMware support refeerenced me to this article -  Win 2012 virtual machines using E1000/E1000e driver experience loss of network connectivity (2109922)

They advised that I change the nic type to the VMXNET3 nic as it is the "recommended" nic type for this OS.  Can anyone confirm that the connectivity issues have in fact been resolved after change the nic type to VMXNET3?  I really do nto want to create more problems in doign this as some people have posted.

Reply
0 Kudos
Shottah
Contributor
Contributor

Has anyone else come across this issue?  Can we assume that the VMXNET3 adaptor type resolves this issue with E1000e NICs running windows 2012?

Reply
0 Kudos
honeybadgerIT
Contributor
Contributor

Running ESXi 6 and 2012 R2 VMs.  With VMXNET3 adapters VMs will drop connectivity completely and will not come back.  Changing network labels around so it switches VLANs and then switching back to the correct label will sometimes bring the VM back and start pinging.

It actually happens on CentOS VMs as well.

Ridiculous, come on VMware.

Reply
0 Kudos
spell08
Contributor
Contributor

Hi,

Receiving similar error in Server 2012 and 2012r2. we've found that e1000e works more efficiently than e1000 for server 2012, but still causes disconnect issues with server2012r2

For compatibility reasons, we cannot use the vmxnet adapter for compatibility reasons. We found the following workaround caused Event ID 27 to subside:

In Command Prompt

       netsh int tcp set global chimney=disabled

       netsh int tcp set global rss=disabled

       netsh int tcp set global netdma=disabled


From network-adapter properties > Advanced settings, Disable following TCP-offloading options:

       -IPv4 Checksum Offload

       -Large Send Offload V2 (IPv4)

       -Large Send Offload V2 (IPv6)

       -TCP Checksum Offload (IPv4)

       -TCP Checksum Offload (IPv6)


Over months, this has yet to cause any problems with our applications and has ceased the errors. This still needs a permanent remediation from VMWare though.



Reply
0 Kudos