After VMXNET3 upgrade, some Win2003 guests lose connection until NIC is disabled & re-enabled in OS

[markyel]

Good morning,

We are in progress updating our Windows Server guests (2003 and 2008) from legacy VMXNET and E1000 NICs to VMXNET3. On a handful of only Windows 2003 guests, our enterprise monitoring reports server down intermittently. Upon closer investigation, the server is up and accessible from console in vCenter but it is not pingable, but the NIC adapter in the OS still shows Enabled. By disabling and re-enabling the adapter, the connection is restored.

I've checked the driver installed and in every case shows vmxnet3 1.0.0.11 from 1/2/2009. All systems are running v7 virtual hardware. The old NIC was removed by using "set devmgr_show_nonpresent_devices=1" and uninstalling from device manager.

Points we believe to be consistent:

- This happens on Windows Server 2003, not 2008.

- Affected servers are among different virtual switches and different ESX hosts, so no common thread there.

Has anyone seen behavior like this?

[Troy_Clavell]

I've seen this happen and it appears the TCP settings, in our case, the gateway goes missing. The workaround we have used is to just go to an E1000 NIC and it seems to resolve the problem.

Can you confirm if the gateway is present on some of the guests that you see the issue on?

[markyel]

Hmm, I haven't noticed a missing gateway so I will check for that. I agree that switching to E1000 eliminates the problem, but we are baselining on VMXNET3, and swapping to E1000 doesn't explain the reason for the problem.

[Ashley004]

We are experiencing very similar issues to this , the only thing is our testing involves Windows 2008 servers, I have done quite a bit of testing with this and have experienced missing gateways on some, disabling and enabling the adapter doesn’t always correct the problem .

[Troy_Clavell]

Just so you know VMware is aware of this issue and it will be addressed. If you haven't already, sign an NDA and get your VMware Sales team or TAM to talk about what is being done.

[Ashley004]

Hi thanks for the reply , when you say are aware of the issue , they are aware of the disconnecting , and it not reconnecting without going through several sequences to try and get it working again , not just the fact that on some VM’s the gateway disappears , I will try and discuss with sales and or log a call with VMware then

[AKWhite]

Any update on this issue? I am seeing similar behaviour on Windows Server 2008 x86 after changing from e1000 to vmxnet3 on ESXi build 261974 (4.0 update 02).

Also seeing Warning event 2012 from Srv in the guest System event log at the time of the outage, is this consistent with anyone else's experience?

[RLI27]

Guten Tag

Besten Dank für Ihr Mail.

Ich bin bis und mit 16. Juli 2010 abwesend. Ihr Mail wird aus Datenschutzgründen nicht weitergeleitet und bis zu meiner Rückkehr nicht bearbeitet.

Bitte wenden Sie sich bei Fragen an das ZID Servicedesk (servicedesk@bl.ch, 061 552 52 22). Besten Dank.

Mit freundlichen Grüssen

Roger Liniger

[Scissor]

ESXi build 261974 (4.0 update 02).

Now that ESXi 4.1 has been released, we enter the magic period of several weeks where everyone can suggest an upgrade to 4.1 to see if it solves the problem

[sven_jambor]

We're running on vSphere 4.1 - and had this issue today. So no, it's not resolved... BTW, with us it happend on a 2008 Rw server, so it's not 2003 specific.

[firestartah]

I have had this problem and once we upgraded to hardware version 7 and the laltest vmtools, remove the network card via the vm settings and re-add a new one. Once we did this the problems never came back.

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Gregg Robertson, VCP3,4 , MCSE, MCSA, MCTS, MCITP

[sven_jambor]

I had it happen the other way around, actually: The VM was on HW version 7 already and the VMWare tools were on the newest level. Only then did I edit the VM settings & Windows to use the VMXNET3 adapters. Everything went fine for one night - and then the VM lost its Gateway while keeping the rst of the ip settings. I put it back in today and hope and pray that it'll stay in there. So the newest VMWare Tools and HW version 7 are no guarantee.

I just posted the comment as heads-up for people (and, hopefully, VMWare Engineering) who thought that the issue would be gone in 4.1.... It's not...

[AndrewAbo]

Having a similar issue, which will likely rear its head again many times as my templates contain VMXNET3 adapters.

Enabling/disabling the adapter does not fix the problem.

Removing/re-adding a network adapter does not fix the problem.

Removing nonpresent hardware does not fix the problem.  (set devmgr_show_nonpresent_hardware=1, etc.)

I saw another post that stated the VMware support explained that other virtual adapters use a virtual PCI slot, whereas VMXNET3 uses a PCI-Express slot.  The entry for this adapter still exists in the registry, even after removing from nonpresent hardware.  The only problem is where, and/or how do you remove this entry?

Going to try to work with VMware support again today.  I'll update this if there is an actual fix.

[JeanFrancoisL]

I'm about to update all my existing servers (2008 and 2003) to virtual hw V7 and VMXNET3, so I'm interested in the feedback you may have fomr VMWare support.

Thanks

[firestartah]

As i stated above once i made those changes I've not had any problems since. So you should have no problems doing the upgrades as long as your hosts are on the latest patch level and the hosts have the latest vmtools installed