Disappearing virtual network device under Windows 2008 R2 SP1 in vSphere 4.1

[vSlapp]

Has anyone seen an issue where after an upgrade from Windows 2008 R2 to Windows 2008 R2 SP1 the virtual network device suddenly and somewhat randomly ejects itself from the VM causing the virtual network device to not only disappear within the VM OS, but also from the virtual machine hardware configuration (essentially modifying the VMX config)?

I understand that the hotplug capability is enabled by default, but I can't seem to figure out why the device is apparently "auto-ejecting" itself. After talking with VMware support, we confirmed that the hotplug disconnect is definitely being initiated (from analyzing the logs through putty on the host where the VM is running).

There is no evidence that anyone is logging in and intentionally ejecting the VMXNET3 Ethernet Controller (either by safely removing the device or by removing it from the device manager), so our assumption at this point is that it is something within the OS that is causing it to occur. The Windows event logs do not show anything preceding the event, other than the errors related to a loss of network connectivity (once the NIC device disappears).

VMware did point out that this can occur in environments where VMware View is involved, but we are not using View in this environment. Additionally, we do not see this phenomenon on non-SP1 Windows 2008 R2 servers (at least not yet).

[wtfmatt]

Wow.  I haven't seen this in our environment, but we haven't gone to SP1 for R2 yet (and after reading this, we'll probably hold off a bit longer)

<conspiracy theory>

Sounds like another 'F-U' from microsoft.  It seems like they are progressively trying to make it more and more difficult to use vmware as they push their hyper-v agenda on everyone.

</conspiracy theory>

Were you at least able to put it back without issue? Or is that VM still without network capabilities?

[idle-jam]

could this help you? http://www.vmwareinfo.com/2009/11/vsphere-disable-device-hot-plug-on.html

[vSlapp]

The fix was easy. We just added a new Network Adapter to the VM config and the problem was solved.

The new adapter appears in Windows, the adapter rebinds to the original IP address settings, and everything is good to go.

This has happened twice now in the last two or three days... once right in front of the VMware support engineer (which served to confirm we weren't crazy).

[vSlapp]

Yes that is the next step. The VMware support engineer referred to a knowledge base article that outlines what you have provided here.

I guess we are looking for more confirmation as to the root cause. I have no doubt that disabling hotplug will address the issue, but I am more curious about the catalyst for why it is happening in the first place (without any apparent human intervention).

[sergeadam]

vSlapp wrote:

(without any apparent human intervention).

Now you've done it. The ACLU will be getting involved for sure. I know for a fact, after a lenghty conversation with one of them, that the gremlins living in computer rooms resent not beiong refered to as human.

[wtfmatt]

Just a shot in the dark, but do you think it's possible that this is related to the hidden_devices / network adapter issue people were experiencing when installing vmxnet3 vnics?

Where, if you didn't remove the old e1000 network adapter prior to installing a vmxnet3 adapter it would cache the IP stack on the old card?  I know that scenario normally resulted in ip conflicts on the device, but maybe there is something to it here where the SP1 install is trying to talk to an old unused (hidden) nic.  Were these vm's created fresh with vmxnet3 adapters? Or were they P2Vs?

[vSlapp]

The system was a clean fresh install of Windows 2008 R2. No phantom NICs in this case.

We have a maintenance scheduled for tonight. At the very minimum we hope to sidestep the issue while we continue researching what is causing it.

[wtfmatt]

Good to know.  Keep us posted, for sure.

[Rubeck]

As the vNIC is being removed from the actual VM config (.vmx), I'd might suspect that the VMtools running in that VM is playing tricks on ya... I mean what else has a comm. access to the "layer below" like that? Does the issue also occur if you stop the VMtools service from running?

/Rubeck

[elcascador44]

hi

for the record we had this probleme too in my companie, one time with 3 VM in 2k8 R2 and one time with a W2K3 R2,each time vmware support says it was a probleme with the OS even if there is no evidence. for the moment we have a running ticket with the support i keep you post .

P.S: sorry for my bad english

[vSlapp]

Disabling hot plug appears to have fixed the issue... although the cause is still unknown.

[elcascador44]

the thing is, I like to have an explanation when i have a problem, and none of those that vmawre give me is convincing.

and above all it's the random aspect of the bug that make me nervous.

in two weeks we lost 8 NIC