Automatic Tools upgrades on 2016 servers causing network cards to be removed?

[drheim]

We were running 6.5u2 and have typically set our Windows VMs to "Check and Upgrade VMWare Tools before each power on".  So almost 100% of our VMs had version 10287.  This has always worked fine as when the servers go to reboot for Windows Updates, they will just reboot again automatically for vmware tools updates during that same maintenance window.

We just upgraded to 6.7u2 and had about (10)2016 servers go offline last night when they were rebooting for windows updates(and attempting to upgrade tools to 10341).,  When I connected to the console on them they had rebooted for windows updates, were complaining that they still needed vmware tools installed, and were missing all of their NICs(vmxnet 3).  So I had to install VMWare tools, configure the nic correctly and reboot them to bring them back online.  We have not had any issues with 2008r2, 2012r2, or 2019 servers and those have all updated normally.  Has anyone else seen this problem? It almost feels like it is removing vmware tools during the upgrade and leaving the servers hanging in limbo until they reboot again, but I was more focused on getting the servers back online than looking for more details about the problem.

[pragg12]

Here's the exact log that was found by us which indicates that VMware tools installer stops at vmxnet3 driver step and installation fails. Currently waiting to hear more from VMware support.

nType=3393,Source=BinaryData,Target=VMUninstallVMXNet3Driver,)

MSI (s) (A8:F0) [05:40:31:972]: Executing op: ActionStart(Name=VM_InstallVMXNet3Drv.FC7BF8B8_27FB_4212_B385_AA45F07D667E,Description=Install

ing VMXNet3 driver,)

MSI (s) (A8:F0) [05:40:31:972]: Executing op: CustomActionSchedule(Action=VM_InstallVMXNet3Drv.FC7BF8B8_27FB_4212_B385_AA45F07D667E,ActionTy

pe=3073,Source=BinaryData,Target=VMInstallVMXNet3Driver,CustomActionData=C:\Program Files\Common Files\VMware\Drivers\vmxnet3\Win8\;2)

MSI (s) (A8:F0) [05:40:31:972]: Creating MSIHANDLE (392) of type 790536 for thread 5104

MSI (s) (A8:A4) [05:40:31:972]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIED7D.tmp, Entrypoint: VMInstallVMXNet3Driver

MSI (s) (A8:A4) [05:40:33:097]: Closing MSIHANDLE (392) of type 790536 for thread 5104

CustomAction VM_InstallVMXNet3Drv.FC7BF8B8_27FB_4212_B385_AA45F07D667E returned actual error code 1603 (note this may not be 100% accurate i

f translation happened inside sandbox)

[drheim]

VMWare is saying this is the problem

VMware Knowledge Base

[pragg12]

I don't think this is the one. I have already gone through the KB. In my case, ESXi is 6.5 U2 and VMware Tools is 10.2.x. The issue I am facing is with only Windows server 2016 and not with any other OS. Also, you can give reference of below article where the network issue with same Windows KB for Windows server 2016 is faced for Hyper-V as well. I believe the issue lies with how this Windows OS KB handles network component update which is in turn causing issues with network drivers. I know this has broken VMware Tools auto update functionality but at least VMware Support should acknowledge it and publish a public KB. Or demonstrate how this KB is not the reason of this breakdown since it has never happened to anyone of us before.

Microsoft Technet

[Devilray]

Did you ever got VMware to tell you if there was a fix for this ?. I need to turn on the automatic updates back .

[pragg12 · Accepted solution]

Hi,

I just heard back from VMware Support that they have finally published KB for this issue.

Windows 2016 Server loses its network card after a reboot with autoupdate of tools enabled (71386)

What I would suggest is:

1. Disable VMware Tools autoupgrade for Windows 2016 VMs in vCenter.

2. Let the Windows update finish all patch installation including patches for September 2019.

3. Enable the VMware Tools autoupgrade for Windows 2016 VMs again in vCenter.

4. Monitor and repeat 1 to 3 if you face same issue again.

[drheim]

Thanks Pragg.  - It took a lot to figure out the exact details of this problem, but your published KB is better than the published KB that vmware support gave me.