network issue after upgrade to ESXi 7.0 Update 3l

[jvvmware]

After I upgraded to ESXi 7.0 Update 3l I ran into issues with guests not having network connectivity. All the settings were still correct and I also checked the switch it was connected to. I upgraded from ESXi 7.0 Update 3k using update in vCenter. I tried rebooting and that didn't help so I ended up rolling back and it's working again. Just curious if anyone had seen the same issue and if you did what the resolution was. Thanks in advance.

[KenWirz]

I experienced the exact same thing when I tried to upgrade today from 3l to 3k. I have opened a case with vmware.

[jvvmware]

Could you let me know what they find? Much appreciated.

[switchy]

We are facing similar issues, especially on hosts having a load balancing with multiple NICs.

It seems that the switches do not get properly notfied about MAC address changes to other hosts on our side.

[switchy]

Any news from VMware?

[KenWirz]

I worked with VMware support for a couple of hours. They confirmed that networking is working inside of ESXi (we could assign static IPs to two Windows Server VMs on the same host and same VLAN and they could ping each other). However the VMs could not ping the default gateway. Traffic leaves the host but doesn't come back.

Support tech reviewed the release notes and noted that the ntg3 driver is now version 4.1.9.0. My hardware vendor only officially supports up to version 4.1.8.0.

My short term solution was to revert my host to the previous version, VMware ESXi, 7.0.3, 21313628, and wait for the hardware vendor to supply an updated ntg3 driver.

I reverted the host and VMs are once again running on it.

[robert_williams]

Rolled back two of my hosts this morning. Might have to do the other 8, crap.

[bsquillace]

HI Ken and Robert,

Would you mind sharing what brand and model network cards you are using in the servers that had this network issue after the upgrade?

[KenWirz]

I am running HP ProLiant DL380 Gen9 servers with "Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet" physical adapters.

[switchy]

We are facing the same issues with HPE DL380 Gen10 servers with "Emulex Corporation HP CN1200E 2P Converged Network Adapter" network cards.

[JulienSoulet]

Experienced the same issue on Dell PowerEdge R640 using Intel(R) I350 Gigabit Network.

Rolled back successfully.

If I correctly understood we need to upgrade the firmware of the server before doing this upgrade (If Dell, HP, ... upgrade correctly that network firmware) ?

[zgtw]

We have the same issues on DL380's. We don't do vlan tagging so removed the etherchannel from our connected cisco switches and let the loadbalancing happen in Esxi.

For now this works.

[switchy]

HPE released a new customized ISO file yesterday with all the latest patches from VMware.

Unfortunately, I'm still facing network issues even with upgrading on of our hosts with this new ISO file.

[bsabol]

Please check this KB

https://kb.vmware.com/s/article/90837?lang=en_US

This problem will be solved in the next driver release.

[switchy]

Does not exactly describe the issue we are facing, but we'll see if it probably also solves other issues as soon as a new update is released.

[jvvmware]

I agree that doesn't seem to be the issue I'm seeing.

[nlies99]

I experienced the exact same thing when updating to U3l last week, also with DL380 G9's and the same Broadcom controller.  I tried opening up a ticket, but the support engineer told me to call the hardware vendor.  Clearly this is an issue with the 4.1.9.0 ntg3 driver.

[ksanderson]

This was my workaround to the problem too with the same NIC's.  Luckily, we don't really need the etherchannel on the effected hosts so not a huge issue for us thankfully.

[bsquillace]

If we are not using Link Aggregation for any ports that Vsphere uses on our physical switches, will we not have this network issue? What if we are using tagged VLANs on our physical switches?  We currently use VLANs setup on our physical switches but are only doing load balancing in our virtual switches. Will this configuration not have any network issues if we upgrade our ESXi hosts to 7.0 U3 L?

[ivanerben]

We have updated DL380 Gen10 and Dell R440 both with ntg3 and there was no problem.