rjposar
Contributor
Contributor

Upgrade to VMware ESXi, 7.0.3, 21424296 breaks teaming/changes virtual switch settings

About 10 days ago, we used Update Manager to update 25 of our 34 ESXi hosts to VMware ESXi, 7.0.3, 21424296. After the update to this version, most of the ESXi hosts immediately had communication issues. We discovered the cause of this communcation issue was that something in the update process changed the "Teaming and Failover" options of our standard vitual switches on these updates ESXi hosts. After manually changing the settings back, we were able bring most of the virtual machines back on-line. We have one cluster that still appears to have issues with HA and DRS. Getting assistance from VMware support has been rather disappointing. We are afraid to update the remaining 8 ESXi hosts in our environment, due to this issue and lack of VMware support. I have attempted to report this Update Manager bug to VMware.

I was curious if anyone else has experienced issues with update manager breaking their ESXi hosts to this version? If so, have you come up with a resolution to the issue? 

Reply
0 Kudos
albatros99
Enthusiast
Enthusiast

have install this update today without any problems!

what is the error messages you have seen?

Reply
0 Kudos
gambi_man
Contributor
Contributor

I have the issue with PowerEdge R740. Vmotion stopped working and HA stopped. 

Tags (1)
Reply
0 Kudos
jlorang
Enthusiast
Enthusiast

Update 21424296 changes the vSwitch to "Route based on originating Virtual port" I have updated several environments and every single update to VMware ESXi, 7.0.3, 21424296 the vSwitch resets the vSwitch from "Route based on IP hash" to "Route based on originating Virtual port"  Bug in the code?

 

 

 

88Kahi
Contributor
Contributor

Same here! Lost connection after this upgrade. Spent couple of hours figuring this out.

In my case vSwitch remained on "Route based on IP hash" but all of the virtual ports in the port group changed to "Route based on originating port ID". 

Reply
0 Kudos
PatrickDLong
Enthusiast
Enthusiast

This issue is fixed in ESXi 7.0 Update 3m build-21686933 released TODAY 2023-05-03
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3m-release-notes.html
PR 3164897: After an upgrade to ESXi 7.0 Update 3l, some ESXi hosts and virtual machines connected to virtual switches might lose network
After an upgrade to ESXi 7.0 Update 3l, some ESXi hosts, their VMs, and other VMkernel ports, such as ports used by vSAN and vSphere Replication, which are connected to virtual switches, might lose connectivity due to an unexpected change in the NIC teaming policy. For example, the teaming policy on a portgroup might change to Route Based on Originating Virtual Port from Route Based on IP Hash. As a result, such a portgroup might lose network connectivity and some ESXi hosts and their VMs become inaccessible.

If you are already experiencing this problem, I do not believe that applying new release 7.0 U3m will revert any now-incorrect teaming policy back to what it was prior to your application of 7.0 U3l - you will likely need to do that manually. But honestly I have not yet applied U3m so I cannot say for sure.

Reply
0 Kudos
albatros99
Enthusiast
Enthusiast

i have installed the depot file 703m yesterday.

After installation and reboot, there problems are the same!

Also problem with fusion 13.0.2 to access to the esxi7 server!

Reply
0 Kudos
rjposar
Contributor
Contributor

I updated one of our VDI clusters to 214242296. This did not appear to make the changes in the virtual switches, but I has complete broken DR and HA. 

Reply
0 Kudos
rjposar
Contributor
Contributor

Yes, I attempted to update from 21424296 to 21686933 and it also broke vmotion, DRS and HA.

Reply
0 Kudos
gambi_man
Contributor
Contributor

VMware ESXi, 7.0.3, 21686933 fixed this for me. 

Reply
0 Kudos
rjposar
Contributor
Contributor

have you tested DRS, HA & VMotion? 21686933 broke these for me.

Reply
0 Kudos
gambi_man
Contributor
Contributor

Yes DRS, HA and Vmotion is working. I have all distributed vswitches. I did notice an issues with my Vsan disk group for one host, but that can be unrelated.  On my Other cluster with Standard switches all is working as well. 

Did you try to reconfigure HA on each host? 

Reply
0 Kudos
rjposar
Contributor
Contributor

I have tried reconfiguring HA & DRS on my cluster.

Reply
0 Kudos
rjposar
Contributor
Contributor

We installed update "M" (21686933) and this without any changes being made to the ESX hosts. This appears to have broken HA/DRS. We have multiple ESX server clusters that host Horizon vm's that make up our VDI environment. 3 of the 4 ESX hosts can run at 90% cpu utilization and with the 4th ESX host no vm's and very low cpu utilization. The high utilization of the  3 ESX hosts running at 90% cpu utilization does NOT trigger migration to the 4 ESX host, which has no machines and 5% cpu utilzation. Also, when attempting to place upgraded ESX hosts into maintenance mode, they do not migrate vm's to other ESX hosts automatically. The migration has to be done manually to evacuate these vm's from the host going into migration. This should be an automated process.

Reply
0 Kudos
rjposar
Contributor
Contributor

As per vmware support, recommendation, we exported the logs from the 4 ESX hosts that were running update 21686933 (M?) and then downgraded them all to update 21424296 (L?). We updated VCenter from version 7.0.3.01300 to version 7.0.3.01400. We then disabled HA and DRS and reenabled and reconfigured each host for HA. HA/DRS/load balancing no longer works with this update and when placing a ESX host into maintenance mode, vm are NOT evacuated, unless done so manually. We exported post upgrade logs and uploaded both to VMware.

Reply
0 Kudos
TomoV
Contributor
Contributor

I have updated to 3m version. The only way to fully solve the problem with NIC teaming was:
- just left only one NIC in active state and move rest of NICs to standby state
- or remove all NICs but one from teaming group or

- or change teaming policy to "Use explicit failover order".

All these fixes really stop teaming.

Reply
0 Kudos