2 separate host lose network connectivity daily

[Orion447]

Hey all,

Ive been experiencing this issue for some time and Im not sure what to do or where to look anymore. I have 2 hosts running Esxi 6.7 with 2 separate nics  both are 
cisco WS-C3560G-48PS

both speeds are set to auto negotiate here are the errors I get 

2021-06-10T14:01:22.461Z: [netCorrelator] 5220302435117us: [vob.net.pg.uplink.transition.down] Uplink: vmnic1 is down. Affected portgroup: Vlan400. 0 uplinks up. Failed criteria: 128
2021-06-10T14:01:22.461Z: [netCorrelator] 5220302435262us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
2021-06-10T14:01:24.002Z: [netCorrelator] 5220303976131us: [esx.problem.net.connectivity.lost] Lost network connectivity on virtual switch "Trunk-Nic2". Physical NIC vmnic1 is down. Affected port groups: "Admin", "Vlan410", "Vlan400", "Admin", "Vlan410", "Vlan400", "Admin", "Vlan410", "Vlan400", "Admin", "Vlan410", "Vlan400", "Admin", "Vlan410", "Vlan400", "Admin", "Vlan410", "Vlan400"
2021-06-10T14:01:45.047Z: [netCorrelator] 5220325022402us: [vob.net.vmnic.linkstate.up] vmnic vmnic1 linkstate up

Both host give out the same errors just with the corresponding nic names. My current remedy is to unplug and re-plug the physical ethernet cable for each nic. There seems to be a disconnect during high utilization of drives but I cant be sure.

system one

Manufacturer : Intel Corporation

Model : S5520UR

CPU : 12 CPUs x Intel(R) Xeon(R) CPU X5680 @ 3.33GHz

Memory : 63.94 GB

System two

Manufacturer : Cisco Systems Inc

Model : R200-1120402

CPU : 8 CPUs x Intel(R) Xeon(R) CPU E5540 @ 2.53GHz

Memory : 55.97 GB

both running  6.7.0 (Build 8169922)

When upgrading system 2  to 6.7 I had to allow legacy cpu in the boot startup. 

Any help or if anyone needs more logs please let me know thank you

[e_espinel]

Hello.
At first sight the two servers are different, you should know the model and brand of the Etherent card with problems.

From an SSH session, run the following commands to list the Nics of the ESxi host (on each server)

esxcli network nic list

Then run the following command according to the number (X) of the nic with problem, to know the Frimware and driver of the vmnic

esxcli network nic get -n vmnicX

Please include your results in the post.

The version 6.7.0 (Build 8169922) is very low and does not even reach Update 1 (Build 10302608). It should be at least on Update 3 (Build 14320388).

[Orion447]

System 1

Advertised Auto Negotiation: true
Advertised Link Modes: 10BaseT/Half, 10BaseT/Full, 100BaseT/Half, 100BaseT/Full, 1000BaseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:01:00.1
Driver: igb
Firmware Version: 2.1.0
Version: 5.0.5.1
Link Detected: true
Link Status: Up
Name: vmnic1
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: false
Pause TX: false
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Virtual Address: 00:50:56:55:6f:9e
Wakeon: MagicPacket(tm)

System 2

Advertised Auto Negotiation: true
Advertised Link Modes: 10BaseT/Half, 10BaseT/Full, 100BaseT/Half, 100BaseT/Full, 1000BaseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:0b:00.0
Driver: igb
Firmware Version: 1.4.3
Version: 5.0.5.1
Link Detected: true
Link Status: Up
Name: vmnic0
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: false
Pause TX: false
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Virtual Address: 00:50:56:5f:74:9e
Wakeon: MagicPacket(tm)

[e_espinel]

Hello.
As far as I can see both servers are using the same igb driver, which is normally used by Intel ethernet cards.
The Nics listing is missing, you need to confirm the manufacturer and the model of the problem card.
Also indicate the Build of version 6.7 that you are using.

[Orion447]

Intel® 82575EB Gigabit Ethernet Controller and build version 6.7.0 (Build 8169922)

[e_espinel · Accepted solution]

Hello.
The build you indicate is the base version 6.7, i.e. without any patches. Every operating system recommends to keep up to date with patches. You should update at least to Update 3 (Build 14320388). Attached is a link where you can get the patches.

https://my.vmware.com/group/vmware/patch#search

If you can get the Update 3 ISO image from the server manufacturer it would be better.

The Intel 82575EB Gigabit Ethernet card and its driver (5.0.5.1.1-5vmw.670.0.0.0.8169922) have had serious problems in both versions 6.5 and 6.7. Upgrading the driver to the latest version 5.2.5 has resulted in worse problems.
The options that have been tried by the users have been to change the driver for another one with different results. In short, nothing concrete or recommended by VMware or Intel.

The Intel 82575EB ethernet card is too old to expect any changes or support.

You can attach which drivers you currently have installed
with the following command
esxcli software vib list | grep gb

You could try to install the following driver and disable the driver with version 5.0.5, to see if it activates the Nics.
VMware ESXi 6.7 igbn 1.5.2.0 NIC Driver for Intel Ethernet Controllers 82580
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=DT-ESXI67-INTEL-IGBN-1520&product...

It is recommended to perform this kind of tests in an offline time window of the server and be clear on how to disable and enable the drivers. It may be necessary to have direct access to the server console.

[Orion447]

igbn 0.1.0.0-15vmw.670.0.0.8169922 VMW VMwareCertified 2021-01-18
ixgben 1.4.1-11vmw.670.0.0.8169922 VMW VMwareCertified 2021-01-18
net-igb 5.0.5.1.1-5vmw.670.0.0.8169922 VMW VMwareCertified 2021-01-18
net-ixgbe 3.7.13.7.14iov-20vmw.670.0.0.8169922 VMW VMwareCertified 2021-01-18

I will try the driver that you have linked.