"Network connectivity lost" alert - how to find which NIC is this ?

[bickyz]

Hi, We have HA cluster with VCenter 5.5 and 2 X ESXi servers.

Each host has 4 NIC.

One of the host has "Network connectivity lost" alert that was triggered 02:00am in the morning. Is there anyway to check which NIC is this ?

The Events tab dont says which NIC is that.

Any help would be much appreciated, thank you.

[a_p_]

Unless I'm mistaken, the ESXi host's vmkernel log should have entries regarding such issues.

André

[tedg_vCrumbs · Accepted solution]

You should see some sort of event in both the UI and the vmkernal.log and vobd.log.

vmkernel.log

~ # more /var/log/vmkernel.log | grep "link down"

2014-11-05T20:17:20.770Z cpu0:33263)netsched: NetSchedMClkNotify:2329: vmnic1: link down notification

~ # more /var/log/vmkernel.log | grep "link up"

2014-11-05T20:17:33.314Z cpu0:33265)netsched: NetSchedMClkNotify:2321: vmnic1: link up notification

2014-11-05T20:17:33.656Z cpu1:33411)netsched: NetSchedMClkWatchdogSysWorld:3786: vmnic1: link up event received, device running at 1000 Mbps so setting queue depth to 8646 bytes with expected 131 bytes/us

2014-11-05T20:17:33.735Z cpu1:32808)netsched: NetSchedMClkNotify:2321: vmnic1: link up notification

2014-11-05T20:17:33.736Z cpu0:36689)netsched: NetSchedMClkWatchdogSysWorld:3786: vmnic1: link up event received, device running at 1000 Mbps so setting queue depth to 8646 bytes with expected 131 bytes/us

~ #

~ # more /var/log/vmkernel.log | grep "NIC Link is Down"

2014-11-05T20:21:25.915Z cpu0:33267)<6>e1000: vmnic5: e1000_watchdog_task: NIC Link is Down

~ # more /var/log/vmkernel.log | grep "NIC Link is Up"

2014-11-05T20:21:41.660Z cpu1:33268)<6>e1000: vmnic5: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

vobd.log

Just did a test for ya.

~ # tail -f /var/log/vobd.log

2014-11-05T20:17:33.418Z: [netCorrelator] 620684191us: [vob.net.dvport.uplink.transition.up] Uplink: vmnic1 is up. Affected dvPort: 5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e. 1 uplinks up

2014-11-05T20:17:33.418Z: [netCorrelator] 620684194us: [vob.net.dvport.uplink.transition.up] Uplink: vmnic1 is up. Affected dvPort: 5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e. 1 uplinks up

2014-11-05T20:17:35.000Z: [netCorrelator] 622285373us: [esx.clear.net.dvport.connectivity.restored] Network connectivity restored on DVPorts: "5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e", "5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e". Physical NIC vmnic1 is up

2014-11-05T20:17:35.000Z: [netCorrelator] 622285495us: [esx.clear.net.dvport.redundancy.restored] Uplink redundancy restored on DVPorts: "5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e", "5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e", "5/a6 eb 07 50 28 7b c8 5a-a0 a6 99 e8 ad 8b 95 6e". Physical NIC vmnic1 is up

2014-11-05T20:21:26.094Z: [netCorrelator] 853350673us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down

2014-11-05T20:21:27.001Z: [netCorrelator] 854286718us: [esx.problem.net.vmnic.linkstate.down] Physical NIC vmnic5 linkstate is down

2014-11-05T20:21:42.149Z: [netCorrelator] 869405317us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up

2014-11-05T20:21:43.002Z: [netCorrelator] 870287321us: [esx.clear.net.vmnic.linkstate.up] Physical NIC vmnic5 linkstate is up

2014-11-05T20:24:38.723Z: [netCorrelator] 1045972218us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down

2014-11-05T20:24:39.000Z: [netCorrelator] 1046284854us: [esx.problem.net.vmnic.linkstate.down] Physical NIC vmnic5 linkstate is down

2014-11-05T20:24:42.230Z: [netCorrelator] 1049478491us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up

2014-11-05T20:24:43.001Z: [netCorrelator] 1050286523us: [esx.clear.net.vmnic.linkstate.up] Physical NIC vmnic5 linkstate is up

*Reminder to set up Syslog.global.logHost (software) advanced setting for log collection and review.

[bhards4]

Hi,

Use the command line

esxcli network nic list

[bickyz]

thank you everyone for the help.

[jvsraju234]

what is the solution for this issue