VMware Cloud Community
montespcs
Contributor
Contributor

All Network (NIC) interfaces failed at one time

Has anyone ever seen a total disconnect on all interfaces (all vswitches, vmnics, and eths) at the same time?

I have redundant connections for each interface (dual port NC380T NIC's) to redundant swtiches. Per our networking team, no glitches, outages, or events happened at this time. However, all our VM's, SC, and iSCSI (VM software connected) saw a blip and disconnected. HP Insight Manager didn't detect any failures for the NIC's. Is this a vSwitch issue or a greater internal ESX problem I am unaware of?

System is a HP Proliant DL580 G5.

Here's the log from /var/log/messages.2

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth0 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth1 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth2 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth3 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth4 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth5 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth6 failed

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: init_module: No such device

Jun 20 06:51:52 ESX-01 insmod: Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters. You ma

y find more information in syslog or the output from dmesg

Jun 20 06:51:52 ESX-01 insmod: /lib/modules/2.4.21-47.0.1.ELvmnix/kernel/drivers/net/bnx2/bnx2.o: insmod eth7 failed

Jun 20 07:01:02 ESX-01 syslogd 1.4.1: restart.

Jun 20 07:55:47 ESX-01 sshd[27840]: Connection closed by 10.10.10.54

Reply
0 Kudos
5 Replies
stinger2008
Enthusiast
Enthusiast

hi

I have the same problem and as a result backups on my hosts have failed to run. in the SC the message file is full of these error messages but in VC all the connections are present. Did you get an answer for your problem?

Sting

Reply
0 Kudos
Karunakar
Hot Shot
Hot Shot

hi,

all teh network ports cannot go down , but as they are using a single module "bnx2" (as per your log), could you please check if this module has been unloaded .

because if this is done, then all the network cards using this module will be down.

check for this module is loaded or not from "vmkload_mod -l", if the module is not loaded then try to reload using "vmkload_mod bnx2"

-Karunakar

Reply
0 Kudos
depping
Leadership
Leadership

probably the module gone bad, or that's the way it seems. Maybe the complete nic went off and on for a sec. This is why I try to always use 2 different vendors or models in a vSwitch!

Duncan

Blogging: http://www.yellow-bricks.com

If you find this information useful, please award points for "correct" or "helpful".

Reply
0 Kudos
Paul_Lalonde
Commander
Commander

/var/log/messages contain messages and warnings for the Service Console, not the VMkernel. Your network drivers aren't supposed to load in the Service Console, so the errors / warnings are probably normal. The VMkernel loads these drivers when the hypervisor is initiated.

The only time the Service Console loads the network drivers is when you select "Service Console Only (troubleshooting mode".

Hope this helps.

Paul

montespcs
Contributor
Contributor

We found that the G5's had a BIOS issue that would cause an NMI which in turn would cause an ASR. After applying this BIOS fix/upgrade, we haven't come across this issue since June. I believe it is now fixed for good using the 09/23/08 BIOS fix.

Thanks!

Reply
0 Kudos