VMware Cloud Community
avinchakov
Enthusiast
Enthusiast
Jump to solution

esxi bnx2x problem!

Hello, today the esxi server failed in PSOD. In a dumping such events:

[7m2015-09-24T22:50:54.755Z cpu19:33255)WARNING: LinNet: netdev_watchdog:3678: NETDEV WATCHDOG: vmnic0: transmit timed out [0m

2015-09-24T22:50:54.755Z cpu19:33255)WARNING: at vmkdrivers/src_92/vmklinux_92/vmware/linux_net.c:3707/netdev_watchdog() (inside vmklinux)

2015-09-24T22:50:54.755Z cpu19:33255)Backtrace for current CPU #19, worldID=33255, rbp=0x43063118ecb0

2015-09-24T22:50:54.755Z cpu19:33255)0x4390cf39be10:[0x41800d696dfe]vmk_LogBacktraceMessage@vmkernel#nover+0x22 stack: 0x13, 0x41800dd1f

2015-09-24T22:50:54.755Z cpu19:33255)0x4390cf39be30:[0x41800dd1f7b7]watchdog_work_cb@com.vmware.driverAPI#9.2+0x27f stack: 0x43063117cce

2015-09-24T22:50:54.755Z cpu19:33255)0x4390cf39bea0:[0x41800dd45a5f]vmklnx_workqueue_callout@com.vmware.driverAPI#9.2+0xd7 stack: 0x4306

2015-09-24T22:50:54.755Z cpu19:33255)0x4390cf39bf30:[0x41800d64fa52]helpFunc@vmkernel#nover+0x4e6 stack: 0x0, 0x43063117cce0, 0x27, 0x0,

2015-09-24T22:50:54.755Z cpu19:33255)0x4390cf39bfd0:[0x41800d812aee]CpuSched_StartWorld@vmkernel#nover+0xa2 stack: 0x0, 0x0, 0x0, 0x0, 0

2015-09-24T22:51:16.840Z cpu10:33254)<3>[bnx2x_clean_tx_queue:1626(vmnic0)]timeout waiting for queue[0]: txdata->tx_pkt_prod(65183) != txdata->tx_pkt_cons(63170)

2015-09-24T22:51:38.901Z cpu10:33254)<3>[bnx2x_clean_tx_queue:1626(vmnic0)]timeout waiting for queue[0]: txdata->tx_pkt_prod(65183) != txdata->tx_pkt_cons(63170)

2015-09-24T22:51:51.244Z cpu16:33415)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x12 (0x43a5c8c7cd00, 0) to dev "naa.600508b1001c06cde91a898a5f4a6294" on path "vmhba0:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2015-09-24T22:53:29.623Z cpu0:33254)<3>[bnx2x_state_wait:319(vmnic0)]timeout waiting for state 7

2015-09-24T22:53:29.857Z cpu0:33254)IntrCookie: 1903: cookie 0x1c moduleID 4110 <vmnic0> exclusive, flags 0x25

2015-09-24T22:53:29.857Z cpu0:33254)IntrCookie: 1903: cookie 0x1e moduleID 4110 <vmnic0-fp-0> exclusive, flags 0x25

2015-09-24T22:53:29.857Z cpu0:33254)<6>bnx2x 0000:02:00.0: vmnic0: using MSI-X  IRQs: sp 16  fp[0] 26 ... fp[7] 33

2015-09-24T22:53:31.078Z cpu0:33254)IntrCookie: 1903: cookie 0x1f moduleID 4110 <vmnic0-fp-1> exclusive, flags 0x25

2015-09-24T22:53:31.122Z cpu9:33249)<3>bnx2x: [bnx2x_attn_int_deasserted3:4518(vmnic0)]MC assert!

2015-09-24T22:53:31.122Z cpu9:33249)<3>bnx2x: [bnx2x_mc_assert:894(vmnic0)]XSTORM_ASSERT_LIST_INDEX 0x2

2015-09-24T22:53:31.122Z cpu9:33249)<3>bnx2x: [bnx2x_mc_assert:908(vmnic0)]XSTORM_ASSERT_INDEX 0x0 = 0x00000000 0x00000000 0x00000000 0x00010026

2015-09-24T22:53:31.122Z cpu9:33249)<3>bnx2x: [bnx2x_mc_assert:922(vmnic0)]Chip Revision: everest2, FW Version: 7_8_52

2015-09-24T22:53:31.122Z cpu9:33249)<3>bnx2x: [bnx2x_attn_int_deasserted3:4524(vmnic0)]driver assert

2015-09-24T22:53:31.123Z cpu0:33254)<3>[bnx2x_esx_set_vlan_stripping:5947(vmnic0)]Failed to configure VLAN stripping for Queue 1

2015-09-24T22:53:31.123Z cpu0:33254)IntrCookie: 1903: cookie 0x20 moduleID 4110 <vmnic0-fp-2> exclusive, flags 0x25

2015-09-24T22:53:31.145Z cpu0:33254)<3>bnx2x: [bnx2x_setup_queue:9147(vmnic0)]Queue(2) SETUP failed

2015-09-24T22:53:31.145Z cpu0:33254)<3>[bnx2x_esx_setup_queue:599(vmnic0)]Queue 2 setup failed[0xfffffffb]

2015-09-24T22:53:31.145Z cpu0:33254)<3>[bnx2x_esx_init_netqs:932(vmnic0)]Could not start tx netq[-5]:2

2015-09-24T22:53:31.166Z cpu0:33254)<3>[bnx2x_queue_chk_transition:5310(vmnic0)]Blocking transition since pending was 20

2015-09-24T22:53:31.166Z cpu0:33254)<3>[bnx2x_queue_state_change:4499(vmnic0)]check transition returned an error. rc -16

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1097(vmnic0)]begin crash dump -----------------

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1103(vmnic0)]def_idx(0x2)  def_att_idx(0x3)  attn_state(0x101)  spq_prod_idx(0x6) next_stats_cnt(0x0)

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1108(vmnic0)]DSB: attn bits(0x0)  ack(0x101)  id(0x10)  idx(0x3)

<3>bnx2x: [bnx2x_panic_dump:1109(vmnic0)]     def (0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0)  2015-09-24T22:53:31.360Z cpu9:33249)igu_sb_id(0x10)  igu_seg_id(0x0) pf_id(0x0)  vnic_id(0x0)  vf_id(0xff)  vf_valid (0x0) state(0x1)

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1150(vmnic0)]fp0: rx_bd_prod(0x3fd)  rx_bd_cons(0x0)  rx_comp_prod(0x40c)  rx_comp_cons(0x1)  *rx_cons_sb(0x1)

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1153(vmnic0)]     rx_sge_prod(0x0)  last_max_sge(0x0)  fp_hc_idx(0x1)

2015-09-24T22:53:31.360Z cpu9:33249)<3>bnx2x: [bnx2x_panic_dump:1163(vmnic0)]fp0: tx_pkt_prod(0x152)  tx_pkt_cons(0x0)  tx_bd_prod(0x2a6)  tx_bd_cons(0x0)  *tx_cons_sb(0x0)

I understand the driver is broke. you couldn't help?

Tags (1)
1 Solution

Accepted Solutions
Techie01
Hot Shot
Hot Shot
Jump to solution

Is this esxi 6.0 or later?  If yes, this is a known issue . Please refer KB VMware KB: ESXi 6.0 network connectivity is lost with NETDEV WATCHDOG timeouts in the vmkernel.log 

There is a workaround script given in the KB .

View solution in original post

0 Kudos
4 Replies
Techie01
Hot Shot
Hot Shot
Jump to solution

Is this esxi 6.0 or later?  If yes, this is a known issue . Please refer KB VMware KB: ESXi 6.0 network connectivity is lost with NETDEV WATCHDOG timeouts in the vmkernel.log 

There is a workaround script given in the KB .

0 Kudos
avinchakov
Enthusiast
Enthusiast
Jump to solution

Thank you! I will try!

0 Kudos
SeanH2309
Contributor
Contributor
Jump to solution

Update released yesterday as 1a. http://kb.vmware.com/kb/2132153

avinchakov
Enthusiast
Enthusiast
Jump to solution

Yes, I saw it. Thank you

0 Kudos