Solved: vmnic failover issue with HP blade server

nsial · ‎11-07-2017

Hello, I'm a newbie engineer.

I have an issue but It's hard to find solution

ESXi are installed on HP BL420C Gen8

Problem

- Suddenly VM1, VM2 had an issue (ping fail)

- They checked the virtual standard switch and There was an 'X' mark on the vmnic0

- NIC Teaming setting was 'Active-Standby'

- Although Failover setting was enabled, Failover didn't work. Because Both of VM still had a problem (Ping fail)

- After both of VM were migrated to other host, They worked well

- It was judged Hardware Issue (Switch)

(But I'm not sure whether it is physical switch or switch in the blade server)

Question

Question

- I think It might be hardware issue. (not vmware issue). But I don't know how to prove it

- I just want to know why both of VM couldn't ping even if there is one normal NIC that is alive.

- What should I check for this issue?

Finikiez · ‎11-07-2017

What do you use as SW1 and SW2?

What network adapters do you use?

Do you use standard switch or distributed switch on ESXi?

What is ESXi version?

The most common issue is that not all needed VLANs are configured on physical switch port.

Other common thing is driver\FW issue on network card in case if it's Emulex.

View solution in original post

Finikiez · ‎11-07-2017

What do you use as SW1 and SW2?

What network adapters do you use?

Do you use standard switch or distributed switch on ESXi?

What is ESXi version?

The most common issue is that not all needed VLANs are configured on physical switch port.

Other common thing is driver\FW issue on network card in case if it's Emulex.

Finikiez · ‎11-07-2017

Also I would recommend to check if this can be reproduced.

So move VMs to an affeced host and bring down network port on SW1. And check if the issue appeared or not.

nsial · ‎11-16-2017

I have opened the case and vmware replied

It was a NIC driver issue

He told me that this issue was happened becasue of driver crash.

It caused problem to all of NICs in the same hardware (so vMotion didn't work)

If I want to solve this problem, They said that Driver should be reloaded or Host should be rebooted.

thanks to give me an advice!

Have a good day

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_stats_update:1282(vmnic13)]storm stats were not updated for 3 times

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_stats_update:1283(vmnic13)]driver assert

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1139(vmnic13)]begin crash dump -----------------

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1149(vmnic13)]def_idx(0x1ea1) def_att_idx(0xa9b6) attn_state(0x0) spq_prod_idx(0xa4) next_stats_cnt(0x1e9e)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1154(vmnic13)]DSB: attn bits(0x0) ack(0x40) id(0x0) idx(0xa9b6)

<3>bnx2x: [bnx2x_panic_dump:1155(vmnic13)] def (0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x8c2d 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0) 2017-11-06T05:34:40.925Z cpu0:33470)igu_sb_id(0x44) igu_seg_id(0x1) pf_id(0x4) vnic_id(0x2) vf_id(0xff) vf_valid (0x0) state(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1206(vmnic13)]fp0: rx_bd_prod(0x3fd) rx_bd_cons(0x0) rx_comp_prod(0x40c) rx_comp_cons(0x1) *rx_cons_sb(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1209(vmnic13)] rx_sge_prod(0x0) last_max_sge(0x0) fp_hc_idx(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1226(vmnic13)]fp0: tx_pkt_prod(0x0) tx_pkt_cons(0x0) tx_bd_prod(0x0) tx_bd_cons(0x0) *tx_cons_sb(0x0)

<3>bnx2x: [bnx2x_panic_dump:1237(vmnic13)] run indexes (0x1 0x0)<3>bnx2x: [bnx2x_panic_dump:1243(vmnic13)] indexes (0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0)2017-11-06T05:34:40.925Z cpu0:33470)pf_id(0x4) vf_id(0xff) vf_valid(0x0) vnic_id(0x2) same_igu_sb_1b(0x1) state(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)SM[0] __flags (0x0) igu_sb_id (0x46) igu_seg_id(0x0) time_to_expire (0x193535) timer_value(0xff)

2017-11-06T05:34:40.925Z cpu0:33470)SM[1] __flags (0x0) igu_sb_id (0x46) igu_seg_id(0x0) time_to_expire (0xffffffff) timer_value(0xff)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[0] flags (0x0) timeout (0x0)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[1] flags (0x2) timeout (0x6)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[2] flags (0x0) timeout (0x0)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[3] flags (0x0) timeout (0x0)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[4] flags (0x1) timeout (0x0)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[5] flags (0x3) timeout (0xc)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[6] flags (0x3) timeout (0xc)

2017-11-06T05:34:40.925Z cpu0:33470)INDEX[7] flags (0x3) timeout (0xc)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1206(vmnic13)]fp1: rx_bd_prod(0x0) rx_bd_cons(0x0) rx_comp_prod(0x1) rx_comp_cons(0x1) *rx_cons_sb(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1209(vmnic13)] rx_sge_prod(0x0) last_max_sge(0x0) fp_hc_idx(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)<3>bnx2x: [bnx2x_panic_dump:1226(vmnic13)]fp1: tx_pkt_prod(0x0) tx_pkt_cons(0x0) tx_bd_prod(0x0) tx_bd_cons(0x0) *tx_cons_sb(0x0)

<3>bnx2x: [bnx2x_panic_dump:1237(vmnic13)] run indexes (0x1 0x0)<3>bnx2x: [bnx2x_panic_dump:1243(vmnic13)] indexes (0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0)2017-11-06T05:34:40.925Z cpu0:33470)pf_id(0x4) vf_id(0xff) vf_valid(0x0) vnic_id(0x2) same_igu_sb_1b(0x1) state(0x1)

2017-11-06T05:34:40.925Z cpu0:33470)SM[0] __flags (0x0) igu_sb_id (0x47) igu_seg_id(0x0) time_to_expire (0x198b20) timer_value(0xff)

2017-11-06T05:34:40.925Z cpu0:33470)SM[1] __flags (0x0) igu_sb_id (0x47) igu_seg_id(0x0) time_to_expire (0xffffffff) timer_value(0xff)