Network issues with HP 10GE card after upgrade from Esxi 5.0 to 5.5

[PMarks]

Dear All,

I upgraded my Esxi to 5.5 on a HP DL 380 G7. After the upgrade was in place, the 10GE (HP NC523 SFP by Qlogic) network card resets again and again. If I use the Broadcom onboard NIC's instead, everything works fine. The newest driver is installed and the card is flashed with the newest firmware but no success. As long as the Card is not assigned to a virtual switch it looks ok, connected with 10000 full. If assign it to a virtual switch, it takes a few seconds, then the connection state changes to down, then the card is not shown, then it returns and shows "connected with 10000 full" and so on. The vmkernel log says:

qlcnic 0000:0b:00.1: vmnic5:qlcnic_process_wait_event:743:Timelimit up.

qlcnic 0000:0b:00.1: vmnic5:qlcnic_dev_request_reset:7213:Changed the adapter dev_state to NEED_RESET.

qlcnic 0000:0b:00.1: vmnic5:qlcnic_check_health:7819:Adapter not in operational state(Heartbit Failure), resetting adapter.

This happens again and again until I remove it from the virtual switch, then it stays connected.

The card is used for network only, not for any storage (iscsi or nfs).

In Esxi 5.0 no problems at all.

Any advice is welcome.

Kind regards

Peer

[memaad]

Hello,

Can you replicate this issue and file support request, share SR number with me.

Regards

Mohammed Emaad

[memaad]

Hello,

Similar issue reported in this VMware Blog , check this out

Dropped Storage Paths Across all Hosts | VMware Support Insider - VMware Blogs

[PMarks]

Hello,

thanks for your reply, my support level is "subscription only", so I can't open an SR.

I have checked the link, but as stated above we are using the latest driver and image for the card and since it worked properly until we did the upgrade to 5.5, I can't believe that the card or the SFP is faulty. I also checked if the card is supported in Esxi 5.5 - yes it is.

Regards

Peer

[TimoW]

Hello Peer,

did you find any solution to this problem? At the moment I'm also fighting the same issue with a NC523SFP 10Gig-Card mounted in a HP DL 360 G6.

I did a BIOS update of the mainboard to latest available release (07/02/2013) and also upgraded to ESX 5.1u2 (got the same issue with ESX 5.0).

The NIC is running firmware version 4.8.22 and the driver version is 5.1.171.

Each 10-15 seconds the link at both NIC ports (and switch ports too) go down and up.

Kind regards,

Timo

PS: maybe some can figure out any helpful information from the log:

~ # tail -f /var/log/vmkernel.log | grep vmnic2

2014-06-27T12:52:40.615Z cpu7:8224)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_netq_get_supported_feat:1617:Netq features supported : LRO,

2014-06-27T12:52:45.086Z cpu7:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:52:49.557Z cpu7:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:52:52.346Z cpu5:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_health:7461:Adapter reset request received.

<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_peg_halt_status:6756:PEG_HALT_STATUS1: 0x0, PEG_HALT_STATUS2: 0x0.

2014-06-27T12:52:52.346Z cpu5:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_detach:2402:Deleted Tx and Rx loopback filters.

2014-06-27T12:52:52.346Z cpu5:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_disable_bus_master:1052:Disabled bus mastering.

2014-06-27T12:52:52.346Z cpu5:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_free_rx_irq:2072:Freed vmnic2_rx[0]_sds[0] irq.

2014-06-27T12:52:52.346Z cpu5:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_set_drv_state:6393:Gave acknowledgement for adapter reset.

2014-06-27T12:53:05.490Z cpu4:8640)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1596:Dual XGb SFP+ LP, Chip rev: 0x54

2014-06-27T12:53:05.490Z cpu4:8640)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1599:Firmware v4.8.22

2014-06-27T12:53:05.490Z cpu4:8640)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1604:[legacy]

2014-06-27T12:53:05.491Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_attach_work:7034:Re enabled bus mastering.

2014-06-27T12:53:05.620Z cpu5:8651)IRQ: 240: 0x81 <vmnic2_rx[0]_sds[0]> exclusive (entropy source), flags 0x10

2014-06-27T12:53:05.641Z cpu1:8193)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_advert_link_change:5617:NIC Link is up

2014-06-27T12:53:06.669Z cpu5:8224)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_netq_get_supported_feat:1617:Netq features supported : LRO,

2014-06-27T12:53:11.126Z cpu4:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:53:15.601Z cpu4:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:53:26.677Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_health:7461:Adapter reset request received.

<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_peg_halt_status:6756:PEG_HALT_STATUS1: 0x0, PEG_HALT_STATUS2: 0x0.

2014-06-27T12:53:26.677Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_detach:2402:Deleted Tx and Rx loopback filters.

2014-06-27T12:53:26.677Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_disable_bus_master:1052:Disabled bus mastering.

2014-06-27T12:53:26.677Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_free_rx_irq:2072:Freed vmnic2_rx[0]_sds[0] irq.

2014-06-27T12:53:26.677Z cpu5:8651)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_set_drv_state:6393:Gave acknowledgement for adapter reset.

2014-06-27T12:53:39.817Z cpu0:8652)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1596:Dual XGb SFP+ LP, Chip rev: 0x54

2014-06-27T12:53:39.817Z cpu0:8652)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1599:Firmware v4.8.22

2014-06-27T12:53:39.817Z cpu0:8652)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_options:1604:[legacy]

2014-06-27T12:53:39.818Z cpu1:8655)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_attach_work:7034:Re enabled bus mastering.

2014-06-27T12:53:40.052Z cpu1:8655)IRQ: 240: 0x81 <vmnic2_rx[0]_sds[0]> exclusive (entropy source), flags 0x10

2014-06-27T12:53:40.074Z cpu6:8205)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_advert_link_change:5617:NIC Link is up

2014-06-27T12:53:50.609Z cpu6:8224)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_netq_get_supported_feat:1617:Netq features supported : LRO,

2014-06-27T12:53:55.087Z cpu6:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:53:59.563Z cpu6:8224)<3>qlcnic 0000:04:00.0: vmnic2:qlcnic_process_wait_event:743:Timelimit up.

2014-06-27T12:54:02.111Z cpu5:8642)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_dev_request_reset:6913:Changed the adapter dev_state to NEED_RESET.

2014-06-27T12:54:02.111Z cpu5:8642)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_health:7519:Adapter not in operational state(Heartbit Failure), resetting adapter.

<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_check_peg_halt_status:6756:PEG_HALT_STATUS1: 0x0, PEG_HALT_STATUS2: 0x0.

2014-06-27T12:54:02.111Z cpu1:8645)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_detach:2402:Deleted Tx and Rx loopback filters.

2014-06-27T12:54:02.111Z cpu1:8645)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_disable_bus_master:1052:Disabled bus mastering.

2014-06-27T12:54:02.111Z cpu1:8645)<6>qlcnic 0000:04:00.0: vmnic2:qlcnic_free_rx_irq:2072:Freed vmnic2_rx[0]_sds[0] irq.

[grep_boy]

I too am having the exact same issue. I noticed on mine that after about 4 and a half hours later, things wake up and the host works fine. After discovering the host was working, I endeavored to update to the latest driver for that card to no avail - same exact issue.

I do have a support case open with VMware on this, I hope to have a solution soon.

If I get one, I'll share it with y'all.

[dhanarajramesh]

may I know the patch no of ESXi 5.5? if less than U2 to update to U2. because U2 updates fixed this issue VMware KB: VMware ESXi 5.5, Patch ESXi550-201409201-UG: Updates ESXi 5.5 esx-base vib

If you use Qlogic UCNA driver for your NIC, the status of the Uplink Port on an ESXi 5.5 host flaps and an error is written to vmkernel.log similar to the following:

vmnic3:qlcnic_issue_cmd:372:Failed card response err_code: 0xn
vmnic3:qlcnic_fw_cmd_destroy_tx_ctx:827:Failed to destroy tx ctx in firmware, err_code : 8
vmnic3:qlcnic_dev_request_reset:6879:Changed the adapter dev_state to NEED_RESET.
vmnic3:qlcnic_check_health:7485:Adapter not in operational state(Heartbit Failure), resetting adapter.
<6>qlcnic 0000:04:00.1:
vmnic3:qlcnic_check_peg_halt_status:6722:PEG_HALT_STATUS1: 0xnnnnnnnn, PEG_HALT_STATUS2: 0xnnnnnn.
vmnic3:qlcnic_detach:2337:Deleted Tx and Rx loopback filters.
vmnic3:qlcnic_disable_bus_master:1042:Disabled bus mastering.
vmnic3:qlcnic_free_rx_irq:2008:Freed vmnic3_rx[0] irq.
vmnic3:qlcnic_ctx_free_tx_irq:1859:Freed vmnic3_txq[0] irq #85.

KB 2079725.

[grep_boy]

Kernel version 2403361, along with all patches -> I downloaded all of the latest ones using Update Manager. (It that U2)?

[dhanarajramesh]

you have patched to U2 patch 4 which is the latest. FYI:in 2013,  this issue was there before also in 5.0 and Qlogic had released new version drivers for this issue. may be now they need to fix this bug and need to relase new driver for ESXi 5.5. please log a case as well with Qlogic support.

[grep_boy]

Well, as it turns out, seems like the final solution was the firmware on the card.

Mine in particular is a QLE3242, and downloading and installing the new firmware on the card, along with new drivers, seems to have fixed this issue.