Received unrecoverable ECC Err, please reboot - Intel 82599 10G SFP+ Adapter

[sumandash]

We are using vmware 6.5 with Intel 82599ES Adapter and we are observing erratic behaviors on the LAN card . Logs are undermentioned. The same setup works fine with 1G Intel NIC but 10G intel NIC are giving us a hard time.

2017-04-27T11:20:35.306Z cpu6:66175)<6>ixgbe 0000:01:00.0: vmnic2: tx hang 109 detected on queue 0, resetting adapter

2017-04-27T11:20:35.306Z cpu6:66103)<3>ixgbe 0000:01:00.0: vmnic2: Reset adapter

2017-04-27T11:20:35.537Z cpu6:66103)<6>ixgbe 0000:01:00.0: vmnic2: detected SFP+: 5

2017-04-27T11:20:35.778Z cpu6:66103)<6>ixgbe 0000:01:00.0: vmnic2: NIC Link is Up 10 Gbps, Flow Control: None

2017-04-27T11:20:37.194Z cpu11:65645)NetPort: 1879: disabled port 0x3000002

2017-04-27T11:20:37.194Z cpu5:71676)NetSched: 628: vmnic2-0-tx: worldID = 71676 exits

2017-04-27T11:20:37.194Z cpu13:71678)NetSched: 628: vmnic2-2-tx: worldID = 71678 exits

2017-04-27T11:20:37.194Z cpu0:71677)NetSched: 628: vmnic2-1-tx: worldID = 71677 exits

2017-04-27T11:20:37.194Z cpu14:71679)NetSched: 628: vmnic2-3-tx: worldID = 71679 exits

2017-04-27T11:20:37.194Z cpu28:71683)NetSched: 628: vmnic2-7-tx: worldID = 71683 exits

2017-04-27T11:20:37.194Z cpu19:71681)NetSched: 628: vmnic2-5-tx: worldID = 71681 exits

2017-04-27T11:20:37.194Z cpu6:71680)NetSched: 628: vmnic2-4-tx: worldID = 71680 exits

2017-04-27T11:20:37.194Z cpu11:65645)NetLB: 2246: Driver claims supporting 7 RX queues, and 7 queues are accepted.

2017-04-27T11:20:37.194Z cpu11:65645)NetLB: 2242: Driver claims supporting 7 TX queues, and 7 queues are accepted.

2017-04-27T11:20:37.194Z cpu21:71682)NetSched: 628: vmnic2-6-tx: worldID = 71682 exits

2017-04-27T11:20:37.194Z cpu11:65645)Uplink: 9893: enabled port 0x3000002 with mac 00:1b:21:bc:7f:0c

2017-04-27T11:20:38.194Z cpu11:65645)NetPort: 1879: disabled port 0x3000002

2017-04-27T11:20:38.194Z cpu5:71684)NetSched: 628: vmnic2-0-tx: worldID = 71684 exits

2017-04-27T11:20:38.194Z cpu29:71686)NetSched: 628: vmnic2-2-tx: worldID = 71686 exits

2017-04-27T11:20:38.194Z cpu3:71685)NetSched: 628: vmnic2-1-tx: worldID = 71685 exits

2017-04-27T11:20:38.194Z cpu20:71687)NetSched: 628: vmnic2-3-tx: worldID = 71687 exits

2017-04-27T11:20:38.194Z cpu6:71689)NetSched: 628: vmnic2-5-tx: worldID = 71689 exits

2017-04-27T11:20:38.194Z cpu22:71691)NetSched: 628: vmnic2-7-tx: worldID = 71691 exits

2017-04-27T11:20:38.194Z cpu12:71690)NetSched: 628: vmnic2-6-tx: worldID = 71690 exits

2017-04-27T11:20:38.194Z cpu11:65645)NetLB: 2246: Driver claims supporting 7 RX queues, and 7 queues are accepted.

2017-04-27T11:20:38.194Z cpu11:65645)NetLB: 2242: Driver claims supporting 7 TX queues, and 7 queues are accepted.

2017-04-27T11:20:38.194Z cpu11:65645)Uplink: 9893: enabled port 0x3000002 with mac 00:1b:21:bc:7f:0c

2017-04-27T11:20:38.194Z cpu11:71688)NetSched: 628: vmnic2-4-tx: worldID = 71688 exits

2017-04-27T11:21:03.031Z cpu14:65578)<6>ixgbe 0000:01:00.0: vmnic2: Received unrecoverable ECC Err, please reboot

2017-04-27T11:21:07.912Z cpu7:66175)<3>ixgbe 0000:01:00.0: vmnic2: Detected Tx Unit Hang

  Tx Queue             <0>

  TDH, TDT             <c5>, <dc>

  next_to_use          <dc>

  next_to_clean        <c5>

2017-04-27T11:21:07.912Z cpu7:66175)<3>ixgbe 0000:01:00.0: vmnic2: tx_buffer_info[next_to_clean]

  time_stamp           <4cd3b>

  jiffies              <4cf23>

2017-04-27T11:21:07.912Z cpu7:66175)<6>ixgbe 0000:01:00.0: vmnic2: tx hang 110 detected on queue 0, resetting adapter

2017-04-27T11:21:07.912Z cpu10:66106)<3>ixgbe 0000:01:00.0: vmnic2: Reset adapter

2017-04-27T11:21:08.144Z cpu10:66106)<6>ixgbe 0000:01:00.0: vmnic2: detected SFP+: 5

2017-04-27T11:21:08.384Z cpu10:66106)<6>ixgbe 0000:01:00.0: vmnic2: NIC Link is Up 10 Gbps, Flow Control: None

2017-04-27T11:21:09.194Z cpu7:65645)NetPort: 1879: disabled port 0x3000002

2017-04-27T11:21:09.194Z cpu1:71692)NetSched: 628: vmnic2-0-tx: worldID = 71692 exits

2017-04-27T11:21:09.194Z cpu20:71693)NetSched: 628: vmnic2-1-tx: worldID = 71693 exits

2017-04-27T11:21:09.194Z cpu17:71694)NetSched: 628: vmnic2-2-tx: worldID = 71694 exits

2017-04-27T11:21:09.194Z cpu8:71695)NetSched: 628: vmnic2-3-tx: worldID = 71695 exits

2017-04-27T11:21:09.194Z cpu14:71696)NetSched: 628: vmnic2-4-tx: worldID = 71696 exits

2017-04-27T11:21:09.194Z cpu5:71697)NetSched: 628: vmnic2-5-tx: worldID = 71697 exits

2017-04-27T11:21:09.194Z cpu12:71699)NetSched: 628: vmnic2-7-tx: worldID = 71699 exits

2017-04-27T11:21:09.194Z cpu13:71698)NetSched: 628: vmnic2-6-tx: worldID = 71698 exits

2017-04-27T11:21:09.194Z cpu7:65645)NetLB: 2246: Driver claims supporting 7 RX queues, and 7 queues are accepted.

2017-04-27T11:21:09.194Z cpu7:65645)NetLB: 2242: Driver claims supporting 7 TX queues, and 7 queues are accepted.

2017-04-27T11:21:09.194Z cpu7:65645)Uplink: 9893: enabled port 0x3000002 with mac 00:1b:21:bc:7f:0c

2017-04-27T11:21:10.194Z cpu29:65645)NetPort: 1879: disabled port 0x3000002

2017-04-27T11:21:10.194Z cpu19:71701)NetSched: 628: vmnic2-0-tx: worldID = 71701 exits

2017-04-27T11:21:10.194Z cpu1:71702)NetSched: 628: vmnic2-1-tx: worldID = 71702 exits

2017-04-27T11:21:10.194Z cpu10:71703)NetSched: 628: vmnic2-2-tx: worldID = 71703 exits

2017-04-27T11:21:10.194Z cpu22:71704)NetSched: 628: vmnic2-3-tx: worldID = 71704 exits

2017-04-27T11:21:10.194Z cpu2:71705)NetSched: 628: vmnic2-4-tx: worldID = 71705 exits

2017-04-27T11:21:10.194Z cpu15:71706)NetSched: 628: vmnic2-5-tx: worldID = 71706 exits

2017-04-27T11:21:10.194Z cpu18:71707)NetSched: 628: vmnic2-6-tx: worldID = 71707 exits

2017-04-27T11:21:10.194Z cpu9:71708)NetSched: 628: vmnic2-7-tx: worldID = 71708 exits

2017-04-27T11:21:10.194Z cpu29:65645)NetLB: 2246: Driver claims supporting 7 RX queues, and 7 queues are accepted.

2017-04-27T11:21:10.194Z cpu29:65645)NetLB: 2242: Driver claims supporting 7 TX queues, and 7 queues are accepted.

2017-04-27T11:21:10.194Z cpu29:65645)Uplink: 9893: enabled port 0x3000002 with mac 00:1b:21:bc:7f:0c

2017-04-27T11:21:17.303Z cpu14:65578)<6>ixgbe 0000:01:00.0: vmnic2: Received unrecoverable ECC Err, please reboot

2017-04-27T11:21:22.501Z cpu5:66175)<3>ixgbe 0000:01:00.0: vmnic2: Detected Tx Unit Hang

Any help on this would be highly appreciated !!

[sumandash]

Nobody faced similar issues with 10G Intel Card ?