Ntg3HandleTxMappingErr

[kangtk]

Hi~

ESXi 6.5 running on Dell R620 server.

I've found logs in vmkernel.log file.

I think these logs had showed up when txhang problem has occurred.

What is the problem and how to solve this problem?

2020-03-16T19:58:38.525Z cpu17:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 114 bytes at 0x2ffffff8e

2020-03-16T19:58:43.922Z cpu12:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 150 bytes at 0x2ffffff6a

2020-03-16T19:59:09.013Z cpu20:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 60 bytes at 0x2ffffffc4

2020-03-16T19:59:09.447Z cpu3:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 146 bytes at 0x2ffffff6e

2020-03-16T19:59:58.130Z cpu18:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 260 bytes at 0x2fffffefc

2020-03-16T20:00:00.879Z cpu19:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 90 bytes at 0x2ffffffa6

2020-03-16T20:00:03.167Z cpu17:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 35 bytes at 0x2ffffffdd

2020-03-16T20:00:04.375Z cpu13:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 110 bytes at 0x2ffffff92

2020-03-16T20:00:21.334Z cpu21:66054)ScsiDeviceIO: 3001: Cmd(0x439d0043fc40) 0x1a, CmdSN 0x316ca from world 0 to dev "naa.6b8ca3a0ec40a400241879c94781ac79" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x

0.

2020-03-16T20:01:23.204Z cpu20:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 66 bytes at 0x2ffffffbe

2020-03-16T20:01:44.601Z cpu13:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 146 bytes at 0x2ffffff6e

2020-03-16T20:01:53.890Z cpu14:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 448 bytes at 0x2fffffe40

2020-03-16T20:02:08.071Z cpu13:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 86 bytes at 0x2ffffffaa

2020-03-16T20:02:17.164Z cpu23:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 455 bytes at 0x2fffffe39

2020-03-16T20:02:17.745Z cpu14:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 116 bytes at 0x2ffffff8c

2020-03-16T20:02:18.667Z cpu13:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 462 bytes at 0x2fffffe32

2020-03-16T20:02:19.363Z cpu3:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 203 bytes at 0x2ffffff35

2020-03-16T20:02:29.491Z cpu15:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 102 bytes at 0x2ffffff9a

2020-03-16T20:02:41.483Z cpu20:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 92 bytes at 0x2ffffffa4

2020-03-16T20:02:59.320Z cpu12:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 142 bytes at 0x2ffffff72

2020-03-16T20:03:24.256Z cpu14:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 420 bytes at 0x2fffffe5c

2020-03-16T20:03:30.174Z cpu17:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 150 bytes at 0x2ffffff6a

2020-03-16T20:03:30.644Z cpu12:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 204 bytes at 0x2ffffff34

2020-03-16T20:03:31.014Z cpu10:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 172 bytes at 0x2ffffff54

2020-03-16T20:03:34.803Z cpu21:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 79 bytes at 0x2ffffffb1

2020-03-16T20:03:48.982Z cpu20:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 238 bytes at 0x2ffffff12

2020-03-16T20:03:49.618Z cpu1:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 569 bytes at 0x2fffffdc7

2020-03-16T20:04:49.864Z cpu9:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 156 bytes at 0x2ffffff64

2020-03-16T20:05:30.006Z cpu8:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 137 bytes at 0x2ffffff77

2020-03-16T20:05:31.360Z cpu10:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 134 bytes at 0x2ffffff7a

2020-03-16T20:05:39.696Z cpu9:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 232 bytes at 0x2ffffff18

2020-03-16T20:05:58.315Z cpu11:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 550 bytes at 0x2fffffdd7

2020-03-16T20:05:59.082Z cpu0:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 164 bytes at 0x2ffffff5c

2020-03-16T20:05:59.532Z cpu16:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 162 bytes at 0x2ffffff5c

2020-03-16T20:05:59.846Z cpu2:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 80 bytes at 0x2ffffffb0

2020-03-16T20:06:11.837Z cpu1:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 132 bytes at 0x2ffffff7c

2020-03-16T20:06:13.363Z cpu5:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 132 bytes at 0x2ffffff7c

2020-03-16T20:06:14.501Z cpu11:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 443 bytes at 0x2fffffe45

2020-03-16T20:06:20.474Z cpu7:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 1093 bytes at 0x2fffffbbb

2020-03-16T20:06:48.048Z cpu1:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 204 bytes at 0x2ffffff34

2020-03-16T20:06:56.447Z cpu20:66171)DVFilter: 5924: Checking disconnected filters for timeouts

2020-03-16T20:07:09.970Z cpu6:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 92 bytes at 0x2ffffffa4

2020-03-16T20:07:20.600Z cpu4:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 162 bytes at 0x2ffffff56

2020-03-16T20:08:00.211Z cpu19:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 100 bytes at 0x2ffffff9c

2020-03-16T20:08:09.617Z cpu2:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 398 bytes at 0x2fffffe72

2020-03-16T20:08:12.100Z cpu16:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 70 bytes at 0x2ffffffba

2020-03-16T20:08:24.328Z cpu16:68090)ntg3:vmnic1:Ntg3HandleTxMappingErr:360:Avoid frag of 286 bytes at 0x2fffffee2

Any help would be appreciated.

Kang

[dariusd]

Which build/update of ESXi are you using?  It seems we've fixed a handful of defects in the ntg3 driver since the release of 6.5, so if you aren't running the latest update, I'd suggest updating.

--

Darius

[kangtk]

Thanks for answering me.

The server was updated 2 months ago.

Also server firmware(bios,nic ...) has been updated.

6.5.0 Update 3 (Build 15256549).

Build 15256549 is not fixed version?

Can you tell me what is fixed version?

esxcli network nic get -n vmnic1

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseT/Full, 100BaseT/Half, 100BaseT/Full, 10BaseT/Half, 10BaseT/Full

   Auto Negotiation: false

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:01:00:1

         Driver: ntg3

         Firmware Version: bc 1.39 ncsi 1.5.12.0

         Version: 4.1.3.2

lspci

0000:01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic0]

0000:01:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic1]

0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic2]

0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic3]

[dariusd]

Most of the fixes went in between 6.5 and 6.5 U1, and a few more between U1 and U3, so if you're on U3, you should be good.

Are you actually encountering TX hangs?  Or is it just that the log messages are worrying you?  The log messages you quoted there are entirely harmless and can be ignored.  If you are actually encountering TX hangs, there might be something further to investigate.

--

Darius

[kangtk]

After update, TX hang problem has occurred about two weeks ago.

After reboot the server, the problem has been solved.

There were same messages in log file.

So I'm worried about that messages.

Then what should I do for this problem?

Thanks.

Kang

[dariusd]

The "Avoid frag of nnn bytes at 0x2fffffnnn" messages do not indicate any actual problem or failure.  Do not worry about them unless they somehow definitely correlate with the TX hang.

Are there any other messages in the log nearer to the time of the TX hang?

--

Darius

[kangtk]

I thought "Avoid frag of nnn bytes at 0x2fffffnnn"  message is the sign before TX hang problem.

TX hang occurred time was UTC 2020-02-29 16:08.

The attached file is the log file that you requested.

Thanks for your kindness.

Kang