esxi 6.0 U3 network issue

[Anthony76LGQ]

Hi,

I opened this ticket ESXI6.0U3 network issue  2 weeks ago.

I updated the tg3 firmware and then I have no issue.

Today, on heavy network load the problem occur at new

the logs are fallowing

2017-11-27T08:28:53.434Z cpu7:36775)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-27T08:28:53.434Z cpu7:36775)NetSched: 677: 0x2000002: received a force quiesce for port 0x200001e, dropped 194 pkts

2017-11-27T08:28:53.434Z cpu7:36775)NetPort: 1780: disabled port 0x200001e

2017-11-27T08:28:53.453Z cpu7:36775)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-27T08:28:53.454Z cpu7:36775)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

2017-11-27T08:28:53.461Z cpu14:36775)NetPort: 1780: disabled port 0x200001e

2017-11-27T08:28:53.469Z cpu14:36775)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-27T08:28:53.469Z cpu14:36775)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

[....................]

2017-11-27T08:29:00.399Z cpu15:36914)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000021

2017-11-27T08:29:00.399Z cpu15:36914)NetPort: 1573: enabled port 0x2000021 with mac 00:0c:29:52:39:92

2017-11-27T08:29:01.706Z cpu6:35589)WARNING: NetPort: 1833: failed to disable port 0x2000008 on vSwitch0: Busy

2017-11-27T08:29:01.706Z cpu6:35589)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000008, dropped 261 pkts

2017-11-27T08:29:01.706Z cpu6:35589)NetPort: 1780: disabled port 0x2000008

2017-11-27T08:29:01.724Z cpu18:35589)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000008

2017-11-27T08:29:01.725Z cpu18:35589)NetPort: 1573: enabled port 0x2000008 with mac 00:0c:29:bd:2a:0d

2017-11-27T08:29:02.164Z cpu12:33341)WARNING: LinNet: netdev_watchdog:3680: NETDEV WATCHDOG: vmnic0: transmit timed out

2017-11-27T08:29:02.164Z cpu12:33341)<3>tg3 : vmnic0: transmit timed out, resetting

2017-11-27T08:29:03.415Z cpu12:33341)<3>tg3 : vmnic0: 0x00000000: 0x165f14e4, 0x00100006, 0x02000000, 0x00800000

2017-11-27T08:29:03.415Z cpu12:33341)<3>tg3 : vmnic0: 0x00000010: 0x91b3000c, 0x00000000, 0x91b4000c, 0x00000000

[.................;]

2017-11-27T08:29:03.416Z cpu12:33341)0x4390d1e9bfd0:[0x41801c616bfe]CpuSched_StartWorld@vmkernel#nover+0xa2 stack: 0x0, 0x0, 0x0, 0x0, 0

2017-11-27T08:29:03.661Z cpu24:33371)<3>tg3 : tg3_stop_block timed out, ofs=1400 enable_bit=2

2017-11-27T08:29:03.761Z cpu24:33371)<3>tg3 : tg3_stop_block timed out, ofs=c00 enable_bit=2

2017-11-27T08:29:03.841Z cpu33:33371)<6>tg3 : vmnic0: Link is down

2017-11-27T08:29:06.970Z cpu2:33410)<6>tg3 : vmnic0: Link is up at 1000 Mbps, full duplex

2017-11-27T08:29:06.970Z cpu2:33410)<6>tg3 : vmnic0: Flow control is off for TX and off for RX

2017-11-27T08:29:06.970Z cpu2:33410)<6>tg3 : vmnic0: EEE is disabled

2017-11-27T08:29:07.696Z cpu13:32886)<3>tg3 : vmnic0: [tg3_netq_remove_rx_filter:832]Filter not allocated on QID 1

2017-11-27T08:29:07.696Z cpu13:32886)<3>tg3 : vmnic0: [tg3_netq_remove_rx_filter:832]Filter not allocated on QID 2

2017-11-27T08:29:07.696Z cpu13:32886)<3>tg3 : vmnic0: [tg3_netq_free_rx_queue:723]Attempt to free a queue that is already free: 10001

2017-11-27T08:29:07.696Z cpu13:32886)<3>tg3 : vmnic0: [tg3_netq_free_rx_queue:723]Attempt to free a queue that is already free: 10002

2017-11-27T08:29:07.697Z cpu13:32886)NetPort: 1780: disabled port 0x2000002

2017-11-27T08:29:07.697Z cpu13:32886)<6>tg3 : vmnic0: [tg3_netq_get_queue_count:430]Using 4 RX NetQ rings

2017-11-27T08:29:07.697Z cpu13:32886)Uplink: 7468: enabled port 0x2000002 with mac 18:66:da:f0:72:e8

2017-11-27T08:29:08.048Z cpu0:35589)WARNING: NetPort: 1833: failed to disable port 0x2000008 on vSwitch0: Busy

2017-11-27T08:29:08.048Z cpu0:35589)NetPort: 1780: disabled port 0x2000008

2017-11-27T08:29:08.062Z cpu0:35589)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000008

2017-11-27T08:29:08.063Z cpu0:35589)NetPort: 1573: enabled port 0x2000008 with mac 00:0c:29:bd:2a:0d

2017-11-27T08:29:08.487Z cpu0:35537)WARNING: NetPort: 1833: failed to disable port 0x2000007 on vSwitch0: Busy

2017-11-27T08:29:08.487Z cpu0:35537)NetPort: 1780: disabled port 0x2000007

2017-11-27T08:29:08.508Z cpu14:36775)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-27T08:29:08.508Z cpu14:36775)NetPort: 1780: disabled port 0x200001e

2017-11-27T08:29:08.517Z cpu10:36775)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

[...........................................]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 0x00007030: 0x000e0000, 0x00004a50, 0x00170030, 0x00000000

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 0: Host status block [00000005:00000072:(0000:0240:0000):(0240:01cb)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 0: NAPI info [0000007f:0000007f:(01b3:01cb:01ff):0149:(0248:0000:0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 1: Host status block [00000001:00000006:(0000:0006:0000):(0006:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 1: NAPI info [00000006:00000006:(0000:0000:01ff):0006:(0105:0000:0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 2: Host status block [00000001:00000001:(0000:0000:0000):(0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 2: NAPI info [00000001:00000001:(0000:0000:01ff):0000:(002e:0000:0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 3: Host status block [00000001:00000001:(0000:0000:0000):(0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 3: NAPI info [00000001:00000001:(0000:0000:01ff):0000:(0745:0000:0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 4: Host status block [00000001:00000001:(0000:0000:0000):(0000:0000)]

2017-11-27T08:30:53.761Z cpu19:33347)<3>tg3 : vmnic0: 4: NAPI info [00000001:00000001:(0000:0000:01ff):0000:(000d:0000:0000:0000)]

2017-11-27T08:30:53.937Z cpu25:33356)<3>tg3 : tg3_stop_block timed out, ofs=1400 enable_bit=2

2017-11-27T08:30:54.037Z cpu25:33356)<3>tg3 : tg3_stop_block timed out, ofs=c00 enable_bit=2

2017-11-27T08:30:54.113Z cpu22:33356)<6>tg3 : vmnic0: Link is down

2017-11-27T08:30:55.032Z cpu13:35589)WARNING: NetPort: 1833: failed to disable port 0x2000008 on vSwitch0: Busy

2017-11-27T08:30:55.033Z cpu13:35589)NetPort: 1780: disabled port 0x2000008

2017-11-27T08:30:55.043Z cpu13:35589)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000008

2017-11-27T08:30:55.044Z cpu13:35589)NetPort: 1573: enabled port 0x2000008 with mac 00:0c:29:bd:2a:0d

2017-11-27T08:30:55.049Z cpu13:35589)NetPort: 1780: disabled port 0x2000008

2017-11-27T08:30:55.058Z cpu13:35589)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000008

2017-11-27T08:30:55.059Z cpu13:35589)NetPort: 1573: enabled port 0x2000008 with mac 00:0c:29:bd:2a:0d

2017-11-27T08:30:57.228Z cpu14:33410)<6>tg3 : vmnic0: Link is up at 1000 Mbps, full duplex

2017-11-27T08:30:57.228Z cpu14:33410)<6>tg3 : vmnic0: Flow control is off for TX and off for RX

2017-11-27T08:30:57.228Z cpu14:33410)<6>tg3 : vmnic0: EEE is disabled

2017-11-27T08:30:57.696Z cpu21:32886)<3>tg3 : vmnic0: [tg3_netq_remove_rx_filter:832]Filter not allocated on QID 1

2017-11-27T08:30:57.696Z cpu21:32886)<3>tg3 : vmnic0: [tg3_netq_free_rx_queue:723]Attempt to free a queue that is already free: 10001

2017-11-27T08:30:57.697Z cpu21:32886)NetPort: 1780: disabled port 0x2000002

2017-11-27T08:30:57.697Z cpu21:32886)<6>tg3 : vmnic0: [tg3_netq_get_queue_count:430]Using 4 RX NetQ rings

2017-11-27T08:30:57.697Z cpu21:32886)Uplink: 7468: enabled port 0x2000002 with mac 18:66:da:f0:72:e8

2017-11-27T08:30:58.697Z cpu22:32886)NetPort: 1780: disabled port 0x2000002

2017-11-27T08:30:58.697Z cpu22:32886)<6>tg3 : vmnic0: [tg3_netq_get_queue_count:430]Using 4 RX NetQ rings

2017-11-27T08:30:58.697Z cpu22:32886)Uplink: 7468: enabled port 0x2000002 with mac 18:66:da:f0:72:e8

2017-11-27T08:30:58.842Z cpu2:36775)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-27T08:30:58.842Z cpu2:36775)NetPort: 1780: disabled port 0x200001e

2017-11-27T08:30:58.851Z cpu2:36775)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-27T08:30:58.851Z cpu2:36775)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

Do you have any ideas about this issue?

Is it a problem with vmxnet3 or other?

Thanks in advance for your help.

Best regards

Anthony

[bluefirestorm]

Have you checked if NetQueue is enabled? From the looks of the log entries you pasted it might still be enabled. NetQueue feature is only useful for 10Gb NIC to gain throughput as close to 10Gbps as possible. As the BCM5720 is 1Gbps only, you don't lose any performance from disabling NetQueue.

2017-11-27T08:29:07.697Z cpu13:32886)<6>tg3 : vmnic0: [tg3_netq_get_queue_count:430]Using 4 RX NetQ rings

https://kb.vmware.com/kb/2035701

[Anthony76LGQ]

Ok, thanks for your answer.

I will disabled the queue and reboot my server this night.

Just a question, Why the queue occur this problem?

Thanks

Best regards

Anthony

[bluefirestorm]

I can only suspect that there is an implementation problem of the VMQ feature either at the tg3 driver level and/or the NIC firmware or both.

In case you are interested, you can see a 10 minute video explaining VMDq (Intel's equivalent to Broadcom); although I can't say the two are exactly the same but I would think the concept would be very similar. The video is from Intel.

https://youtu.be/QvKXbpV6WXk

[kashifkarar01]

net-tg3 patch would help and also validate for the latest drivers after the firmware:

VMware Knowledge Base

[Anthony76LGQ]

Ok, I will look the video.

Tanks

Anthony

[Anthony76LGQ]

I already update the tg3 firmware/

here the release:

driver: tg3

version: 3.137l.v60.1

firmware-version: FFV20.2.17 bc 5720-v1.39

bus-info: 0000:01:00.0

[bluefirestorm]

I forgot to add this to my earlier reply.

The same Broadcom 57xx family also have the lost connectivity problems in Hyper-V; and the same workaround, disable VMQ if the driver is not up-to-date; that's one reason I suspect it is driver and/or firmware implementation problem.

https://support.microsoft.com/en-us/help/2986895/virtual-machines-lose-network-connectivity-when-you...

[Anthony76LGQ]

I updated and reboot my servers last night.

I will give you a feeback asap.

Thanks for your help

[Anthony76LGQ]

Hi,

The problem is still here

2017-11-28T13:00:11.410Z cpu10:35647)WARNING: NetPort: 1833: failed to disable port 0x2000009 on vSwitch0: Busy

2017-11-28T13:00:11.410Z cpu10:35647)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000009, dropped 244 pkts

2017-11-28T13:00:11.410Z cpu0:32973)CpuSched: 596: user latency of 32973 helper24-0 0 changed by 32973 helper24-0 -6

2017-11-28T13:00:11.410Z cpu10:35647)NetPort: 1780: disabled port 0x2000009

2017-11-28T13:00:11.422Z cpu11:35647)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000009

2017-11-28T13:00:11.422Z cpu11:35647)NetPort: 1573: enabled port 0x2000009 with mac 00:0c:29:bd:2a:0d

2017-11-28T13:00:11.428Z cpu11:35647)NetPort: 1780: disabled port 0x2000009

2017-11-28T13:00:11.436Z cpu11:35647)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000009

2017-11-28T13:00:11.437Z cpu11:35647)NetPort: 1573: enabled port 0x2000009 with mac 00:0c:29:bd:2a:0d

2017-11-28T13:00:11.781Z cpu27:36149)WARNING: NetPort: 1833: failed to disable port 0x2000016 on vSwitch0: Busy

2017-11-28T13:00:11.781Z cpu27:36149)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000016, dropped 138 pkts

2017-11-28T13:00:11.782Z cpu27:36149)NetPort: 1780: disabled port 0x2000016

2017-11-28T13:00:11.791Z cpu38:36149)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000016

2017-11-28T13:00:11.791Z cpu38:36149)NetPort: 1573: enabled port 0x2000016 with mac 00:0c:29:e2:7a:70

2017-11-28T13:00:11.798Z cpu38:36149)NetPort: 1780: disabled port 0x2000016

2017-11-28T13:00:11.805Z cpu38:36149)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000016

2017-11-28T13:00:11.805Z cpu38:36149)NetPort: 1573: enabled port 0x2000016 with mac 00:0c:29:e2:7a:70

2017-11-28T13:00:13.512Z cpu30:36664)WARNING: NetPort: 1833: failed to disable port 0x2000021 on vSwitch0: Busy

2017-11-28T13:00:13.512Z cpu30:36664)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000021, dropped 285 pkts

2017-11-28T13:00:13.512Z cpu30:36664)NetPort: 1780: disabled port 0x2000021

2017-11-28T13:00:13.526Z cpu27:36664)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000021

2017-11-28T13:00:13.526Z cpu27:36664)NetPort: 1573: enabled port 0x2000021 with mac 00:0c:29:52:39:92

2017-11-28T13:00:13.531Z cpu27:36664)NetPort: 1780: disabled port 0x2000021

2017-11-28T13:00:13.539Z cpu37:36664)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000021

2017-11-28T13:00:13.539Z cpu37:36664)NetPort: 1573: enabled port 0x2000021 with mac 00:0c:29:52:39:92

2017-11-28T13:00:13.724Z cpu6:36566)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-28T13:00:13.724Z cpu6:36566)NetSched: 677: 0x2000002: received a force quiesce for port 0x200001e, dropped 204 pkts

2017-11-28T13:00:13.724Z cpu6:36566)NetPort: 1780: disabled port 0x200001e

2017-11-28T13:00:13.742Z cpu6:36566)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-28T13:00:13.742Z cpu6:36566)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

2017-11-28T13:00:17.092Z cpu6:36514)WARNING: NetPort: 1833: failed to disable port 0x200001d on vSwitch0: Busy

2017-11-28T13:00:17.092Z cpu6:36514)NetSched: 677: 0x2000002: received a force quiesce for port 0x200001d, dropped 59 pkts

2017-11-28T13:00:17.092Z cpu6:36514)NetPort: 1780: disabled port 0x200001d

2017-11-28T13:00:17.106Z cpu6:36514)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001d

2017-11-28T13:00:17.106Z cpu6:36514)NetPort: 1573: enabled port 0x200001d with mac 00:0c:29:cf:ca:cc

2017-11-28T13:00:17.112Z cpu6:36514)NetPort: 1780: disabled port 0x200001d

2017-11-28T13:00:17.120Z cpu6:36514)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001d

2017-11-28T13:00:17.121Z cpu6:36514)NetPort: 1573: enabled port 0x200001d with mac 00:0c:29:cf:ca:cc

2017-11-28T13:00:17.494Z cpu33:35545)WARNING: NetPort: 1833: failed to disable port 0x2000007 on vSwitch0: Busy

2017-11-28T13:00:17.494Z cpu33:35545)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000007, dropped 320 pkts

2017-11-28T13:00:17.494Z cpu33:35545)NetPort: 1780: disabled port 0x2000007

2017-11-28T13:00:17.541Z cpu33:35545)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000007

2017-11-28T13:00:17.541Z cpu33:35545)NetPort: 1573: enabled port 0x2000007 with mac 00:0c:29:78:c4:fe

2017-11-28T13:00:17.542Z cpu33:35545)NetPort: 1780: disabled port 0x2000007

2017-11-28T13:00:17.574Z cpu33:35545)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000007

2017-11-28T13:00:17.574Z cpu33:35545)NetPort: 1573: enabled port 0x2000007 with mac 00:0c:29:78:c4:fe

2017-11-28T13:00:18.862Z cpu4:36566)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-28T13:00:18.862Z cpu4:36566)NetSched: 677: 0x2000002: received a force quiesce for port 0x200001e, dropped 262 pkts

2017-11-28T13:00:18.863Z cpu4:36566)NetPort: 1780: disabled port 0x200001e

2017-11-28T13:00:18.875Z cpu4:36566)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-28T13:00:18.875Z cpu4:36566)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

2017-11-28T13:00:20.726Z cpu14:33362)WARNING: LinNet: netdev_watchdog:3680: NETDEV WATCHDOG: vmnic0: transmit timed out

2017-11-28T13:00:20.726Z cpu14:33362)<3>tg3 : vmnic0: transmit timed out, resetting

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000000: 0x165f14e4, 0x00100006, 0x02000000, 0x00800000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000010: 0x91b3000c, 0x00000000, 0x91b4000c, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000020: 0x91b5000c, 0x00000000, 0x00000000, 0x1f5b1028

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000030: 0xfffc0000, 0x00000048, 0x00000000, 0x0000010f

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000040: 0x00000000, 0x97000000, 0xc8035001, 0x64002008

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000070: 0x000710b2, 0xf4a02c9f, 0x00000000, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000080: 0x165f14e4, 0x00000012, 0x00000000, 0x00000bed

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000004ec

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x000000b0: 0x10008d81, 0x0010542e, 0x0004cc22, 0x10120040

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x000000d0: 0x0000001f, 0x00000006, 0x00000000, 0x00000001

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x78b0600a

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000100: 0x13c10001, 0x00000000, 0x00018000, 0x000e7030

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000110: 0x00002000, 0x000031c0, 0x000001e0, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000140: 0xdaf072e8, 0x00001866, 0x00000000, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000001

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.977Z cpu14:33362)<3>tg3 : vmnic0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000200: 0x00000000, 0x98000000, 0x00000000, 0x04000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000210: 0x00000000, 0x04000000, 0x00000000, 0x04000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000220: 0x00000000, 0x04000000, 0x00000001, 0x00000001

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000230: 0x00000001, 0x00000001, 0x00000001, 0x00000001

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000240: 0x00000001, 0x00000001, 0x00000001, 0x00000001

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000250: 0x00000001, 0x00000001, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000004ed

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000280: 0x00000000, 0x00000bee, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000300: 0x00000000, 0x000000e0, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000400: 0x18e04808, 0x00400000, 0x00001000, 0x00000880

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000410: 0x00001866, 0xdaf072e8, 0x00001866, 0xdaf072e8

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000420: 0x00001866, 0xdaf072e8, 0x00001866, 0xdaf072e8

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000430: 0x00000400, 0x00000000, 0x00000027, 0x000005f2

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x082a3800

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000460: 0x00000008, 0x00002620, 0x01000102, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000560: 0x00000001, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000005c0: 0x20b75c60, 0xf6e4080c, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000880: 0x00059919, 0x00000000, 0x00000000, 0x000005b5

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000890: 0x00000000, 0x00000011, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000900: 0x125fc597, 0xffffffff, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x0009a8c3

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000970: 0x00000000, 0x0000001f, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000980: 0x00044b61, 0x00000000, 0x00000000, 0x00069aec

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000990: 0x00000019, 0x000000c8, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000009d0: 0x00059919, 0x00000000, 0x000005b5, 0x000082aa

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x000009e0: 0x0001061d, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x00440000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000c80: 0x23b62c41, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000ce0: 0x0a6785f0, 0x00000000, 0x00000044, 0x000400ac

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00000cf0: 0x000003ec, 0x1000007a, 0x80070000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001000: 0x00000002, 0x00000000, 0xa0001c1e, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001010: 0x00f90f91, 0x00001c1e, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001440: 0x000000fb, 0x000000fb, 0x000000fb, 0x000000fb

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001450: 0x000000fb, 0x000000fb, 0x000000fb, 0x000000fb

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001460: 0x000000fb, 0x000000fb, 0x000000fb, 0x000000fb

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001470: 0x000000fb, 0x000000fb, 0x000000fb, 0x000000fb

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00001480: 0x00001114, 0x00000000, 0x00000000, 0x00000000

[................]]]]]]]]]]]]]])

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00006900: 0x1137e098, 0x006d3cd9, 0x00000000, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00007000: 0x00000008, 0x00000000, 0x00000000, 0x00004a4c

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00007010: 0xf8ba8764, 0x01c080f3, 0x00d70081, 0x03008200

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0x00007030: 0x000e0000, 0x00004a50, 0x00170030, 0x00000000

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0: Host status block [00000005:000000cc:(0000:0530:0000):(0d30:00f9)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 0: NAPI info [000000e0:000000e0:(00e0:00f9:01ff):0c37:(0536:0000:0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 1: Host status block [00000001:00000004:(0000:0000:0000):(0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 1: NAPI info [00000004:00000004:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 2: Host status block [00000001:00000004:(0000:0000:0000):(0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 2: NAPI info [00000004:00000004:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 3: Host status block [00000001:00000004:(0000:0000:0000):(0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 3: NAPI info [00000004:00000004:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 4: Host status block [00000001:00000004:(0000:0000:0000):(0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)<3>tg3 : vmnic0: 4: NAPI info [00000004:00000004:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-28T13:00:21.978Z cpu14:33362)WARNING: at vmkdrivers/src_92/vmklinux_92/vmware/linux_net.c:3709/netdev_watchdog() (inside vmklinux)

2017-11-28T13:00:21.978Z cpu14:33362)Backtrace for current CPU #14, worldID=33362, rbp=0x4304cd4b0a80

2017-11-28T13:00:21.978Z cpu14:33362)0x4390d291be10:[0x41802fc9816e]vmk_LogBacktraceMessage@vmkernel#nover+0x22 stack: 0xe, 0x41803033b2

2017-11-28T13:00:21.979Z cpu14:33362)0x4390d291be30:[0x41803033b2bb]watchdog_work_cb@com.vmware.driverAPI#9.2+0x27f stack: 0x4304cd4850e

2017-11-28T13:00:21.979Z cpu14:33362)0x4390d291bea0:[0x4180303617cf]vmklnx_workqueue_callout@com.vmware.driverAPI#9.2+0xd7 stack: 0x4304

2017-11-28T13:00:21.979Z cpu14:33362)0x4390d291bf30:[0x41802fc506a2]helpFunc@vmkernel#nover+0x4e6 stack: 0x0, 0x4304cd4850e0, 0x27, 0x0,

2017-11-28T13:00:21.979Z cpu14:33362)0x4390d291bfd0:[0x41802fe16bfe]CpuSched_StartWorld@vmkernel#nover+0xa2 stack: 0x0, 0x0, 0x0, 0x0, 0

2017-11-28T13:00:22.093Z cpu1:33364)<3>tg3 : tg3_stop_block timed out, ofs=1400 enable_bit=2

2017-11-28T13:00:22.194Z cpu1:33364)<3>tg3 : tg3_stop_block timed out, ofs=c00 enable_bit=2

2017-11-28T13:00:22.275Z cpu1:33364)<6>tg3 : vmnic0: Link is down

2017-11-28T13:00:25.168Z cpu4:36566)WARNING: NetPort: 1833: failed to disable port 0x200001e on vSwitch0: Busy

2017-11-28T13:00:25.169Z cpu4:36566)NetPort: 1780: disabled port 0x200001e

2017-11-28T13:00:25.179Z cpu3:36566)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-28T13:00:25.179Z cpu3:36566)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

2017-11-28T13:00:25.182Z cpu3:36566)NetPort: 1780: disabled port 0x200001e

2017-11-28T13:00:25.191Z cpu3:36566)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x200001e

2017-11-28T13:00:25.191Z cpu3:36566)NetPort: 1573: enabled port 0x200001e with mac 00:0c:29:90:cf:35

2017-11-28T13:00:25.403Z cpu20:33410)<6>tg3 : vmnic0: Link is up at 1000 Mbps, full duplex

2017-11-28T13:00:25.403Z cpu20:33410)<6>tg3 : vmnic0: Flow control is off for TX and off for RX

2017-11-28T13:00:25.403Z cpu20:33410)<6>tg3 : vmnic0: EEE is disabled

2017-11-28T13:00:25.732Z cpu24:32886)NetPort: 1780: disabled port 0x2000002

2017-11-28T13:00:25.733Z cpu24:32886)Uplink: 7468: enabled port 0x2000002 with mac 18:66:da:f0:72:e8

2017-11-28T13:00:27.088Z cpu29:35647)WARNING: NetPort: 1833: failed to disable port 0x2000009 on vSwitch0: Busy

2017-11-28T13:00:27.088Z cpu29:35647)NetPort: 1780: disabled port 0x2000009

2017-11-28T13:00:27.098Z cpu29:35647)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000009

2017-11-28T13:00:27.098Z cpu29:35647)NetPort: 1573: enabled port 0x2000009 with mac 00:0c:29:bd:2a:0d

Do you think that problem is from vmxnet3?

Best regards

[bluefirestorm]

You could try switching from vmxnet3 to e1000.

But it just looks strange the log shows a pattern of, with just differences in the port number, number of packet drops, MAC address

2017-11-28T13:00:11.410Z cpu10:35647)WARNING: NetPort: 1833: failed to disable port 0x2000009 on vSwitch0: Busy

2017-11-28T13:00:11.410Z cpu10:35647)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000009, dropped 244 pkts

2017-11-28T13:00:11.410Z cpu0:32973)CpuSched: 596: user latency of 32973 helper24-0 0 changed by 32973 helper24-0 -6

2017-11-28T13:00:11.410Z cpu10:35647)NetPort: 1780: disabled port 0x2000009

2017-11-28T13:00:11.422Z cpu11:35647)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000009

2017-11-28T13:00:11.422Z cpu11:35647)NetPort: 1573: enabled port 0x2000009 with mac 00:0c:29:bd:2a:0d

2017-11-28T13:00:11.428Z cpu11:35647)NetPort: 1780: disabled port 0x2000009

2017-11-28T13:00:11.436Z cpu11:35647)Vmxnet3: 15265: Using default queue delivery for vmxnet3 for port 0x2000009

2017-11-28T13:00:11.437Z cpu11:35647)NetPort: 1573: enabled port 0x2000009 with mac 00:0c:29:bd:2a:0d

and then followed by

2017-11-28T13:00:20.726Z cpu14:33362)WARNING: LinNet: netdev_watchdog:3680: NETDEV WATCHDOG: vmnic0: transmit timed out

2017-11-28T13:00:20.726Z cpu14:33362)<3>tg3 : vmnic0: transmit timed out, resetting

Sorry it is beyond me to understand whether that pattern has any meaning or not.

If the problem is in a configuration somewhere else, switching to e1000 may not help either. Is there any signficance to the MAC addresses in the log (e.g. does it belong to particular VM(s)?)

[Anthony76LGQ]

Thanks for your answer,

About VMs there is no particular configuration.

It's just Debian VMs with QOS active (tc filter) or web proxy.

This problem seems appear on heavy load network

[Anthony76LGQ]

Hi,

I updated last night all VMs from vmxnet3 to e1000.

To be continued

[Anthony76LGQ]

Hi,

the problem is still here

2017-11-29T14:09:10.496Z cpu16:88031)NetSched: 677: 0x2000002: received a force quiesce for port 0x200002e, dropped 2 pkts

2017-11-29T14:09:23.729Z cpu27:33369)WARNING: LinNet: netdev_watchdog:3680: NETDEV WATCHDOG: vmnic0: transmit timed out

2017-11-29T14:09:23.729Z cpu27:33369)<3>tg3 : vmnic0: transmit timed out, resetting

2017-11-29T14:09:24.981Z cpu27:33369)<3>tg3 : vmnic0: 0x00000000: 0x165f14e4, 0x00100006, 0x02000000, 0x00800000

2017-11-29T14:09:24.981Z cpu27:33369)<3>tg3 : vmnic0: 0x00000010: 0x91b3000c, 0x00000000, 0x91b4000c, 0x00000000

2017-11-29T14:09:24.981Z cpu27:33369)<3>tg3 : vmnic0: 0x00000020: 0x91b5000c, 0x00000000, 0x00000000, 0x1f5b1028

2017-11-29T14:09:24.981Z cpu27:33369)<3>tg3 : vmnic0: 0x00000030: 0xfffc0000, 0x00000048, 0x00000000, 0x0000010f

2017-11-29T14:09:24.981Z cpu27:33369)<3>tg3 : vmnic0: 0x00000040: 0x00000000, 0xa4000000, 0xc8035001, 0x64002008

[......]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 0: Host status block [00000001:000000ba:(0000:047f:0000):(0c7f:01b8)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 0: NAPI info [000000ba:000000ba:(019e:01b8:01ff):0c7f:(057e:0000:0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 1: Host status block [00000001:00000003:(0000:0000:0000):(0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 1: NAPI info [00000003:00000003:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 2: Host status block [00000001:00000003:(0000:0000:0000):(0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 2: NAPI info [00000003:00000003:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 3: Host status block [00000001:00000003:(0000:0000:0000):(0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 3: NAPI info [00000003:00000003:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 4: Host status block [00000001:00000003:(0000:0000:0000):(0000:0000)]

2017-11-29T14:10:27.318Z cpu24:33363)<3>tg3 : vmnic0: 4: NAPI info [00000003:00000003:(0000:0000:01ff):0000:(0000:0000:0000:0000)]

2017-11-29T14:10:27.420Z cpu21:33365)<3>tg3 : tg3_stop_block timed out, ofs=1400 enable_bit=2

2017-11-29T14:10:27.520Z cpu21:33365)<3>tg3 : tg3_stop_block timed out, ofs=c00 enable_bit=2

2017-11-29T14:10:27.601Z cpu24:33365)<6>tg3 : vmnic0: Link is down

2017-11-29T14:10:30.809Z cpu19:33410)<6>tg3 : vmnic0: Link is up at 1000 Mbps, full duplex

2017-11-29T14:10:30.809Z cpu19:33410)<6>tg3 : vmnic0: Flow control is off for TX and off for RX

2017-11-29T14:10:30.809Z cpu19:33410)<6>tg3 : vmnic0: EEE is disabled

2017-11-29T14:10:31.727Z cpu2:32886)NetPort: 1780: disabled port 0x2000002

2017-11-29T14:10:31.727Z cpu2:32886)Uplink: 7468: enabled port 0x2000002 with mac 18:66:da:f0:72:e8

2017-11-29T14:11:16.500Z cpu10:88031)NetSched: 677: 0x2000002: received a force quiesce for port 0x200002e, dropped 6 pkts

2017-11-29T14:11:21.606Z cpu20:86658)NetSched: 677: 0x2000002: received a force quiesce for port 0x2000028, dropped 6 pkts

2017-11-29T14:11:30.402Z cpu37:33359)WARNING: LinNet: netdev_watchdog:3680: NETDEV WATCHDOG: vmnic0: transmit timed out

2017-11-29T14:11:30.402Z cpu37:33359)<3>tg3 : vmnic0: transmit timed out, resetting

2017-11-29T14:11:31.654Z cpu37:33359)<3>tg3 : vmnic0: 0x00000000: 0x165f14e4, 0x00100006, 0x02000000, 0x00800000

If anyone has an idea....

Thanks

[bluefirestorm]

Does this vSwitch busy and/or transmit timeout happen only for vmnic0? What about vmnic1, 2, or 3?

Is this a single quadport card? Or four separate single port cards or two dual-port cards?

It looks like there is an ESXi 6.0 patch 6 https://kb.vmware.com/kb/2143832 but I can't seem to find release notes for it.

There is also ESXi 6.0 U3a (patch 5), there are a number of network related issues resolved based on the release notes, but nothing that describes/matches the logs that you have. The closest thing would be the memory leak fix for Etherswitch

https://docs.vmware.com/en/VMware-vSphere/6.0/rn/vsphere-esxi-60u3a-release-notes.html#networking-is...

[Anthony76LGQ]

Hi,

This problem occur on vmnic0 because this interface has a lot of network load.

Sometime I have also this error on other vmnic.

The network card is a single card (integrated card on motherboard)

In the same time I called Dell support that said me to upgrade the bios version . I did it the last night....

Note: the VMware ESXi 6.0 Update 3a Release Notes  show issue but it seems to be not my issue

[RPGRN]

Hi,

I am also getting same error. did you get any solution?

Pls help.