VMware Cloud Community
Jailaubek
Contributor
Contributor

do we have to concern about these logs?

Hi guys,

In esxi 6.7.0 9484548

we have below logs in vmkernel and vmkwarning. and we got strange datastore latency spike (over 10P sec) on some VMs.

is there any relationship between these latency and below logs?

in vmkernel.log

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:27.559Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:27.559Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:27.559Z cpu1:2098144)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:27.559Z cpu1:2098144)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:27.559Z cpu33:2123569)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:27.559Z cpu33:2123569)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:52.914Z cpu17:2103237)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:52.914Z cpu17:2103237)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:52.914Z cpu49:2104327)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:52.914Z cpu49:2104327)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:52.914Z cpu1:2100996)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:52.914Z cpu1:2100996)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:52.914Z cpu33:2102936)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:52.914Z cpu33:2102936)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:57.560Z cpu17:2123573)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:57.560Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:57.560Z cpu17:2123573)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:57.560Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:57.560Z cpu1:2098145)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:57.560Z cpu1:2098145)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

sizeof(struct fcoe_header) = 14, frame_len =32

2018-10-08T04:32:57.560Z cpu33:2124489)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf

2018-10-08T04:32:57.560Z cpu33:2124489)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,

and in vmkwarning.log

2018-10-06T21:47:06.121Z cpu6:2098395)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000026" state in doubt; requested fast path state update...

2018-10-07T06:16:30.708Z cpu18:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d31000831500000000000000001e" state in doubt; requested fast path state update...

2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic0] NetQ could not add RX filter, no filters for queue 0

2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic2] NetQ could not add RX filter, no filters for queue 0

2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic3] NetQ could not add RX filter, no filters for queue 0

2018-10-08T01:52:02.410Z cpu59:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T01:56:12.413Z cpu73:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic0] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic2] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic3] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:09:22.410Z cpu74:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:10:57.410Z cpu64:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:14:57.410Z cpu49:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:24:27.410Z cpu56:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:25:27.410Z cpu58:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:26:57.346Z cpu17:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000022" state in doubt; requested fast path state update...

2018-10-08T02:26:57.346Z cpu14:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...

2018-10-08T02:35:57.412Z cpu67:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:39:33.363Z cpu17:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...

2018-10-08T02:40:23.364Z cpu19:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...

2018-10-08T02:40:27.410Z cpu55:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:41:36.366Z cpu0:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...

2018-10-08T02:44:12.410Z cpu50:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:47:02.410Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:52:12.409Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:54:52.410Z cpu65:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:55:02.409Z cpu66:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T02:59:17.411Z cpu44:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:13:42.409Z cpu82:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:15:12.411Z cpu62:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:21:12.410Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:23:27.411Z cpu66:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:26:37.409Z cpu77:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:33:02.409Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:42:37.412Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:43:22.410Z cpu49:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:43:57.410Z cpu78:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:45:27.409Z cpu64:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:47:02.409Z cpu74:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:51:02.409Z cpu55:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:51:40.138Z cpu1:2098395)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000011" state in doubt; requested fast path state update...

2018-10-08T03:52:52.409Z cpu67:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:55:12.409Z cpu76:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:57:52.409Z cpu86:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T03:59:22.409Z cpu44:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T04:07:12.409Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

2018-10-08T04:12:47.409Z cpu80:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0

Reply
0 Kudos
4 Replies
dbalcaraz
Expert
Expert

Hi,

Could share logs from this commands:

vmkchdev -l | grep -i vmhbaX

vmkchdev -l | grep -i vmnicX

-------------------------------------------------------- "I greet each challenge with expectation"
Reply
0 Kudos
Jailaubek
Contributor
Contributor

hi,

     output below for vmnicX, and we use FCoE

0000:01:00.0 14e4:16a1 1028:1f7a vmkernel vmnic0

0000:01:00.1 14e4:16a1 1028:1f7a vmkernel vmnic1

0000:01:00.2 14e4:16a1 1028:1f7a vmkernel vmnic2

0000:01:00.3 14e4:16a1 1028:1f7a vmkernel vmnic3

Reply
0 Kudos
TotesHagopes
VMware Employee
VMware Employee

Given the CNA information, you may want to confirm if you are on the correct driver/firmware combination: VMware Compatibility Guide - I/O Device Search

Reply
0 Kudos
ivgivanov
Contributor
Contributor

Hi,

Please take a look at VMware Knowledge Base 58874 We were affected by PSOD as described there and I also noticed those logs. After applying the workaround mentioned in the KB, those logs dissapeared.

We are on 6.7 U1 and our driver/fw levels were as recommended in the HCL.

I hope that helps.

Reply
0 Kudos