Hi guys,
In esxi 6.7.0 9484548
we have below logs in vmkernel and vmkwarning. and we got strange datastore latency spike (over 10P sec) on some VMs.
is there any relationship between these latency and below logs?
in vmkernel.log
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:27.559Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:27.559Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:27.559Z cpu1:2098144)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:27.559Z cpu1:2098144)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:27.559Z cpu33:2123569)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:27.559Z cpu33:2123569)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:52.914Z cpu17:2103237)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:52.914Z cpu17:2103237)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:52.914Z cpu49:2104327)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:52.914Z cpu49:2104327)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:52.914Z cpu1:2100996)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:52.914Z cpu1:2100996)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:52.914Z cpu33:2102936)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:52.914Z cpu33:2102936)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:57.560Z cpu17:2123573)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:57.560Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:57.560Z cpu17:2123573)qfle3f:vmhba67:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:57.560Z cpu49:2097308)qfle3f:vmhba71:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:57.560Z cpu1:2098145)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:57.560Z cpu1:2098145)qfle3f:vmhba65:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
sizeof(struct fcoe_header) = 14, frame_len =32
2018-10-08T04:32:57.560Z cpu33:2124489)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:704: ELS 0xf
2018-10-08T04:32:57.560Z cpu33:2124489)qfle3f:vmhba69:qfle3f_processL2FrameCompletion:739: sizeof(vmk_EthHdr) = 14, sizeof(vmk_VLANHdr) = 4,
and in vmkwarning.log
2018-10-06T21:47:06.121Z cpu6:2098395)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000026" state in doubt; requested fast path state update...
2018-10-07T06:16:30.708Z cpu18:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d31000831500000000000000001e" state in doubt; requested fast path state update...
2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic0] NetQ could not add RX filter, no filters for queue 0
2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic2] NetQ could not add RX filter, no filters for queue 0
2018-10-08T01:51:37.407Z cpu63:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic3] NetQ could not add RX filter, no filters for queue 0
2018-10-08T01:52:02.410Z cpu59:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T01:56:12.413Z cpu73:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic0] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic2] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:01:37.407Z cpu60:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic3] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:09:22.410Z cpu74:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:10:57.410Z cpu64:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:14:57.410Z cpu49:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:24:27.410Z cpu56:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:25:27.410Z cpu58:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:26:57.346Z cpu17:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000022" state in doubt; requested fast path state update...
2018-10-08T02:26:57.346Z cpu14:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...
2018-10-08T02:35:57.412Z cpu67:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:39:33.363Z cpu17:2098397)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...
2018-10-08T02:40:23.364Z cpu19:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...
2018-10-08T02:40:27.410Z cpu55:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:41:36.366Z cpu0:2098401)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000008" state in doubt; requested fast path state update...
2018-10-08T02:44:12.410Z cpu50:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:47:02.410Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:52:12.409Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:54:52.410Z cpu65:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:55:02.409Z cpu66:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T02:59:17.411Z cpu44:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:13:42.409Z cpu82:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:15:12.411Z cpu62:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:21:12.410Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:23:27.411Z cpu66:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:26:37.409Z cpu77:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:33:02.409Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:42:37.412Z cpu47:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:43:22.410Z cpu49:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:43:57.410Z cpu78:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:45:27.409Z cpu64:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:47:02.409Z cpu74:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:51:02.409Z cpu55:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:51:40.138Z cpu1:2098395)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310008315000000000000000011" state in doubt; requested fast path state update...
2018-10-08T03:52:52.409Z cpu67:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:55:12.409Z cpu76:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:57:52.409Z cpu86:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T03:59:22.409Z cpu44:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T04:07:12.409Z cpu46:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
2018-10-08T04:12:47.409Z cpu80:2097460)WARNING: qfle3: qfle3_apply_queue_mac_filter:1985: [vmnic1] NetQ could not add RX filter, no filters for queue 0
Hi,
Could share logs from this commands:
vmkchdev -l | grep -i vmhbaX
vmkchdev -l | grep -i vmnicX
hi,
output below for vmnicX, and we use FCoE
0000:01:00.0 14e4:16a1 1028:1f7a vmkernel vmnic0
0000:01:00.1 14e4:16a1 1028:1f7a vmkernel vmnic1
0000:01:00.2 14e4:16a1 1028:1f7a vmkernel vmnic2
0000:01:00.3 14e4:16a1 1028:1f7a vmkernel vmnic3
Given the CNA information, you may want to confirm if you are on the correct driver/firmware combination: VMware Compatibility Guide - I/O Device Search
Hi,
Please take a look at VMware Knowledge Base 58874 We were affected by PSOD as described there and I also noticed those logs. After applying the workaround mentioned in the KB, those logs dissapeared.
We are on 6.7 U1 and our driver/fw levels were as recommended in the HCL.
I hope that helps.