An abstract of my vmkernel.log
We had storage incidents (crashes for unresolved bugs, being a readcache merge problem and a backend driver issue) on Unity 650F.
Dell is writing on a 'custom' fix on both issues. In the mean time we were asked to mitigate the controller autoresets, and upgrade to OE 5.0.3 which they agreed upon will not resolve the current controller resets. After complaints from our side, they digged into every component of our infra, to mitigate on the impact/issues on their storage. The following SCSI sense codes were found in the vmkernel log and we are referred to further seek host support to suppress 'illegal scsi commands'
These according to them are to addressed as they are contributing problems to crashe of a controller node , as target reset attempts are being made by the hosts (as seen from the storage side persfective)
2020-08-17T01:49:15.677Z cpu108:65805)ScsiDeviceIO: 3015: Cmd(0x439e4341dfc0) 0xfe, CmdSN 0xbbc2e5 from world 65687 to dev "naa.60060160e8004b00e2ca985c1400127d" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-08-17T01:49:15.677Z cpu65:3876714)NMP: nmp_ThrottleLogForDevice:3630: Cmd 0xf1 (0x439e4368a5c0, 65687) to dev "naa.60060160e8004b007e9d9a5cf732ff8e" on path "vmhba2:C0:T11:L47" Failed: H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2020-08-17T01:49:15.677Z cpu65:3876714)ScsiDeviceIO: 3015: Cmd(0x439e43720dc0) 0xfe, CmdSN 0x719b30 from world 65687 to dev "naa.60060160e8004b007e9d9a5cf732ff8e" failed H:0x8 D:0x0 P:0x0 Invalid sense data: 0x80 0x41 0x0.
2020-08-17T01:49:15.677Z cpu65:3876714)NMP: nmp_ThrottleLogForDevice:3630: Cmd 0xf1 (0x439e435e02c0, 65687) to dev "naa.60060160e8004b0058fbdc5d4a63c4ba" on path "vmhba3:C0:T10:L102" Failed: H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2020-08-17T01:49:15.677Z cpu65:3876714)ScsiDeviceIO: 3015: Cmd(0x439e43502940) 0xfe, CmdSN 0x382ef0 from world 65687 to dev "naa.60060160e8004b0058fbdc5d4a63c4ba" failed H:0x8 D:0x0 P:0x0 Invalid sense data: 0x80 0x41 0x0.
2020-08-17T01:49:15.877Z cpu65:3876714)NMP: nmp_ThrottleLogForDevice:3630: Cmd 0xf1 (0x439e435079c0, 65687) to dev "naa.60060160e8004b00de1a995cd70e3c6a" on path "vmhba3:C0:T12:L30" Failed: H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
I entred those value to
so according to this, the HBA does a reset of the target, which in this case is a DELL EMC Unity FC port.
And now what ?
dev "naa.6006" is DELL EMC Storage, in my case Dell EMC Unity 650f running OE 4.5.1 (UWDC01) & OE 5.0.3 (UWDC02)
Current Dell EMC Unity Target code is OE 5.0.3
[root@esx070:~] esxcfg-scsidevs -m | grep "naa.60060160de004b005f5d2a5fbbcad438"
naa.60060160de004b005f5d2a5fbbcad438:1 /vmfs/devices/disks/naa.60060160de004b005f5d2a5fbbcad438:1 5f2a5df1-f99059c6-eed8-20040ff4978e 0 UWDC01_IT-PROD-WDC_V005
[root@esx070:~] esxcfg-scsidevs -m | grep "naa.60060160e8004b00e595255ec02cf074"
naa.60060160e8004b00e595255ec02cf074:1 /vmfs/devices/disks/naa.60060160e8004b00e595255ec02cf074:1 5e2596ab-2bec8188-f141-20040ff4978e 0 UWDC02_IT-PROD-WDC_V103
[root@esx070:~] vmkchdev -l | grep vmhba
0000:00:11.5 8086:a1d2 1734:1230 vmkernel vmhba0
0000:00:17.0 8086:a182 1734:1230 vmkernel vmhba1
0000:17:00.0 1077:2261 1077:029b vmkernel vmhba2 ----------------> FC HBA
0000:6d:00.0 1077:2261 1077:029b vmkernel vmhba3 ----------------> FC HBA
[root@esx070:~] /usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -d
Dumping all key-value instance names:
Key Value Instance: vmhba3/qlogic
Key Value Instance: vmhba2/qlogic
Key Value Instance: vmhba1/vmw_ahci
Key Value Instance: vmhba0/vmw_ahci
Key Value Instance: MOD_PARM/qlogic
[root@esx070:~] /usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -l -i vmhba2/qlogic
Listing keys:
Name: ADAPTER
Type: string
value:
QLogic 16Gb 1-port FC to PCIe Gen3 x8 Adapter for QLE2690:
FC Firmware Version: 8.05.61 (d0d5), Driver version 2.1.73.0
Host Device Name vmhba2
BIOS version 3.61
FCODE version 4.11
EFI version 6.11
Flash FW version 8.05.61
ISP: ISP2261, Serial# RFD1722T35676
MSI-X enabled
Request Queue = 0x4309f6548000, Response Queue = 0x4309f6569000
Request Queue count = 2048, Response Queue count = 512
Number of response queues for CPU affinity operation: 4
CPU Affinity mode enabled
Total number of MSI-X interrupts on vector 0 (handler = 23) = 26676
Total number of MSI-X interrupts on vector 1 (handler = 24) = 2186
Total number of MSI-X interrupts on vector 2 (handler = 25) = 1090148738
Total number of MSI-X interrupts on vector 3 (handler = 26) = 583007145
Total number of MSI-X interrupts on vector 4 (handler = 27) = 2055128386
Total number of MSI-X interrupts on vector 5 (handler = 28) = 1406005796
Device queue depth = 0x8
Number of free request entries = 1271
FAWWN support: disabled
FEC support: Disabled
Total number of outstanding commands: 0
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 29
Host adapter:Loop State = [READY], flags = 0x20ae200
Link speed = [16 Gbps]
Dpc flags = 0x0
Link down Timeout = 010
Port down retry = 010
Login retry count = 010
Execution throttle = 2048
ZIO mode = 0x6, ZIO timer = 1
Commands retried with dropped frame(s) = 297
Product ID = 4953 5020 2261 0001
NPIV Supported : Yes
Max Virtual Ports = 254
SCSI Device Information:
scsi-qla0-adapter-node=20000024ff149042:160a00:0;
scsi-qla0-adapter-port=21000024ff149042:160a00:0;
Name: TARGET
Type: string
value:
Driver version 2.1.73.0
Host Device Name vmhba2
FC Target-Port List:
scsi-qla0-target-0=500000e0da81df29:122300:0:Online;
scsi-qla0-target-1=500000e0da81df39:142300:1:Online;
scsi-qla0-target-2=5006016249e4121e:140000:2:Online;
scsi-qla0-target-3=5006016349e0121e:120000:3:Online;
scsi-qla0-target-4=5006016849e4121e:140100:4:Online;
scsi-qla0-target-5=5006016a49e4121e:120200:5:Online;
scsi-qla0-target-6=5006016249e415ff:0e0000:6:Online;
scsi-qla0-target-7=5006016349e015ff:100000:7:Online;
scsi-qla0-target-8=5006016849e415ff:100100:8:Online;
scsi-qla0-target-9=5006016a49e415ff:0e0100:9:Online;
scsi-qla0-target-10=5006016249e41688:0e0500:a:Online;
scsi-qla0-target-11=5006016349e01688:100200:b:Online;
scsi-qla0-target-12=5006016849e41688:100300:c:Online;
scsi-qla0-target-13=5006016a49e41688:0e0300:d:Online;
Name: NPIV
Type: string
value:
Driver version 2.1.73.0
Host Device Name vmhba2
NPIV Supported : Yes
Looking at the Qlogic Site (Marvell Nowadays) and looing for the QL2690, we are at version -1 compared to the latest
QLogic / Marvell Driver Download
README
Read1st for Cavium Flash Image Package
--------------------------------------
**** ONLY FOR 268x/269x/27xx Series Adapters ****
1. Contents Of Flash Package
--------------------------------
The files contained in this Flash image package are zipped into a file that
will expand to provide the following versions for the 268x/269x/276x Series Adapters.
* Flash Image Version 01.01.91
BK010191.BIN contains:
----------------------
Bootcode FC
FC BIOS v3.62
FC FCode v4.11 (Initiator)
FC FCode v4.10 (Target)
FC EFI v7.00 (Signed)
FC Firmware v8.08.231
MPI Firmware v1.00.19
PEP Firmware(Quad-port) v1.0.27
PEP Firmware(Single/Dual port) v2.0.12
PEP SoftROM(Quad port) v1.0.16
PEP SoftROM(Single/Dual port) v2.0.11
EFlash tool v1.18
