Re: Upgrade to 5.5U3 on IBM BladeCenter H causes V...

Kahonu84 · ‎03-07-2017

Aloha,

We have a BladeCenter H that was acquired thru a merger with another IT dept. It has four blades all running 5.1. I can't say for

sure but my guess is nothing has been upgraded since it was set up long ago.

I am to upgrade the guy to 5.5U3. VCenter went well. I upgraded the first blade and procedurally the upgrade succeeded. However

when I vmotion a vm to 5.5, I lose all network connectivity.

I talked to VMWare support and they advised me to upgrade the NIC driver. I did so and find my situation unchanged. I am wondering

if there is something on the hardware side that needs upgrading. Like I said, I would bet some cash that it is in the same state as

it was the day it was unpacked.

Please understand that I am not very familiar with this architecture - so please reply sloooowly. And yeah, I know it's really old.

Mahalo,

Bill

msripada · ‎03-07-2017

Hello Kahonu84,

Before updating the drivers, have you tried updating the respective firmware for the latest driver as per the VMware HCL.

How did you upgrade the ESXi host, is it using iso or using command line or VMware update manager?

When you use ESXCLI to upgrade an ESXi 4.x or 5.0.x host to version 5.1 or 5.5, the vMotion and Fault Tolerance Logging (FT Logging) settings of any VMKernel port group are lost after the upgrade
If you use the command esxcli software profile update <options> to upgrade an ESXi 4.x or 5.0.x host to version 5.1 or 5.5, the upgrade succeeds, but the vMotion and FT Logging settings of any VMkernel port group are lost. As a result, vMotion and FT Logging are restored to the default setting (disabled).

Workaround: Perform an interactive or scripted upgrade, or use vSphere Update Manager to upgrade hosts. If you use the esxcli command, apply vMotion and FT Logging settings manually to the affected VMkernel port group after the upgrade.

Were you able to ping the VM when you tried vmotion after upgrade ??

Thanks,

MS

Kahonu84 · ‎03-07-2017

To be perfectly honest, I don't know the architecture well enough to know exactly what there is that is upgrade-able.

I upgraded by booting from an iso.

Once I vmotion a vm to the blade I upgraded, no more pingy.

Bill

rcporto · ‎03-07-2017

Can you specify witch blade server you're using? HS22, HS22V, HS23? And did you know the brand/model of your network card? And if possible, attach the vmkernel.log of the affected host.

---

Richardson Porto
Senior Infrastructure Specialist
LinkedIn: http://linkedin.com/in/richardsonporto

Kahonu84 · ‎03-07-2017

He's an HS23 with (6) Emulex Corp. OneConnect 10Gb NIC (be3).

log attached.

Mahalo,

Bill

PS Before I could even vmotion, we had to change the vmotion network from a dedicated vmotion network to the management network. Before doing that,

vmotion stalled out at 14%

rcporto · ‎03-08-2017

Please, let me know the version of device driver you're using and the firmware version. And see that workaround: Re: Vsphere 5.5 and Emulex OneConnect 10Gb NIC trouble

---

Richardson Porto
Senior Infrastructure Specialist
LinkedIn: http://linkedin.com/in/richardsonporto

msripada · ‎03-08-2017

Hello Khaonu84,

Thank you for the logs..

from the vmkernel.log

2017-03-07T01:43:12.761Z cpu3:60466)Migrate: vm 60467: 3286: Setting VMOTION info: Source ts = 1488850990629777, src ip = <172.22.55.236> dest ip = <172.22.55.235> Dest wid = 51289955 using SHARED swap

2017-03-07T01:43:12.762Z cpu9:61034)CpuSched: 583: user latency of 61034 vmotionStreamHelper0-60467 0 changed by 61034 vmotionStreamHelper0-60467 -1

2017-03-07T01:43:12.762Z cpu17:61035)CpuSched: 583: user latency of 61035 vmotionStreamHelper1-60467 0 changed by 61035 vmotionStreamHelper1-60467 -1

2017-03-07T01:43:12.763Z cpu14:61037)CpuSched: 583: user latency of 61037 vmotionRecvHelper-60467 0 changed by 61037 vmotionRecvHelper-60467 -1

2017-03-07T01:43:12.764Z cpu14:61036)CpuSched: 583: user latency of 61036 vmotionSendHelper-60467 0 changed by 61036 vmotionSendHelper-60467 -1

2017-03-07T01:43:12.764Z cpu3:60466)Tcpip_Vmk: 1288: Affinitizing 172.22.55.236 to world 61036, Success

2017-03-07T01:43:12.764Z cpu3:60466)VMotion: 2735: 1488850990629777 S: Set ip address '172.22.55.236' worldlet affinity to send World ID 61036

2017-03-07T01:43:12.764Z cpu3:60466)Hbr: 3340: Migration start received (worldID=60467) (migrateType=1) (event=0) (isSource=1) (sharedConfig=1)

2017-03-07T01:43:12.764Z cpu2:33782)MigrateNet: vm 33782: 2096: Accepted connection from <172.22.55.235>

2017-03-07T01:43:12.764Z cpu2:33782)MigrateNet: vm 33782: 2138: data socket size 0 is less than config option 562140

2017-03-07T01:43:12.764Z cpu2:33782)MigrateNet: vm 33782: 2166: dataSocket 0x410ae3bc3a80 receive buffer size is 562140

2017-03-07T01:43:12.765Z cpu14:61036)VMotionUtil: 3396: 1488850990629777 S: Stream connection 1 added.

2017-03-07T01:43:12.765Z cpu14:61036)VMotionUtil: 3396: 1488850990629777 S: Stream connection 2 added.

2017-03-07T01:43:33.638Z cpu2:60467)VMotion: 4661: 1488850990629777 S: Another pre-copy iteration needed with 26020 pages left to send (prev2 1048576, prev 1048576, network bandwidth ~111.004 MB/s, 24639% t2d)

2017-03-07T01:43:34.547Z cpu4:60467)VMotion: 4574: 1488850990629777 S: Stopping pre-copy: only 1197 pages left to send, which can be sent within the switchover time goal of 0.500 seconds (network bandwidth ~111.525 MB/s, 2053% t2d)

2017-03-07T01:43:34.638Z cpu4:60911)VSCSI: 6448: handle 8193(vscsi4:0):Destroying Device for world 60467 (pendCom 0)

2017-03-07T01:43:34.642Z cpu8:60911)NetPort: 1632: disabled port 0x5000012

2017-03-07T01:43:34.650Z cpu1:60911)CBT: 1561: Disconnecting the cbt device 3b0ff6-cbt with filehandle 3870710

2017-03-07T01:43:35.127Z cpu20:61036)VMotionSend: 3645: 1488850990629777 S: Sent all modified pages to destination (network bandwidth ~109.901 MB/s)

2017-03-07T01:43:36.309Z cpu1:60466)Hbr: 3434: Migration end received (worldID=60467) (migrateType=1) (event=1) (isSource=1) (sharedConfig=1)

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic2: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic5: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic4: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic3: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic1: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)WARNING: netsched: NetSchedMClkRemoveCfg:3510: vmnic0: failed to find queue netsched.pools.vm.83886098 to delete it: Not found

2017-03-07T01:43:36.315Z cpu6:32857)Net: 3348: dissociate dvPort 178 from port 0x5000012

2017-03-07T01:43:36.315Z cpu6:32857)Net: 3354: disconnected client from port 0x5000012

2017-03-07T01:43:36.434Z cpu14:34396)DLX: 3768: vol 'EXPANSION_B_10K', lock at 145264640: [Req mode 1] Checking liveness:

2017-03-07T01:43:36.434Z cpu14:34396)[type 10c00001 offset 145264640 v 1214, hb offset 3956736

gen 15243, mode 1, owner 5609dd77-34406aac-ce49-3440b5e222f8 mtime 12320974

num 0 gblnum 0 gblgen 0 gblbrk 0]

2017-03-07T01:43:40.438Z cpu14:34396)DLX: 4297: vol 'EXPANSION_B_10K', lock at 145264640: [Req mode: 1] Not free:

2017-03-07T01:43:40.438Z cpu14:34396)[type 10c00001 offset 145264640 v 1214, hb offset 3956736

gen 15243, mode 1, owner 5609dd77-34406aac-ce49-3440b5e222f8 mtime 12320974

num 0 gblnum 0 gblgen 0 gblbrk 0]

2017-03-07T01:43:40.483Z cpu19:33976)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

2017-03-07T01:48:33.240Z cpu2:32807)ScsiDeviceIO: 2353: Cmd(0x412ec0418800) 0x1a, CmdSN 0x20f4 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T02:18:33.258Z cpu2:32807)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec042b000, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T02:48:33.274Z cpu12:32817)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec0410040, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T02:58:33.279Z cpu20:32825)ScsiDeviceIO: 2353: Cmd(0x412ec0410040) 0x1a, CmdSN 0x2384 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T03:18:33.290Z cpu0:32805)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec53482c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T03:48:33.306Z cpu0:32805)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec98e3780, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T04:08:33.317Z cpu13:32818)ScsiDeviceIO: 2353: Cmd(0x412eca5ebe00) 0x1a, CmdSN 0x2622 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T04:18:33.322Z cpu17:32822)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec9e50980, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T04:53:33.337Z cpu4:32809)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec52fe940, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T05:03:33.340Z cpu8:32813)ScsiDeviceIO: 2353: Cmd(0x412ec041cf40) 0x1a, CmdSN 0x282b from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T05:23:33.353Z cpu6:32811)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec534b4c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T05:53:33.369Z cpu1:32806)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5349a80, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T05:58:33.375Z cpu13:32818)ScsiDeviceIO: 2353: Cmd(0x412ec043bc80) 0x1a, CmdSN 0x2a26 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T06:23:33.385Z cpu9:32814)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec04365c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T06:58:33.405Z cpu13:32818)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca649740, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T07:03:33.406Z cpu4:32809)ScsiDeviceIO: 2353: Cmd(0x412ec98e4cc0) 0x1a, CmdSN 0x2c97 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T07:28:33.423Z cpu8:32813)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec0416bc0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T07:32:02.578Z cpu15:32858)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x8a failed <1/18> sid x140200, did x140f00, oxid xffff Time Out Returning Host Error

2017-03-07T07:32:02.578Z cpu7:33589)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x8a (0x412eca8d4780, 32927) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T07:32:02.578Z cpu7:33589)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T07:32:02.578Z cpu7:33589)ScsiDeviceIO: 2369: Cmd(0x412eca8d4780) 0x8a, CmdSN 0xb734 from world 32927 to dev "naa.60050768028087cce000000000000021" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T07:32:08.998Z cpu14:33448)lpfc: lpfc_rportStats:3640: 1:(0) Compression log for fcp target 1, path is ok, IO errors: busy 0, retry 0, no_connect 0, fcperr 24, tmo 3

2017-03-07T07:35:32.573Z cpu15:32833)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x8a failed <1/18> sid x140200, did x140f00, oxid xffff Time Out Returning Host Error

2017-03-07T07:35:32.573Z cpu7:32812)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x8a (0x412ec5872600, 32927) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T07:35:32.573Z cpu7:32812)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T07:35:32.573Z cpu7:32812)ScsiDeviceIO: 2369: Cmd(0x412ec5872600) 0x8a, CmdSN 0xb856 from world 32927 to dev "naa.60050768028087cce000000000000021" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T07:37:14.117Z cpu16:33448)lpfc: lpfc_rportStats:3640: 1:(0) Compression log for fcp target 1, path is ok, IO errors: busy 0, retry 0, no_connect 0, fcperr 24, tmo 4

2017-03-07T07:58:33.439Z cpu3:32808)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec90c9b00, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T08:08:33.444Z cpu22:32827)ScsiDeviceIO: 2353: Cmd(0x412ec03e91c0) 0x1a, CmdSN 0x2efa from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T08:28:33.455Z cpu5:32810)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec03c6200, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T08:58:33.471Z cpu4:32809)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec040ba40, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T09:13:33.475Z cpu2:32807)ScsiDeviceIO: 2353: Cmd(0x412ec5732240) 0x1a, CmdSN 0x315d from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T09:28:33.487Z cpu6:32811)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5732240, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T09:36:49.382Z cpu23:33077)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x8a failed <1/18> sid x140200, did x140f00, oxid xffff Time Out Returning Host Error

2017-03-07T09:36:49.383Z cpu6:33589)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x8a (0x412ec55bb9c0, 32927) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T09:36:49.383Z cpu6:33589)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T09:36:49.383Z cpu6:33589)ScsiDeviceIO: 2369: Cmd(0x412ec55bb9c0) 0x8a, CmdSN 0xdea2 from world 32927 to dev "naa.60050768028087cce000000000000021" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T09:39:16.985Z cpu12:33448)lpfc: lpfc_rportStats:3640: 1:(0) Compression log for fcp target 1, path is ok, IO errors: busy 0, retry 0, no_connect 0, fcperr 24, tmo 5

2017-03-07T09:58:33.503Z cpu10:32815)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5734900, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T10:28:33.519Z cpu6:32811)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca5ede80, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T10:28:33.519Z cpu6:32811)ScsiDeviceIO: 2353: Cmd(0x412eca5ede80) 0x1a, CmdSN 0x341a from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T11:03:33.537Z cpu1:32806)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca64bf40, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T11:13:59.536Z cpu22:33080)lpfc0 log compression on ending, messages have quiesced with 0 compressed.

2017-03-07T11:13:59.536Z cpu22:33080)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number xfd35, skip_abts 0

2017-03-07T11:13:59.537Z cpu19:33073)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/18> sid x140200, did x140f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T11:13:59.537Z cpu19:32824)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x412ec90cc440, 32797) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T11:13:59.537Z cpu19:32824)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T11:13:59.537Z cpu19:32824)ScsiDeviceIO: 2369: Cmd(0x412ec90cc440) 0x89, CmdSN 0xfd35 from world 32797 to dev "naa.60050768028087cce000000000000021" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T11:14:03.537Z cpu6:32927)HBX: 2961: Waiting for timed out [HB state abcdef02 offset 3379200 gen 23 stampUS 42318007756 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60] on vol 'EXPANSION_A_15K'

2017-03-07T11:14:05.467Z cpu9:165647)ScsiCore: 63: Starting taskmgmt handler world 165647/2

2017-03-07T11:14:05.468Z cpu9:165647)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x458a, skip_abts 0

2017-03-07T11:14:05.469Z cpu7:33061)lpfc0 log compression on target 1 ending, messages have quiesced with 20 compressed.

2017-03-07T11:14:05.469Z cpu7:33061)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 0:(0):3271: FCP cmd x89 failed <1/2> sid x0a0200, did x0a0f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T11:14:05.469Z cpu19:32824)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x412ec03dcdc0, 32797) to dev "naa.60050768028087cce000000000000022" on path "vmhba1:C0:T1:L2" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T11:14:05.469Z cpu19:32824)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000022" state in doubt; requested fast path state update...

2017-03-07T11:14:05.469Z cpu19:32824)ScsiDeviceIO: 2369: Cmd(0x412ec03dcdc0) 0x89, CmdSN 0x458a from world 32797 to dev "naa.60050768028087cce000000000000022" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T11:14:19.539Z cpu22:33080)ScsiCore: 98: Stopping taskMgmt handler world 330801

2017-03-07T11:14:25.488Z cpu2:165688)ScsiCore: 63: Starting taskmgmt handler world 165688/2

2017-03-07T11:14:25.488Z cpu9:165647)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number xfd35, skip_abts 0

2017-03-07T11:14:25.562Z cpu22:32840)HBX: 270: Reclaimed heartbeat for volume 55db6fdf-e2139e8e-4e72-3440b5e22400 (EXPANSION_A_15K): [Timeout] Offset 3379200

2017-03-07T11:14:25.562Z cpu22:32840)[HB state abcdef02 offset 3379200 gen 23 stampUS 42352032283 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60]

2017-03-07T11:14:27.473Z cpu2:165688)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x458a, skip_abts 0

2017-03-07T11:14:27.547Z cpu4:32841)HBX: 270: Reclaimed heartbeat for volume 55dcbdc8-d7de1174-b073-3440b5e22400 (EXPANSION_B_10K): [Timeout] Offset 3379712

2017-03-07T11:14:27.547Z cpu4:32841)[HB state abcdef02 offset 3379712 gen 1 stampUS 42354017206 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 5813582> drv 14.60]

2017-03-07T11:15:48.671Z cpu17:33448)lpfc: lpfc_rportStats:3654: 0:(0) Compression log for fcp target 1, path is ok, TMGMT: abts=1, tgt_rst=0, lun_rst=0

2017-03-07T11:15:54.293Z cpu17:33448)lpfc: lpfc_rportStats:3654: 1:(0) Compression log for fcp target 1, path is ok, TMGMT: abts=1, tgt_rst=0, lun_rst=0

2017-03-07T11:28:33.551Z cpu4:32809)ScsiDeviceIO: 2353: Cmd(0x412eca5edac0) 0x1a, CmdSN 0x3650 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T11:38:33.556Z cpu13:32818)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca90e340, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T11:44:27.491Z cpu9:165647)ScsiCore: 98: Stopping taskMgmt handler world 1656471

2017-03-07T12:08:33.572Z cpu8:32813)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec9e55700, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T12:33:33.582Z cpu5:32810)ScsiDeviceIO: 2353: Cmd(0x412ec98e4680) 0x1a, CmdSN 0x38b5 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T12:48:33.593Z cpu14:32819)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca64a640, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T13:18:33.609Z cpu8:32813)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5355480, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T13:28:33.615Z cpu1:32806)ScsiDeviceIO: 2353: Cmd(0x412eca8b1bc0) 0x1a, CmdSN 0x3abc from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T13:53:33.628Z cpu1:32806)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec57294c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T14:23:33.644Z cpu0:32805)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec990c6c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T14:23:33.644Z cpu0:32805)ScsiDeviceIO: 2353: Cmd(0x412ec990c6c0) 0x1a, CmdSN 0x3cc5 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T14:53:33.660Z cpu20:32825)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec03eb880, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T15:03:34.462Z cpu4:165688)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x14ba3, skip_abts 0

2017-03-07T15:03:34.462Z cpu18:33048)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/18> sid x140200, did x140f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T15:03:34.462Z cpu18:32823)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x4136c0414f40, 33323) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T15:03:34.462Z cpu18:32823)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T15:03:34.462Z cpu18:32823)ScsiDeviceIO: 2369: Cmd(0x4136c0414f40) 0x89, CmdSN 0x14ba3 from world 33323 to dev "naa.60050768028087cce000000000000021" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T15:03:44.859Z cpu12:207458)ScsiCore: 63: Starting taskmgmt handler world 207458/2

2017-03-07T15:03:44.861Z cpu12:207458)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x14ba4, skip_abts 0

2017-03-07T15:03:44.861Z cpu18:33072)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/18> sid x140200, did x140f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T15:03:44.861Z cpu18:32823)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T15:03:44.861Z cpu18:32823)ScsiDeviceIO: 2369: Cmd(0x4136c75a60c0) 0x89, CmdSN 0x14ba4 from world 32797 to dev "naa.60050768028087cce000000000000021" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T15:03:46.863Z cpu14:207463)ScsiCore: 63: Starting taskmgmt handler world 207463/3

2017-03-07T15:03:46.864Z cpu14:207463)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x5772, skip_abts 0

2017-03-07T15:03:46.865Z cpu18:33048)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/2> sid x140200, did x140f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T15:03:46.865Z cpu18:32823)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x4136c55dba40, 32797) to dev "naa.60050768028087cce000000000000022" on path "vmhba2:C0:T1:L2" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T15:03:46.865Z cpu18:32823)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000022" state in doubt; requested fast path state update...

2017-03-07T15:03:46.865Z cpu18:32823)ScsiDeviceIO: 2369: Cmd(0x4136c55dba40) 0x89, CmdSN 0x5772 from world 32797 to dev "naa.60050768028087cce000000000000022" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T15:03:54.465Z cpu4:165688)ScsiCore: 98: Stopping taskMgmt handler world 1656882

2017-03-07T15:03:58.877Z cpu8:207488)ScsiCore: 63: Starting taskmgmt handler world 207488/3

2017-03-07T15:03:58.878Z cpu8:207488)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x14ba3, skip_abts 0

2017-03-07T15:03:58.879Z cpu18:32799)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/18> sid x140200, did x140f00, oxid xffff SCSI Chk Cond - 0xe: Data(x2:xe:x1d:x0)

2017-03-07T15:03:58.879Z cpu18:32823)ScsiDeviceIO: 2369: Cmd(0x4136c6c80d40) 0x89, CmdSN 0x14bab from world 33323 to dev "naa.60050768028087cce000000000000021" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

2017-03-07T15:03:58.879Z cpu12:33323)J3: 3239: Aborting txn (0x410c6d53b730) callerID: 0xc1d00001 due to failure pre-committing: Timeout

2017-03-07T15:03:58.880Z cpu12:33323)WARNING: J3: 3438: Committing transaction failed: Timeout

2017-03-07T15:03:58.880Z cpu12:33699)HBX: 2961: Waiting for timed out [HB state abcdef02 offset 3379200 gen 23 stampUS 56100009545 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60] on vol 'EXPANSION_A_15K'

2017-03-07T15:04:06.865Z cpu12:207458)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x14ba4, skip_abts 0

2017-03-07T15:04:06.938Z cpu12:32840)HBX: 270: Reclaimed heartbeat for volume 55db6fdf-e2139e8e-4e72-3440b5e22400 (EXPANSION_A_15K): [Timeout] Offset 3379200

2017-03-07T15:04:06.938Z cpu12:32840)[HB state abcdef02 offset 3379200 gen 23 stampUS 56133288377 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60]

2017-03-07T15:04:07.687Z cpu3:33033)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 0:(0):3271: FCP cmd x89 failed <1/18> sid x0a0200, did x0a0f00, oxid xffff SCSI Chk Cond - 0xe: Data(x2:xe:x1d:x0)

2017-03-07T15:04:07.687Z cpu1:32806)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x4136c0436d40, 33323) to dev "naa.60050768028087cce000000000000021" on path "vmhba1:C0:T1:L18" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

2017-03-07T15:04:07.687Z cpu1:32806)ScsiDeviceIO: 2369: Cmd(0x4136c0436d40) 0x89, CmdSN 0x14d64 from world 33323 to dev "naa.60050768028087cce000000000000021" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

2017-03-07T15:04:08.869Z cpu8:207488)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x5772, skip_abts 0

2017-03-07T15:04:08.943Z cpu4:32841)HBX: 270: Reclaimed heartbeat for volume 55dcbdc8-d7de1174-b073-3440b5e22400 (EXPANSION_B_10K): [Timeout] Offset 3379712

2017-03-07T15:04:08.943Z cpu4:32841)[HB state abcdef02 offset 3379712 gen 1 stampUS 56135292679 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 5813582> drv 14.60]

2017-03-07T15:04:44.715Z cpu12:33448)lpfc: lpfc_rportStats:3654: 1:(0) Compression log for fcp target 1, path is ok, TMGMT: abts=4, tgt_rst=0, lun_rst=0

2017-03-07T15:18:33.673Z cpu7:32812)ScsiDeviceIO: 2353: Cmd(0x412ec9909740) 0x1a, CmdSN 0x3ece from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T15:28:33.679Z cpu4:32809)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5b04cc0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T15:34:08.885Z cpu14:207463)ScsiCore: 98: Stopping taskMgmt handler world 2074632

2017-03-07T15:34:08.887Z cpu12:207458)ScsiCore: 98: Stopping taskMgmt handler world 2074581

2017-03-07T15:44:26.681Z cpu8:207488)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x15a1e, skip_abts 0

2017-03-07T15:44:26.681Z cpu7:33590)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x89 failed <1/18> sid x140200, did x140f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T15:44:26.681Z cpu18:32823)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x412ec52d4b80, 32797) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T15:44:26.681Z cpu18:32823)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T15:44:26.681Z cpu18:32823)ScsiDeviceIO: 2369: Cmd(0x412ec52d4b80) 0x89, CmdSN 0x15a1e from world 32797 to dev "naa.60050768028087cce000000000000021" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T15:44:33.211Z cpu14:214609)ScsiCore: 63: Starting taskmgmt handler world 214609/2

2017-03-07T15:44:33.212Z cpu14:214609)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x5a99, skip_abts 0

2017-03-07T15:44:33.212Z cpu21:33051)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 0:(0):3271: FCP cmd x89 failed <1/2> sid x0a0200, did x0a0f00, oxid xffff Abort Requested Host Abort Req

2017-03-07T15:44:33.212Z cpu2:32807)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x89 (0x412eca91bc40, 32797) to dev "naa.60050768028087cce000000000000022" on path "vmhba1:C0:T1:L2" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T15:44:33.212Z cpu2:32807)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000022" state in doubt; requested fast path state update...

2017-03-07T15:44:33.212Z cpu2:32807)ScsiDeviceIO: 2369: Cmd(0x412eca91bc40) 0x89, CmdSN 0x5a99 from world 32797 to dev "naa.60050768028087cce000000000000022" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T15:44:33.681Z cpu2:32927)HBX: 2961: Waiting for timed out [HB state abcdef02 offset 3379200 gen 23 stampUS 58545008581 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60] on vol 'EXPANSION_A_15K'

2017-03-07T15:44:46.682Z cpu8:207488)ScsiCore: 98: Stopping taskMgmt handler world 2074881

2017-03-07T15:44:51.229Z cpu12:214676)ScsiCore: 63: Starting taskmgmt handler world 214676/2

2017-03-07T15:44:51.231Z cpu12:214676)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <1:18> for cmd serial number x15a1e, skip_abts 0

2017-03-07T15:44:51.304Z cpu12:32840)HBX: 270: Reclaimed heartbeat for volume 55db6fdf-e2139e8e-4e72-3440b5e22400 (EXPANSION_A_15K): [Timeout] Offset 3379200

2017-03-07T15:44:51.304Z cpu12:32840)[HB state abcdef02 offset 3379200 gen 23 stampUS 58577632789 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 13827363> drv 14.60]

2017-03-07T15:44:55.217Z cpu14:214609)lpfc: lpfc_taskmgmt:3560: 0:(0):3267 Task Mgmt Abort - IO not found on <0:2> for cmd serial number x5a99, skip_abts 0

2017-03-07T15:44:55.290Z cpu4:32841)HBX: 270: Reclaimed heartbeat for volume 55dcbdc8-d7de1174-b073-3440b5e22400 (EXPANSION_B_10K): [Timeout] Offset 3379712

2017-03-07T15:44:55.290Z cpu4:32841)[HB state abcdef02 offset 3379712 gen 1 stampUS 58581618417 uuid 58bdf0f9-c2dbcd86-ede1-3440b5e22428 jrnl <FB 5813582> drv 14.60]

2017-03-07T15:45:20.067Z cpu14:33448)lpfc: lpfc_rportStats:3654: 0:(0) Compression log for fcp target 1, path is ok, TMGMT: abts=2, tgt_rst=0, lun_rst=0

2017-03-07T15:45:25.683Z cpu14:33448)lpfc: lpfc_rportStats:3654: 1:(0) Compression log for fcp target 1, path is ok, TMGMT: abts=5, tgt_rst=0, lun_rst=0

2017-03-07T15:58:33.695Z cpu3:32808)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec9e5cf80, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T16:13:33.700Z cpu0:32805)ScsiDeviceIO: 2353: Cmd(0x412eca631740) 0x1a, CmdSN 0x40d7 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T16:14:55.233Z cpu14:214609)ScsiCore: 98: Stopping taskMgmt handler world 2146091

2017-03-07T16:28:33.711Z cpu17:32822)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec03dac00, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T16:58:33.727Z cpu1:32806)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca61ba40, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T17:03:33.727Z cpu10:32815)ScsiDeviceIO: 2353: Cmd(0x412eca61ba40) 0x1a, CmdSN 0x42b3 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T17:28:33.744Z cpu8:32813)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec03f5f40, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T17:58:33.760Z cpu3:32808)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412eca908d00, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T17:58:33.760Z cpu3:32808)ScsiDeviceIO: 2353: Cmd(0x412eca908d00) 0x1a, CmdSN 0x44ae from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T18:28:33.776Z cpu22:32827)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec534d900, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T18:36:33.538Z cpu19:34688)lpfc: lpfc_scsi_cmd_iocb_cmpl:2057: 1:(0):3271: FCP cmd x8a failed <1/18> sid x140200, did x140f00, oxid xffff Time Out Returning Host Error

2017-03-07T18:36:33.538Z cpu0:33589)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x8a (0x412ec042c7c0, 32927) to dev "naa.60050768028087cce000000000000021" on path "vmhba2:C0:T1:L18" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2017-03-07T18:36:33.539Z cpu0:33589)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60050768028087cce000000000000021" state in doubt; requested fast path state update...

2017-03-07T18:36:33.539Z cpu0:33589)ScsiDeviceIO: 2369: Cmd(0x412ec042c7c0) 0x8a, CmdSN 0x190b8 from world 32927 to dev "naa.60050768028087cce000000000000021" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-03-07T18:38:19.750Z cpu14:33448)lpfc: lpfc_rportStats:3640: 1:(0) Compression log for fcp target 1, path is ok, IO errors: busy 0, retry 0, no_connect 0, fcperr 25, tmo 6

2017-03-07T18:58:33.792Z cpu0:32805)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec55bafc0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T18:58:33.792Z cpu0:32805)ScsiDeviceIO: 2353: Cmd(0x412ec55bafc0) 0x1a, CmdSN 0x46e4 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T19:28:33.808Z cpu3:32808)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec5873640, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T19:48:33.818Z cpu18:32823)ScsiDeviceIO: 2353: Cmd(0x412eca61a3c0) 0x1a, CmdSN 0x48c0 from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-03-07T19:58:33.824Z cpu10:32815)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec55b8a40, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T20:28:33.840Z cpu4:32809)NMP: nmp_ThrottleLogForDevice:2458: Cmd 0x1a (0x412ec03fb8c0, 0) to dev "mpx.vmhba33:C0:T0:L0" on path "vmhba33:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-03-07T20:38:33.844Z cpu4:32809)ScsiDeviceIO: 2353: Cmd(0x412eca629c40) 0x1a, CmdSN 0x4a9c from world 0 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

This continues to happen and am seeing this an issue with the HBA

Host Status

[0x5]

ABORT

This status is returned if the driver has to abort commands in-flight to the target. This can occur due to a command timeout or parity error in the frame.

Host Status

[0x7]

ERROR

This status is returned when a device has been reset due to a Storage Initiator Error. This typically occurs due to an outdated HBA firmware or possibly (though rarely) as the result of a bad HBA.

We are seeing other errors also but possibly an issue with bad HBA or timeout i suspect. Can you check if the VM is residing on the same datastore either naa.60050768028087cce000000000000021 or naa.60050768028087cce000000000000021

Thanks,

MS

Kahonu84 · ‎03-08-2017

The LUN ID the VM is on ends in 22.

msripada · ‎03-08-2017

Hello Kahonu84,

It is not the Lun ID. check the Datastore on which the VM is residing, you can find it by clicking on the VM summary page - Storage/Datastore -> Once you identified the datastore name

Click on the ESXi host -> Configuration -> Storage -> find the datastore - > You have to expand Device column and find the naa.<number> there and check if it is matching with any of those storage which I pointed out in my previous reply.

Thanks,

MS

All

Upgrade to 5.5U3 on IBM BladeCenter H causes VM's to lose network connectivity