VMware Cloud Community
MarvinHuffaker
Contributor
Contributor

ESXI 7.0.2 Host Hanging after upgrade from 6.5 to 7

I have an ESXI hang situation that started after upgrading to version 7.0.2. It sounds very similar to this other thread: https://communities.vmware.com/t5/ESXi-Discussions/ESXi-hanging-issues-on-multiple-hosts/td-p/284191...

The server is a Dell PowerEdge R630 / which is fully certified to be compatible. The server was previously running ESXI version 6.5 with no issues.  I upgraded it to version 7 recently. Since then I am experiencing hangs after a couple days of operations.

Server Config:

  • PowerEdge R630 Server, Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  • All firmware is updated to current levels.
  • Direct Attached Storage. Uses a Perc H730 Controller as well as two Samsung 1725 PCIe SSD Cards (Purchased from Dell with the server)
  • ESXi 7.0.2 Build 17630552
  • ESXI is installed on a dell SD Card.
  • I used the Dell custom esxi image to upgrade using Update Manager.

Symptoms:

  • Doing basic tasks at the command line will hang the console for a long time before displaying anything.  For example, a directory listing with "ls" or just viewing a log file such as "cat /var/log/mvkernel.log"
  • Cannot power on/off any VM on the system. It will sit at 0% for a long time and eventually error out with a connection timeout.
  • Cannot run Veeam replication jobs, they all fail.
  • Cannot do snapshots
  • Eventually cannot access the ESXI web console page
  • As the condition degrades, eventually the host disconnects from vcenter.
  • Cannot access any VM through either the remote console or web console. Blank screen or eventually an error "unable to connect to the MKS"
  • Cannot reboot the esxi host cleanly. I had to reset the power from the DRAC to clear this up last time it happened.
  • Utilization seems to run high and although VM's continue to run, they start to experience erratic performance issues. 

Snippets from the The vmkernel.log:

Spoiler
2021-04-26T00:35:44.485Z cpu12:2097360)Queued:1
2021-04-26T00:35:44.485Z cpu2:2097341)ScsiVmas: 1057: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L0 failed with error Timeout
2021-04-26T00:35:44.486Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c631c0) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:35:48.735Z cpu12:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0dd6ac0) 0x28, cmdId.initiator=0x4305118d2080 CmdSN 0x1 from world 2100012 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:35:48.735Z cpu12:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:35:48.735Z cpu12:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0dd6ac0) 0x28, cmdId.initiator=0x4305118d2080 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:35:48.735Z cpu12:2097360)Queued:1
2021-04-26T00:35:48.736Z cpu9:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:35:55.487Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c14fc0) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:35:55.487Z cpu9:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:36:04.486Z cpu4:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 6 times
2021-04-26T00:36:06.486Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c48dc0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:36:06.486Z cpu3:2097361)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x0 (0x45d8c0c48dc0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:36:06.486Z cpu3:2097361)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x453891e1bc58 CmdSN 0x0
2021-04-26T00:36:06.486Z cpu11:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:36:24.485Z cpu12:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e2127f40) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e8d9 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:36:24.485Z cpu12:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:36:24.485Z cpu12:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e2127f40) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e8d9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:36:24.485Z cpu12:2097360)Queued:2
2021-04-26T00:36:24.487Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c631c0) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:36:28.736Z cpu12:2097198)ScsiDeviceIO: 4315: Cmd(0x45d8e375f580) 0x9e, cmdId.initiator=0x45389639a3f8 CmdSN 0x1e8db from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer. Cmd count Active:1
2021-04-26T00:36:28.736Z cpu12:2097198)Queued:0
2021-04-26T00:36:28.737Z cpu12:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0cc62c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8da from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:36:28.738Z cpu12:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:36:28.738Z cpu12:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0cc62c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8da from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:36:28.738Z cpu12:2097360)Queued:1
2021-04-26T00:36:28.738Z cpu10:2097348)ScsiDevice: 979: Could not flush cache of local device mpx.vmhba32:C0:T0:L0. Failure
2021-04-26T00:36:34.485Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37e7480) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:36:34.486Z cpu10:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:36:45.485Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d112c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:36:45.485Z cpu9:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:37:04.485Z cpu4:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 5 times
2021-04-26T00:37:08.739Z cpu13:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8e375f580) 0x25, cmdId.initiator=0x45389639a4a8 CmdSN 0x1e8e0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:37:08.739Z cpu13:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x25 (0x45d8e375f580, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:37:08.739Z cpu13:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a4a8 CmdSN 0x1e8e0
2021-04-26T00:37:08.739Z cpu13:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:37:08.739Z cpu13:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8e375f580) 0x25, cmdId.initiator=0x45389639a4a8 CmdSN 0x1e8e0 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:37:08.739Z cpu13:2097360)Queued:0
2021-04-26T00:37:19.485Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3721780) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:37:19.485Z cpu14:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:37:48.740Z cpu10:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8e375f580) 0x1a, cmdId.initiator=0x45389639a3e8 CmdSN 0x1e8e3 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:37:48.740Z cpu10:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:37:48.740Z cpu10:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8e375f580) 0x1a, cmdId.initiator=0x45389639a3e8 CmdSN 0x1e8e3 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:37:48.740Z cpu10:2097360)Queued:0
2021-04-26T00:37:59.486Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c14fc0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:37:59.486Z cpu8:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:38:04.487Z cpu3:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 3 times
2021-04-26T00:38:28.741Z cpu10:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8e375f580) 0x1a, cmdId.initiator=0x45389639a4c8 CmdSN 0x1e8e4 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:38:28.741Z cpu10:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8e375f580, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:38:28.741Z cpu10:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a4c8 CmdSN 0x1e8e4
2021-04-26T00:38:28.741Z cpu10:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:38:28.741Z cpu10:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8e375f580) 0x1a, cmdId.initiator=0x45389639a4c8 CmdSN 0x1e8e4 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:38:28.741Z cpu10:2097360)Queued:0
2021-04-26T00:38:39.486Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e368cb80) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:38:39.486Z cpu10:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:39:04.485Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 1 times
2021-04-26T00:39:08.744Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8e375f580) 0x28, cmdId.initiator=0x430511976f00 CmdSN 0x1 from world 2100012 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:39:08.744Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x28 (0x45d8e375f580, 2100012) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:39:08.744Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430511976f00 CmdSN 0x1
2021-04-26T00:39:08.744Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:39:08.744Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8e375f580) 0x28, cmdId.initiator=0x430511976f00 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:39:08.744Z cpu9:2097360)Queued:0
2021-04-26T00:39:08.744Z cpu8:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:39:08.744Z cpu8:2100597)LVM: 6192: Could not open device mpx.vmhba32:C0:T0:L0:7, vol [60724578-09db3a26-f7cb-000af7abb22c, 60724578-09db3a26-f7cb-000af7abb22c, 1]: Timeout
2021-04-26T00:39:08.745Z cpu8:2100597)Vol3: 2129: Could not open device 'mpx.vmhba32:C0:T0:L0:7' for volume open: Not found
2021-04-26T00:39:08.745Z cpu8:2100597)Vol3: 4339: Failed to get object 28 type 1 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 0 gen 0 :Not found
2021-04-26T00:39:08.745Z cpu8:2100597)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 6072457a 332348ce a005c7a 2cb2abf7 0 0 0 0 0 0 0
2021-04-26T00:39:08.745Z cpu8:2100597)Vol3: 4339: Failed to get object 28 type 2 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 4 gen 1 :Not found
2021-04-26T00:39:19.486Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c5a5c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:39:19.486Z cpu8:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:39:48.744Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0ca50c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8e7 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:39:48.744Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:39:48.744Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0ca50c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8e7 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:39:48.744Z cpu9:2097360)Queued:1
2021-04-26T00:39:48.744Z cpu8:2097348)ScsiDevice: 979: Could not flush cache of local device mpx.vmhba32:C0:T0:L0. Failure
2021-04-26T00:39:48.748Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x9e, cmdId.initiator=0x45389639a458 CmdSN 0x1e8e8 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:39:48.748Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x9e, cmdId.initiator=0x45389639a458 CmdSN 0x1e8e8 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:39:48.748Z cpu9:2097360)Queued:0
2021-04-26T00:39:59.485Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d126c0) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:39:59.485Z cpu14:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:40:04.485Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 4 times
2021-04-26T00:40:14.486Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9d3c0) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:40:14.486Z cpu6:2097361)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x0 (0x45d8c0c9d3c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:40:14.486Z cpu6:2097361)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x453885e1bc58 CmdSN 0x0
2021-04-26T00:40:14.486Z cpu14:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:40:28.750Z cpu10:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x25, cmdId.initiator=0x45389639a508 CmdSN 0x1e8e9 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:40:28.750Z cpu10:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:40:28.750Z cpu10:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x25, cmdId.initiator=0x45389639a508 CmdSN 0x1e8e9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:40:28.750Z cpu10:2097360)Queued:1
2021-04-26T00:40:39.489Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c7e4c0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:40:39.489Z cpu9:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:40:44.488Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37d7f80) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:40:44.489Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e6fafec0) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e8ee from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:40:44.489Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:40:44.489Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e6fafec0) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e8ee from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:40:44.489Z cpu8:2097360)Queued:1
2021-04-26T00:40:44.489Z cpu2:2097341)ScsiVmas: 1057: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L0 failed with error Timeout
2021-04-26T00:40:55.486Z cpu2:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3660580) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:40:55.486Z cpu9:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:41:04.485Z cpu6:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 5 times
2021-04-26T00:41:08.750Z cpu14:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x1a, cmdId.initiator=0x45389639a448 CmdSN 0x1e8ef from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:41:08.750Z cpu14:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8c0c9c4c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:41:08.750Z cpu14:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a448 CmdSN 0x1e8ef
2021-04-26T00:41:08.750Z cpu14:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:41:08.750Z cpu14:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x1a, cmdId.initiator=0x45389639a448 CmdSN 0x1e8ef from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:41:08.750Z cpu14:2097360)Queued:1
2021-04-26T00:41:19.487Z cpu2:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37a4680) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:41:19.487Z cpu8:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:41:24.488Z cpu2:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37d7f80) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:41:24.490Z cpu14:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e6fafec0) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e8f0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:41:24.490Z cpu14:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:41:24.490Z cpu14:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e6fafec0) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e8f0 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:41:24.490Z cpu14:2097360)Queued:1
2021-04-26T00:41:35.488Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37aaa80) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:41:35.488Z cpu14:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:41:48.750Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x1a, cmdId.initiator=0x45389639a528 CmdSN 0x1e8f1 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:41:48.750Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:41:48.751Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x1a, cmdId.initiator=0x45389639a528 CmdSN 0x1e8f1 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:41:48.751Z cpu8:2097360)Queued:0
2021-04-26T00:41:59.485Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c4bfc0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:41:59.485Z cpu9:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:42:04.485Z cpu6:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 6 times
2021-04-26T00:42:28.751Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x28, cmdId.initiator=0x430511963440 CmdSN 0x1 from world 2100012 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:42:28.751Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x28 (0x45d8c0c9c4c0, 2100012) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:42:28.751Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430511963440 CmdSN 0x1
2021-04-26T00:42:28.751Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:42:28.751Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x28, cmdId.initiator=0x430511963440 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:42:28.751Z cpu9:2097360)Queued:0
2021-04-26T00:42:28.751Z cpu8:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:42:39.487Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d329c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:42:39.487Z cpu8:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:43:04.487Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 1 times
2021-04-26T00:43:08.752Z cpu8:2097194)ScsiDeviceIO: 4315: Cmd(0x45d8c0c2c1c0) 0x9e, cmdId.initiator=0x45389639a458 CmdSN 0x1e8f9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer. Cmd count Active:1
2021-04-26T00:43:08.752Z cpu8:2097194)Queued:0
2021-04-26T00:43:10.485Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c9c4c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8f8 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:43:10.486Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x35 (0x45d8c0c9c4c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:43:10.486Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45388621be18 CmdSN 0x1e8f8
2021-04-26T00:43:10.486Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:43:10.486Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c9c4c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e8f8 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:43:10.486Z cpu9:2097360)Queued:1
2021-04-26T00:43:10.486Z cpu8:2097348)ScsiDevice: 979: Could not flush cache of local device mpx.vmhba32:C0:T0:L0. Failure
2021-04-26T00:43:20.486Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e372a380) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:43:20.486Z cpu15:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:43:48.755Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c2c1c0) 0x25, cmdId.initiator=0x45389639a508 CmdSN 0x1e8fa from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:43:48.755Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:43:48.755Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c2c1c0) 0x25, cmdId.initiator=0x45389639a508 CmdSN 0x1e8fa from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:43:48.755Z cpu9:2097360)Queued:0
2021-04-26T00:43:59.487Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e368da80) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:43:59.487Z cpu10:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:44:04.487Z cpu5:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 3 times
2021-04-26T00:44:28.757Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c2c1c0) 0x1a, cmdId.initiator=0x45389639a448 CmdSN 0x1e8fd from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:44:28.757Z cpu8:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8c0c2c1c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:44:28.757Z cpu8:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a448 CmdSN 0x1e8fd
2021-04-26T00:44:28.757Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:44:28.757Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c2c1c0) 0x1a, cmdId.initiator=0x45389639a448 CmdSN 0x1e8fd from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:44:28.757Z cpu8:2097360)Queued:1
2021-04-26T00:44:28.759Z cpu7:2101729)ALERT: hostd performance has degraded due to high system latency
2021-04-26T00:44:39.486Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e377b280) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:44:39.486Z cpu8:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:45:01.981Z cpu10:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x9e, cmdId.initiator=0x45389269a7d8 CmdSN 0x1e909 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:45:01.981Z cpu10:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:45:01.981Z cpu10:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x9e, cmdId.initiator=0x45389269a7d8 CmdSN 0x1e909 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:45:01.981Z cpu10:2097360)Queued:1
2021-04-26T00:45:04.486Z cpu2:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 2 times
2021-04-26T00:45:08.757Z cpu10:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c2c1c0) 0x1a, cmdId.initiator=0x45389639a528 CmdSN 0x1e90a from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:08.757Z cpu10:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8c0c2c1c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:45:08.757Z cpu10:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a528 CmdSN 0x1e90a
2021-04-26T00:45:08.757Z cpu10:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:45:08.757Z cpu10:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c2c1c0) 0x1a, cmdId.initiator=0x45389639a528 CmdSN 0x1e90a from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:45:08.757Z cpu10:2097360)Queued:2
2021-04-26T00:45:12.487Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e37ece80) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:12.487Z cpu11:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:45:23.490Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3626380) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:23.490Z cpu15:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:45:41.981Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x25, cmdId.initiator=0x45389269a888 CmdSN 0x1e90d from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:45:41.981Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:45:41.981Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x25, cmdId.initiator=0x45389269a888 CmdSN 0x1e90d from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:45:41.981Z cpu11:2097360)Queued:2
2021-04-26T00:45:44.488Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e71efac0) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e912 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:44.488Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:45:44.488Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e71efac0) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e912 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:45:44.488Z cpu11:2097360)Queued:2
2021-04-26T00:45:44.488Z cpu2:2097341)ScsiVmas: 1057: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L0 failed with error Timeout
2021-04-26T00:45:44.491Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d00ec0) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:48.759Z cpu9:2097195)ScsiDeviceIO: 4315: Cmd(0x45d8c0c2c1c0) 0x28, cmdId.initiator=0x4305119491c0 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer. Cmd count Active:1
2021-04-26T00:45:48.759Z cpu9:2097195)Queued:1
2021-04-26T00:45:48.759Z cpu9:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:45:48.759Z cpu9:2100597)LVM: 6192: Could not open device mpx.vmhba32:C0:T0:L0:7, vol [60724578-09db3a26-f7cb-000af7abb22c, 60724578-09db3a26-f7cb-000af7abb22c, 1]: Timeout
2021-04-26T00:45:48.760Z cpu12:2100597)Vol3: 2129: Could not open device 'mpx.vmhba32:C0:T0:L0:7' for volume open: Not found
2021-04-26T00:45:48.760Z cpu12:2100597)Vol3: 4339: Failed to get object 28 type 1 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 0 gen 0 :Not found
2021-04-26T00:45:48.760Z cpu12:2100597)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 6072457a 332348ce a005c7a 2cb2abf7 0 0 0 0 0 0 0
2021-04-26T00:45:48.760Z cpu12:2100597)Vol3: 4339: Failed to get object 28 type 2 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 4 gen 1 :Not found
2021-04-26T00:45:48.796Z cpu11:2100351 opID=6e7a8f17)World: 11986: VC opID HB-SpecSync-host-15@1251066-79bed989-9b-ec66 maps to vmkernel opID 6e7a8f17
2021-04-26T00:45:48.796Z cpu11:2100351 opID=6e7a8f17)Config: 716: "HostLocalSwapDirEnabled" = 0, Old Value: 0, (Status: 0x0)
2021-04-26T00:45:52.486Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3657480) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:45:52.486Z cpu8:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:46:02.490Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3639480) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:46:02.490Z cpu15:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:46:04.485Z cpu4:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 7 times
2021-04-26T00:46:21.981Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x1a, cmdId.initiator=0x45389269a7c8 CmdSN 0x1e913 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:46:21.981Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8c0c19fc0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:46:21.981Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389269a7c8 CmdSN 0x1e913
2021-04-26T00:46:21.981Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:46:21.981Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x1a, cmdId.initiator=0x45389269a7c8 CmdSN 0x1e913 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:46:21.981Z cpu11:2097360)Queued:2
2021-04-26T00:46:24.489Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e71efac0) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e914 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:46:24.489Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:46:24.489Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e71efac0) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e914 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:46:24.489Z cpu11:2097360)Queued:2
2021-04-26T00:46:24.493Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d00ec0) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:46:28.761Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d414c0) 0x9e, cmdId.initiator=0x45389639a3a8 CmdSN 0x1e915 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:46:28.761Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d414c0) 0x9e, cmdId.initiator=0x45389639a3a8 CmdSN 0x1e915 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:46:28.761Z cpu8:2097360)Queued:1
2021-04-26T00:46:32.490Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0cad2c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:46:32.490Z cpu11:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:46:43.489Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3630880) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:46:43.490Z cpu15:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:47:01.981Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x1a, cmdId.initiator=0x45389269a8a8 CmdSN 0x1e91a from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:47:01.981Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:47:01.981Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x1a, cmdId.initiator=0x45389269a8a8 CmdSN 0x1e91a from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:47:01.981Z cpu11:2097360)Queued:1
2021-04-26T00:47:04.487Z cpu5:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 6 times
2021-04-26T00:47:08.761Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d414c0) 0x25, cmdId.initiator=0x45389639a458 CmdSN 0x1e91f from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:47:08.761Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x25 (0x45d8c0d414c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:47:08.761Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a458 CmdSN 0x1e91f
2021-04-26T00:47:08.761Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:47:08.761Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d414c0) 0x25, cmdId.initiator=0x45389639a458 CmdSN 0x1e91f from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:47:08.761Z cpu11:2097360)Queued:1
2021-04-26T00:47:12.486Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3694d80) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:47:12.486Z cpu9:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:47:23.486Z cpu1:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3648480) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:47:23.487Z cpu8:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:47:41.982Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x85, cmdId.initiator=0x4305118ff740 CmdSN 0x5e1 from world 2100065 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:47:41.982Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:47:41.982Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x85, cmdId.initiator=0x4305118ff740 CmdSN 0x5e1 from world 2100065 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:47:41.983Z cpu11:2097360)Queued:1
2021-04-26T00:47:48.761Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d414c0) 0x1a, cmdId.initiator=0x45389639a398 CmdSN 0x1e920 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:47:48.761Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:47:48.761Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d414c0) 0x1a, cmdId.initiator=0x45389639a398 CmdSN 0x1e920 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:47:48.761Z cpu11:2097360)Queued:1
2021-04-26T00:47:52.488Z cpu5:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e368e480) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:47:52.488Z cpu11:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:48:03.489Z cpu5:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0cd57c0) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:48:03.489Z cpu15:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:48:04.487Z cpu7:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 6 times
2021-04-26T00:48:21.985Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x85, cmdId.initiator=0x4305118ff740 CmdSN 0x5e2 from world 2100065 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:48:21.985Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x85 (0x45d8c0c19fc0, 2100065) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:48:21.985Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4305118ff740 CmdSN 0x5e2
2021-04-26T00:48:21.985Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:48:21.985Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c19fc0) 0x85, cmdId.initiator=0x4305118ff740 CmdSN 0x5e2 from world 2100065 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:48:21.985Z cpu11:2097360)Queued:1
2021-04-26T00:48:21.992Z cpu10:2097746)ScsiDeviceIO: 4298: Cmd(0x45d8c0c19fc0) 0x85, CmdSN 0x5e5 from world 2100065 to dev "naa.61866da0a3e72500280513c4a60ad036" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0
2021-04-26T00:48:28.761Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d414c0) 0x1a, cmdId.initiator=0x45389639a478 CmdSN 0x1e921 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:48:28.761Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:48:28.761Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d414c0) 0x1a, cmdId.initiator=0x45389639a478 CmdSN 0x1e921 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:48:28.761Z cpu9:2097360)Queued:0
2021-04-26T00:48:32.489Z cpu7:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c19fc0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:48:32.489Z cpu9:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:48:43.488Z cpu7:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3711880) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:48:43.488Z cpu10:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:49:04.487Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 3 times
2021-04-26T00:49:08.761Z cpu11:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d414c0) 0x28, cmdId.initiator=0x430511941d80 CmdSN 0x1 from world 2100012 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:49:08.761Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x28 (0x45d8c0d414c0, 2100012) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:49:08.761Z cpu11:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430511941d80 CmdSN 0x1
2021-04-26T00:49:08.761Z cpu11:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:49:08.762Z cpu11:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d414c0) 0x28, cmdId.initiator=0x430511941d80 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:49:08.762Z cpu11:2097360)Queued:0
2021-04-26T00:49:08.762Z cpu8:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:49:19.487Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c497c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:49:19.487Z cpu8:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:49:48.762Z cpu8:2097194)ScsiDeviceIO: 4315: Cmd(0x45d8c0d37ec0) 0x9e, cmdId.initiator=0x45389639a3a8 CmdSN 0x1e930 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from device layer. Cmd count Active:1
2021-04-26T00:49:48.762Z cpu8:2097194)Queued:0
2021-04-26T00:49:48.762Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c023c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e92f from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:49:48.762Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:49:48.762Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0c023c0) 0x35, cmdId.initiator=0x45388621be18 CmdSN 0x1e92f from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:49:48.762Z cpu8:2097360)Queued:1
2021-04-26T00:49:48.762Z cpu11:2097348)ScsiDevice: 979: Could not flush cache of local device mpx.vmhba32:C0:T0:L0. Failure
2021-04-26T00:49:59.487Z cpu2:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e363da80) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:49:59.488Z cpu15:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:50:04.486Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 3 times
2021-04-26T00:50:14.491Z cpu2:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e3734880) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:50:14.491Z cpu2:2097361)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x0 (0x45d8e3734880, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:50:14.491Z cpu2:2097361)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x45389f39bc58 CmdSN 0x0
2021-04-26T00:50:14.491Z cpu9:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:50:28.762Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d37ec0) 0x25, cmdId.initiator=0x45389639a458 CmdSN 0x1e931 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:50:28.762Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:50:28.762Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d37ec0) 0x25, cmdId.initiator=0x45389639a458 CmdSN 0x1e931 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:50:28.762Z cpu9:2097360)Queued:1
2021-04-26T00:50:39.487Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e369bb80) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:50:39.487Z cpu8:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:50:44.490Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e202d040) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e936 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:50:44.490Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e36dfd80) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:50:44.490Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:50:44.490Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e202d040) 0x12, cmdId.initiator=0x453885e9bcb8 CmdSN 0x1e936 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:50:44.490Z cpu9:2097360)Queued:1
2021-04-26T00:50:44.490Z cpu4:2097341)ScsiVmas: 1057: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L0 failed with error Timeout
2021-04-26T00:50:55.489Z cpu6:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c78fc0) 0x0, cmdId.initiator=0x4538a099bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:50:55.489Z cpu8:2111251)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:51:04.486Z cpu0:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 5 times
2021-04-26T00:51:08.762Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d37ec0) 0x1a, cmdId.initiator=0x45389639a398 CmdSN 0x1e937 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:51:08.762Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x1a (0x45d8c0d37ec0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:51:08.762Z cpu9:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x45389639a398 CmdSN 0x1e937
2021-04-26T00:51:08.762Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:51:08.762Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d37ec0) 0x1a, cmdId.initiator=0x45389639a398 CmdSN 0x1e937 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:51:08.762Z cpu9:2097360)Queued:1
2021-04-26T00:51:19.488Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d658c0) 0x0, cmdId.initiator=0x453891e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-04-26T00:51:19.488Z cpu15:2111249)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:51:24.491Z cpu3:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8e36dfd80) 0xa0, cmdId.initiator=0x453885d9b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:51:24.492Z cpu9:2097360)ScsiPath: 8058: Cancelled Cmd(0x45b8e202d040) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e938 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:51:24.492Z cpu9:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:51:24.492Z cpu9:2097360)ScsiDeviceIO: 4315: Cmd(0x45b8e202d040) 0x12, cmdId.initiator=0x453885e9bcc8 CmdSN 0x1e938 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:51:24.492Z cpu9:2097360)Queued:1
2021-04-26T00:51:35.487Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0c3e8c0) 0x0, cmdId.initiator=0x45389f39bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:51:35.487Z cpu8:2111250)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:51:48.764Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d37ec0) 0x1a, cmdId.initiator=0x45389639a478 CmdSN 0x1e939 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:51:48.764Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:51:48.764Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d37ec0) 0x1a, cmdId.initiator=0x45389639a478 CmdSN 0x1e939 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:51:48.764Z cpu8:2097360)Queued:0
2021-04-26T00:51:59.489Z cpu0:2097361)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d6fdc0) 0x0, cmdId.initiator=0x453885e1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-26T00:51:59.489Z cpu10:2097340)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-04-26T00:52:04.487Z cpu1:2097624)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 6 times
2021-04-26T00:52:28.764Z cpu8:2097360)ScsiPath: 8058: Cancelled Cmd(0x45d8c0d37ec0) 0x28, cmdId.initiator=0x430511963440 CmdSN 0x1 from world 2100012 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:0.
2021-04-26T00:52:28.764Z cpu8:2097360)NMP: nmp_ThrottleLogForDevice:3861: Cmd 0x28 (0x45d8c0d37ec0, 2100012) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-04-26T00:52:28.764Z cpu8:2097360)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430511963440 CmdSN 0x1
2021-04-26T00:52:28.764Z cpu8:2097360)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-04-26T00:52:28.764Z cpu8:2097360)ScsiDeviceIO: 4315: Cmd(0x45d8c0d37ec0) 0x28, cmdId.initiator=0x430511963440 CmdSN 0x1 from world 2100012 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-26T00:52:28.764Z cpu8:2097360)Queued:0
2021-04-26T00:52:28.764Z cpu8:2100597)LVM: 6817: Forcing APD unregistration of devID 60724579-9ffb2d25-6912-000af7abb22c in state 1.
2021-04-26T00:52:28.764Z cpu8:2100597)LVM: 6192: Could not open device mpx.vmhba32:C0:T0:L0:7, vol [60724578-09db3a26-f7cb-000af7abb22c, 60724578-09db3a26-f7cb-000af7abb22c, 1]: Timeout
2021-04-26T00:52:28.765Z cpu8:2100597)Vol3: 2129: Could not open device 'mpx.vmhba32:C0:T0:L0:7' for volume open: Not found
2021-04-26T00:52:28.765Z cpu8:2100597)Vol3: 4339: Failed to get object 28 type 1 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 0 gen 0 :Not found
2021-04-26T00:52:28.765Z cpu8:2100597)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 6072457a 332348ce a005c7a 2cb2abf7 0 0 0 0 0 0 0
2021-04-26T00:52:28.765Z cpu8:2100597)Vol3: 4339: Failed to get object 28 type 2 uuid 6072457a-332348ce-5c7a-000af7abb22c FD 4 gen 1 :Not found
Reply
0 Kudos
10 Replies
batkite972
Contributor
Contributor

Hi,

Same situation here...

Nothing new on your end ?

 

Thanks.

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

Your massive log dump should either be added as a zipped attachment, or pasted into a “spoiler” which is the triangle icon on the toolbar with an exclamation mark.

Spoiler
Here is the result, MUCH nicer to scroll through a thread!

-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
PatrickDLong
Enthusiast
Enthusiast

Welcome to the "ESXi 7.0 U2a REALLY does NOT like your SD card boot media" party - this issue is 100% because ESXi has disconnected the filesystem of your SC card media, as evidenced by your vmkernel.log entries indicating timeouts to device mpx.vmhba32:C0:T0:L0  Let's call this Issue #1 - Read on!

This is a SEPARATE issue from Issue #2 with ESXi 7.x and  USB-based (which is classified as 'low-endurance' ) devices - including SD-Card, which is hanging off your motherboard's USB hub - where in the course of normal operations ESXi 7 is corrupting that low-endurance boot media due to higher-than-previous-releases I/O to those devices - in fact ESXi 7 removed throttling of I/O to those devices altogether. See Issue #2 References below.

YOUR Issue #1 is different, however - but it is likely caused by the vmkusb driver ( 0.1-1vmw.702.0.0.17867351) included in the custom ESXi image (and in fact ALL released 7.0 U2a images from any vendor including the vanilla image from VMware.)  Credit where it is due, I learned about this from excellent blog post here:  https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/#comment...

Essentially you can most likely regain control of your host and return it to normal responsiveness by issuing esxcfg-rescan -d vmhba32 one or more times (wait a few minutes between) until the command completes without error.  This removes the dead paths to the vmhba32 device.  Then issue esxcfg-rescan vmhba32 command and the filesystem on your SD card should be rediscovered;  verify by running ls -n and make sure nothing appears in red. - your host should now be back to normal responsiveness.  You may have to restart management agents:  /etc/init.d/hostd restart    and   /etc/init.d/vpxa restart .  Evacuate your host of running vm's and proceed to mitigation options below, then reboot the host. I've also seen it referenced that stopping the USB Arbitrator service can have a positive impact in resolving this issue (obvs this removes ability to map host USB to vm's, so...)

 

Known mitigation options at this time include:

1 - Roll back to prior release if you can to avoid the issue altogether until it is fully understood and resolved.

2 - make sure your scratch is pointed to a high-endurance media like the DAS connected to your Perc or your PCIe SSD cards (you should be doing this anyway if using SD or USB boot media, regardless of ESXi version)

3 - enable  /UserVars/ToolsRamdisk to minimize I/O related to host-based VMTools actions from hitting your SD card. This seems to make this issue re-occur less frequently and in some cases not re-occur at all...yet.  But it's usefulness as a long-term solution are not proven at this time.

4 - Stop USB Arbitrator service after boot if you do not need to pass through host USB to vm's:  /etc/init.d/usbarbitrator stop

Or, you can wait for VMware to release a specific fix for this issue in the form of a new vmkusb vib, Some have reported getting a new debugging vmkusb driver 0.1-2vmw.702.0.20.45179358 from VMware  but I have not had luck getting this from GSS to date.

Or wait for U3 sometime in August which *should* contain the fix for this issue.

 

Issue #1 References:

https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/#comment...

https://www.dell.com/community/PowerEdge-Hardware-General/VMware-7-0-U2-losing-contact-with-SD-card/...

https://www.reddit.com/r/vmware/comments/napgvr/fyi_vmkusb_is_buggy_in_7x_local_storage_failure/

https://communities.vmware.com/t5/ESXi-Discussions/Issues-with-vmkusb-on-7-0U1c-with-SD-Card-boot/m-...

 

Issue #2 References

https://kb.vmware.com/s/article/83376 - VMFS-L Locker partition corruption on SD cards in ESXi 7.0

https://kb.vmware.com/s/article/2149257 - High frequency of read operations on VMware Tools image may cause SD card corruption

 

Background references for ESXi7.x boot device storage changes you should be aware of:

https://blogs.vmware.com/vsphere/2020/05/vsphere-7-esxi-system-storage-changes.html

https://blogs.vmware.com/vsphere/2020/07/vsphere-7-system-storage-when-upgrading.html

https://kb.vmware.com/s/article/2145210 - vSphere SSD and Flash Device Support

https://kb.vmware.com/s/article/2004784 - Installing ESXi on a supported USB flash drive or SD flash card

 

 

 

 

Reply
0 Kudos
batkite972
Contributor
Contributor

Thank you PatrickDLong

Couldn't be more precise... and I had the same symptoms as described.

Might revert to ESXi 6.7 until VMware realize that they are going to have 10 000 calls per day if they don't take this seriously very soon.

Can't believe they do not realize how many clients are running ESXi on SD card as it's only a boot media... well at least it was...

Tags (1)
Reply
0 Kudos
PatrickDLong
Enthusiast
Enthusiast

@batkite972  True - I've spent the last decade at two different employers getting rid of spinning rust which was my #1 failure point in both ESXi hosts and SAN arrays and using on-board SD or micro-SD cards as boot media.  Now VMware changes tack and "prefers" high-endurance boot media.  "Just add a pair of redundant SSD's or M.2 devices to your hosts" they say.  Ha! who's paying for THAT in a 200+ host environment? Maybe I'll just deduct that cost from the next licensing renewal quote I get before paying it;-) There is absolutely no reason that ESXi boot devices shouldn't continue to be used in the manner they always have been - read-once at boot time or when backing up the config and only written to for hypervisor patches and upgrades and config changes/restores.  All other I/O should be redirectable to other more high-endurance media like local storage if you have it or SAN arrays.

Tags (1)
Reply
0 Kudos
MarvinHuffaker
Contributor
Contributor

Thank you, scott28t, I fixed the spoiler thing. I didn't know that was an option 🙂

Reply
0 Kudos
MarvinHuffaker
Contributor
Contributor

So I did a clean reinstall with a newer 7.0.2 media per some advice from another forum post I found.  It helped a little, meaning it now takes 7-8 days to manifest the issue where before it was 3-4 days.

If this issue is actually the SDRAM card is not able to handle it, how are these servers even VMware certified with this version?  

I appreciate the explanations provided, and if it's all true, this is extremely frustrating. 

I'll report back once I do some testing with the commands next time the system fails. 

Marvin

Reply
0 Kudos
PatrickDLong
Enthusiast
Enthusiast

@MarvinHuffakerIt is unclear from  your post - after installing 7.0.2 cleanly (I assume using the latest  custom Dell .iso " VMware-VMvisor-Installer-7.0.0.update02-17867351.x86_64-DellEMC_Customized-A03.iso" Release Date: 2021-05-27, yes?) did you perform any of the recommended steps to minimize I/O to your SD card boot device listed a few posts up or did you just let it run without redirecting scratch and productLocker?   I have done those mitigations (I'm unable to rollback without a reinstall due to subsequent .vib installations) on my HPE servers -which are a mix of Gen9 blades and Gen10 Synergy compute modules.  All use HPE-branded quality micro-SD cards and I still see very sporadic I/O hitting the device in esxtop and I have not seen the issue in nearly a month;  my impression is that this sporadic I/O is likely heartbeating or keepalive based on the small volume and infrequency at which I see it. 

This is a helpful post with details about SD card ratings  https://communities.vmware.com/t5/ESXi-Discussions/SD-Boot-issue-Solution-in-7-x/m-p/2852027/highlig...

...and a link to supported media http://partnerweb.vmware.com/programs/server_docs/Approved%20Flash%20Devices.pdf  - AFAIK, this information is not searchable on the VMware Compatibility Guide.

 

Reply
0 Kudos
lukaslang
Enthusiast
Enthusiast

Well, HPE lists their 32GB SD Cards as supported for vSphere 7, but who knows if they calculated this through with the behavior of U2.

Redirecting Logs, Scratch, Logdump and VMware Tools to SAN seems to run fine (at the moment).

From what I've seen, an Upgrade to 7.0 U2a cannot be considered. We had so many strange issues upgrading from 6.7 EP15 to U2a that we ditched the upgrade and are going straight to fresh installs.

Issues we have seen so far:

- Hosts are hanging on vmw_satp_alua while booting even we do not have any RDM Disks

- NSX-T Transport Nodes are unable to establish a connection to the managers (and these Hosts were fresh installed with U2 and upgraded to U2a, yes I know U2 got pulled, but none of the mentioned issues "crypto64.efi" happend so there has to be a lot more)

- Remediation stuck or incomplete

All things were resolved by a fresh install. If you have time, I would go that path.

Who knows how many other issues are created in the depths of vSphere during the upgrade path that will show up in the future.

Maybe U3 will resolve many of the mentioned problems, but who knows.

I am really disappointed by this release.

Reply
0 Kudos
MarvinHuffaker
Contributor
Contributor

Yes I always move my scratch to other locations.  Otherwise I get the constant nag status in vcenter about non-persistent storage.

ProductLocker is a different story, I need to go back and check that. I've never looked at that configuration before.

Reply
0 Kudos