Zinnia_f
Contributor
Contributor

Command storcli/df hung in ESXi shell

Getting some storage performance event from one ESXi,messages as below, would like to collect Log of RAID controller,however it hung when run command "df" or  "storcli show"

"esxcli storage filesystem list" list the mount point without problems。How do I strace what it hang when run command "df" or "storcli show"?

2019-06-16T22:43:15.458Z cpu0:32815)ScsiDeviceIO: 2595: Cmd(0x43a60e45d740) 0x2a, CmdSN 0xfb from world 40433067 to dev "naa.6c00709558c783d0208980f904975886" failed H:0x8 D:0x0 P:0x0

2019-06-16T22:43:15.458Z cpu0:32815)ScsiDeviceIO: 2595: Cmd(0x43a61040ae00) 0x2a, CmdSN 0x54 from world 40433067 to dev "naa.6c00709558c783d0208980f904975886" failed H:0x8 D:0x0 P:0x0

2019-06-16T22:43:15.458Z cpu0:32815)ScsiDeviceIO: 2595: Cmd(0x43a600149d40) 0x8a, CmdSN 0x38 from world 40433067 to dev "naa.6c00709558c783d0208980f904975886" failed H:0x8 D:0x0 P:0x0

2019-06-16T22:43:15.458Z cpu0:32815)ScsiDeviceIO: 2595: Cmd(0x43a6002099c0) 0x2a, CmdSN 0x12 from world 40433067 to dev "naa.6c00709558c783d0208980f904975886" failed H:0x8 D:0x0 P:0x0

2019-06-16T22:43:15.458Z cpu0:32858)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6c00709558c783d0208980f904975886" state in doubt; requested fast path state update...

2019-06-16T22:43:15.458Z cpu0:32858)ScsiDeviceIO: 2595: Cmd(0x43a60006bac0) 0x8a, CmdSN 0xf4 from world 40433067 to dev "naa.6c00709558c783d0208980f904975886" failed H:0x8 D:0x0 P:0x0

2019-06-16T22:43:15.458Z cpu0:32858)NMP: nmp_ThrottleLogForDevice:3219: last error status from device naa.6c00709558c783d0208980f904975886 repeated 10 times

2019-06-16T22:43:15.458Z cpu26:32857)FS3Misc: 1759: Long VMFS rsv time on 'DataStore-10.100.31.74' (held for 22030 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

2019-06-16T22:43:15.458Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:254: Processing taskMgmt virt reset for device: vmhba2:C2:T1:L0

2019-06-16T22:43:15.458Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:258: VIRT_RESET cmd # 74427698

2019-06-16T22:43:15.458Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:262: ABORT

2019-06-16T22:43:16.459Z cpu0:32858)lsi_mr3: fusionWaitForOutstanding:2531: megasas: [ 0]waiting for 21 commands to complete

2019-06-16T22:43:16.639Z cpu0:19978690)HBX: 2802: 'Data-10.100.31.74': HB at offset 3424256 - Waiting for timed out HB:

2019-06-16T22:43:16.639Z cpu0:19978690)  [HB state abcdef02 offset 3424256 gen 3 stampUS 51822000695098 uuid 59f077b3-5f00d7bc-deb9-883fd3332128 jrnl <FB 14474800> drv 14.61 lockImpl 3]

2019-06-16T22:43:17.461Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:254: Processing taskMgmt virt reset for device: vmhba2:C2:T1:L0

2019-06-16T22:43:17.461Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:258: VIRT_RESET cmd # 74427698

2019-06-16T22:43:17.461Z cpu0:32858)lsi_mr3: mfi_TaskMgmt:262: ABORT

[root@SZEsxi001:~] ps -Tcjstv |head -1; ps -Tcjstv |grep storcli

WID       CID       WorldName                                                                    GID      Type    State   Wait    CPU   Time          Command

42853668  42853668  storcli                                                                      42853668  User,Native    WAIT    NOEVQ   0-31      0.141628  ./storcli show

42853669  42853668  storcli                                                                      42853668  User,Clone,Native    WAIT    UPOL    0-31      0.34      ./storcli show

42853854  42853854  grep                                                                         42853854  User,Native    WAIT    UPIPER  0-31      0.0       grep storcli

[root@SZEsxi001:~] ps -Tcjstv |head -1; ps -Tcjstv |grep df

WID       CID       WorldName                                                                    GID      Type    State   Wait    CPU   Time          Command

42852168  42852168  python                                                                       42852168  User,Native    WAIT    NOEVQ   0-31      0.364534  python /bin/df

42852169  42852168  python                                                                       42852168  User,Clone,Native    WAIT    UPOL    0-31      0.42      python /bin/df

42852567  42852567  grep                                                                         42852567  User,Native    WAIT    UPIPER  0-31      0.0       grep df

[root@SZEsxi001:~]

0 Kudos
0 Replies