VMware Cloud Community
zx93
Contributor
Contributor
Jump to solution

Local RAID5; latency spikes every 5 minutes. Log: vmw_psp_fixed "This may lead to path thrashing"

I'm seeing latency spikes every 5 minutes up to almost a second in length. The timestamps match up with the "This may lead to path thrashing" in the vmkernal.log. This is with vSphere 7.0 U2 on a Dell PowerEdge R640 with a PERC H740P and Dell SAS + SATA SSD. Not sure what is causing this.

Log snippet that repeats every 5 minutes:

2021-05-20T16:18:50.020Z cpu9:1048764)ScsiVmas: 1057: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L1 failed with error Not supported
2021-05-20T16:18:50.140Z cpu15:1048763)WARNING: VMW_SATP_LOCAL: satp_local_updatePath:807: Marking path vmhba32:C0:T0:L0 as STANDBY
2021-05-20T16:18:50.140Z cpu15:1048763)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L0 in STANDBY state for device mpx.vmhba32:C0:T0:L0.
2021-05-20T16:18:50.140Z cpu15:1048763)WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L0 for device mpx.vmhba32:C0:T0:L0 to activate. This may lead to path thrashing.
2021-05-20T16:18:50.140Z cpu15:1048763)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L0 in STANDBY state for device mpx.vmhba32:C0:T0:L0.
2021-05-20T16:18:50.140Z cpu15:1048763)WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L0 for device mpx.vmhba32:C0:T0:L0 to activate. This may lead to path thrashing.
2021-05-20T16:18:50.140Z cpu15:1048763)WARNING: VMW_SATP_LOCAL: satp_local_tryToActivatePath:1415: Checking TUR on STANDBY path vmhba32:C0:T0:L0
2021-05-20T16:18:50.740Z cpu15:1048763)WARNING: VMW_SATP_LOCAL: satp_local_updatePath:807: Marking path vmhba32:C0:T0:L1 as STANDBY
2021-05-20T16:18:50.740Z cpu15:1048763)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L1 in STANDBY state for device mpx.vmhba32:C0:T0:L1.
2021-05-20T16:18:50.740Z cpu15:1048763)WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L1 for device mpx.vmhba32:C0:T0:L1 to activate. This may lead to path thrashing.
2021-05-20T16:18:50.740Z cpu15:1048763)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L1 in STANDBY state for device mpx.vmhba32:C0:T0:L1.
2021-05-20T16:18:50.740Z cpu15:1048763)WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L1 for device mpx.vmhba32:C0:T0:L1 to activate. This may lead to path thrashing.
2021-05-20T16:18:50.740Z cpu15:1048763)WARNING: VMW_SATP_LOCAL: satp_local_tryToActivatePath:1415: Checking TUR on STANDBY path vmhba32:C0:T0:L1
Reply
0 Kudos
1 Solution

Accepted Solutions
zx93
Contributor
Contributor
Jump to solution

Hi @e_espinel, thanks for the reply. The server is running Dell's manufacturer image. The mkload_mod -s Name Driver | grep Version command does not work (vmkmod: VMKModCheckMod: Access to file /usr/lib/vmware/vmkmod/Name failed: Not found), but I was able to find what I needed. The issue is the basic Performance > Disk view doesn't tell me what device is actually causing the latency. But seeing the cli output from esxcli storage core device list matched up with what's in the vmkernal.log and was just the Dell iDrac USB device.... For the Googlers, to get rid of the latency spikes in the log, I went into the Dell iDrac > Configuration > Virtual Media > Attached Media > Attach Mode > set to AutoAttach (it was previously set to Attached). The devices in VMware disappeared immediately as so did the latency spikes. I'm guessing the latency spikes were just device detection queries.

View solution in original post

Reply
0 Kudos
2 Replies
e_espinel
Virtuoso
Virtuoso
Jump to solution

Hello
It is always advisable to use the manufacturer's custom image in this case Dell.


https://www.dell.com/support/manuals/en-us/vmware-esxi-7.x/vmware_7.0_icg_pub/introduction?guid=guid...

With the following commands we can know the adapter and the driver you are using and also know the version of the driver that is loaded.

# esxcli storage filesystem list

# vmkload_mod -s Name Driver |grep Version

e_espinel_0-1621540257668.png

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43392

 

 

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
zx93
Contributor
Contributor
Jump to solution

Hi @e_espinel, thanks for the reply. The server is running Dell's manufacturer image. The mkload_mod -s Name Driver | grep Version command does not work (vmkmod: VMKModCheckMod: Access to file /usr/lib/vmware/vmkmod/Name failed: Not found), but I was able to find what I needed. The issue is the basic Performance > Disk view doesn't tell me what device is actually causing the latency. But seeing the cli output from esxcli storage core device list matched up with what's in the vmkernal.log and was just the Dell iDrac USB device.... For the Googlers, to get rid of the latency spikes in the log, I went into the Dell iDrac > Configuration > Virtual Media > Attached Media > Attach Mode > set to AutoAttach (it was previously set to Attached). The devices in VMware disappeared immediately as so did the latency spikes. I'm guessing the latency spikes were just device detection queries.

Reply
0 Kudos