VMware Cloud Community
Zinnia798
Contributor
Contributor

I/O latency,ESXi 6.0 with MegaRAID controller storage

Hi Everyone,

Recently,My VMs(linux and windows) on ESXi6.0 getting slow with low disk performance, the host has gotten a lot of I/O latency warning. the response is slow even on host when checking something about volumes.  most of VM's disk are thin provisionning.

Anyone has some suggestions about this kind performance tunning?  thanks.

vmkernel.log

2018-03-03T10:09:49.873Z cpu21:33475)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x85 (0x43a600f02000, 34564) to dev "naa.6c0048945a09800821f4248c2d7e2dce" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2018-03-03T10:09:49.895Z cpu21:33475)ScsiDeviceIO: 2651: Cmd(0x43a600f02000) 0x1a, CmdSN 0x8af19 from world 0 to dev "naa.6c0048945a09800821f4264e4857a865" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-03-03T10:09:49.895Z cpu21:33475)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x85 (0x43a600f02000, 34564) to dev "naa.6c0048945a09800821f4264e4857a865" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2018-03-03T10:09:49.895Z cpu21:33475)ScsiDeviceIO: 2635: Cmd(0x43a600f02000) 0x85, CmdSN 0x3c80 from world 34564 to dev "naa.6c0048945a09800821f4264e4857a865" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2018-03-03T10:09:49.895Z cpu21:33475)ScsiDeviceIO: 2651: Cmd(0x43a600f02000) 0x4d, CmdSN 0x3c81 from world 34564 to dev "naa.6c0048945a09800821f4264e4857a865" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2018-03-03T10:09:49.895Z cpu21:33475)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x1a (0x43a600f02000, 34564) to dev "naa.6c0048945a09800821f4264e4857a865" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2018-03-03T10:09:49.895Z cpu21:33475)ScsiDeviceIO: 2651: Cmd(0x43a600f02000) 0x1a, CmdSN 0x3c82 from world 34564 to dev "naa.6c0048945a09800821f4264e4857a865" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-03-03T10:09:50.235Z cpu18:33475)ScsiDeviceIO: 2651: Cmd(0x43a600f02000) 0x1a, CmdSN 0x8af1e from world 0 to dev "naa.6c0048945a09800821f4264e4857a865" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-03-03T10:09:50.235Z cpu18:33475)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x85 (0x43a600f02000, 34564) to dev "naa.6c0048945a09800821f4264e4857a865" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2018-03-03T10:09:50.389Z cpu1:32816)ScsiDeviceIO: 1217: Device naa.6c0048945a09800821f4248c2d7e2dce performance has improved. I/O latency reduced from 16077 microseconds to 6098 microseconds.

2018-03-03T10:09:54.212Z cpu17:33475)WARNING: ScsiDeviceIO: 1243: Device naa.6c0048945a09800821f4248c2d7e2dce performance has deteriorated. I/O latency increased from average value of 3094 microseconds to 68800 microseconds.

2018-03-03T10:09:55.116Z cpu15:33474)ScsiDeviceIO: 1217: Device naa.6c0048945a09800821f4248c2d7e2dce performance has improved. I/O latency reduced from 68800 microseconds to 13682 microseconds.

2018-03-03T10:09:56.974Z cpu0:33474)ScsiDeviceIO: 1217: Device naa.6c0048945a09800821f4248c2d7e2dce performance has improved. I/O latency reduced from 13682 microseconds to 6102 microseconds.

esxtop--disk

Tags (1)
0 Kudos
1 Reply
parmarr
VMware Employee
VMware Employee

Looks like two volumes are affected - naa.6c0048945a09800821f4248c2d7e2dce and naa.6c0048945a09800821f4264e4857a865. Where are these volumes located? Local or SAN? Please review KB Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205) - https://ikb.vmware.com/s/article/1008205. Contact your Storage vendor to review Storage if these volumes are on the SAN.

Sincerely, Rahul Parmar VMware Support Moderator
0 Kudos