VMware Cloud Community
Pollard
Enthusiast
Enthusiast

Esxi 6.7 ssd i/o latency spikes

Seeing latency spikes in excess of 100ms on my 2 samsung 860 evo ssd's. The throughput is fine getting 500Mbps+ on them but the vmkwarning.log has "performance has deteriorrated. I/O latency increasted from average value" warnings. This is a home server thats why I'm using regular samsung ssd's.

ESXi version: 6.7.0

ESXi building number: 15160138

Tags (1)
Reply
0 Kudos
2 Replies
Pollard
Enthusiast
Enthusiast

Capture.PNG

Reply
0 Kudos
alphenit
Enthusiast
Enthusiast

I've seen similar issues (high latency spikes of 200ms ) with even the latest 6.7U3 as of September 2020. This in combination with Samsung EVO 860 SSD.
Which driver is your sata-port using?
Have you tried disabling the vmware ahci driver? (below command and a reboot will make ESXi use the ahci driver instead of the vmw_ahci driver).
you can check this under storage adapters through the host ui. When using the ahci driver you should be seeing ALL your sata ports. (the vmw_ahci only showed me 1 or 2).

esxcli system module set --enabled=false --module="vmw_ahci"
https://anthonyspiteri.net/homelab-supermicro-5020d-tnt4-storage-driver-performance-issues-and-fix/


Also did some some dd tests after cd-ing into the specific vmfs volume but not too sure what that signifies or how to interpret the results(saw it used in another thread) and using ahci driver made the dd command finish quite a lot faster:

using vmw_ahci:

time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

655360+0 records in

655360+0 records out

real    2m 42.31s

user    0m 0.00s

sys     0m 0.00s

using ahci:

time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

655360+0 records in

655360+0 records out

real    0m 47.97s

user    0m 0.00s

sys     0m 0.00s

Wish VMware would fix those issues with vmw_ahci permanently so admins don't have to troubleshoot their own Lab hardware during the weekends.

Consider marking this response as "Correct" or "Helpful" if you think my response helped you in any way.

*Please consider awarding points if my response was helpful*
Reply
0 Kudos