VMware Cloud Community
PeterQuiring
Contributor
Contributor
‎05-16-2022 12:44 PM

disk latency suddenly spikes after running for 5 months

We have two VMWare servers (7.0) and are running VSA iSCSI VMs that mirror each other that run on each server.  Suddenly after 5 months of running the disk latency of both VSA VMs have spiked to over 200ms.  The servers are all flash. The storage for the VSAs are a RAID-5 setup.

Servers are HP ProLiant DL380 G9

The VSA is an older version of HP VirtualStorage v12.8.

The latency issue hit server 1 around May 6th, and around May 11th for server 2 (very odd that both experienced the same problem near the same time).

Any ideas what would cause this?  I'm going to shutdown and reboot servers ASAP.

Thanks,

Preview file
116 KB
Preview file
104 KB
Reply
0 Kudos
4 Replies
robc_yk
Enthusiast
Enthusiast
‎05-17-2022 09:33 AM

Has any firmware been applied to the hosts or the backend storage system? I have seen in the past a bad HBA firmware / driver combination cause latency issues.

Has the workload or number of VM's changed? Doing some math, is there enough IOPS available based on disk and VM demand?

Look at the VM's disks, is there one or more that seems to be chewing up the service time?

Reply
0 Kudos
degvm
Enthusiast
Enthusiast
‎05-17-2022 11:33 PM

Check the aging of the VMs. Doublecheck Disk Fragmentation in case of windows VM, install a second scsi adapter to have on for Operating system and second for the data disk.

Reply
0 Kudos
AnatoxinD
Contributor
Contributor
‎05-17-2022 11:56 PM

Storage performance problems is hard to troubleshoot without a good starting point, you will get millions of ideas here :).

I suggest you install some sort of performance monitor that look both from VMware's perspective and from the backed storage, that way you can start narrow it down to a read thread.

I use XorMon (Free without support) for this type of monitoring

Reply
0 Kudos
PeterQuiring
Contributor
Contributor
‎05-18-2022 05:20 AM

I think I know what the problem is.  VMWare knows its local disks are SSD so it's not a driver issue.  But these disks are used by the HP StoreVirtual VSA to create a mirrored iSCSI disk which is NOT showing as SSD to VMware.  Could be a miss-configured option within StoreVirtual.  VMWare does have a procedure to force a disk to be SSD.

https://kb.vmware.com/s/article/2013188

I'm going to try that on the weekend.  Anybody familiar with HP StoreVirtual ?

Thanks,

Reply
0 Kudos