VM performance issue

[ManivelR]

Hi All,

Small setup with one ESXi server/one production VM.VM is running on local data store of ESXi server(RAID 6 configured).Raid 6 configured in hardware level and created a datastore with multiple hard disks.Data store size 18.19 TB

Please note:- ESXi os is running on one local data store.VM is running on another data store which size is 18.19 TB

Only one production VM is running on ESXi server.VM has 3 VMDK and the size is around 12 TB.

We are facing performance issue and currently suspecting this performance issue is related to disk latency.Could you please help me to dig further on this case?

CPU & memory usage is under normal and seeing disk latency in periodic intervals.

ESXi version is 6.0.0 U2

Windows 2012 VM -->VMtools running(Out of date)

realtime stats:-

Disk stats:- Accepted latency should be less than 10 ms.here it is above 20 ms in regular intervals.

vdisk stats: Accepted latency should be less than 5 ms.Here it is above 15 ms in regular intervals.

Commands checked:- Device Max Queue Depth: 128.Both OS data store and VM data store queue depth is 128.

esxcli storage core device list --device naa.60030480197e

Device Max Queue Depth: 128

   No of outstanding IOs with competing worlds: 32

esxtop results:

Thanks,

Manivel R

[dbalcaraz]

Hi,

1st thing, VMtools outdated?

[daphnissov]

You said this is a RAID-6 array. How many disks and what type? Please be specific.

[ManivelR]

i hope it is 10 disks with RAID 6.The size is 18 TB local hard disk.

[daphnissov]

You "hope" it's 10 disks? Meaning you don't know? What type of disks are they? What is their capacity?

[ManivelR]

All the disks type are SATA. i could not remember each disk capacity from hardware.I can only say that all the disks are coming under RAID 6. Post that we created a data store with single 18 TB.

[daphnissov]

Without diving into more performance data (which wasn't provided), your issue is more than likely that your storage is under performing. RAID-6 is literally the worst possible mechanism for performance, especially when it comes to random writes. When you put RAID-6 on top of cheap SATA drives (5,400 or even 7,200 RPM) you exacerbate the problem even further when it comes to write performance as double parity calculations, writes, and acknowledgements must be sent back up to the hypervisor. This all manifests in high latency, which is what you're seeing presently. So your options are as follows, in order of recommendation:

1. Don't use RAID-6 and use either RAID-1 or something else.
2. Use 10K SAS drives to try and mitigate some of these random write speed issues.
3. Add more spindles.

[ManivelR]

Thanks so much.

I will check and update.

[dbalcaraz]

Couldn't agree more with daphnissov

RAID-6 with 5.4k and 7.2k disks is just horrible.

RAID-10 for example is a great raid I think, but the 10k SAS drives disks or some SSDs make the difference.

[Dave_the_Wave]

While it makes sense to use more than 1 drive for parity in an array of so many disks (10 is a lot), it would be better with old fashion RAID5 with extra hot spares.

When one drive has failed, the controller starts re-striping the data on a hot spare drive. All while the server is live and in production. By the time you physically get there on-site, the array has rebuilt itself, and all you need to do remove the bad drive and take your sweet time 1-day priming another one over.

Folks do this all the time with HP ProLiant Smart Array since the dawn of time, I'm sure something similar is avail on other hardware cards.