ESXi 6.0.0U3 very slow disk i/o

[WalkingWounded]

I have ESXi 6.0.0U3 installed on an HP Proliant ML310e Gen 8 V2 server (server model is listed as a supported model for this version). I have observed that disk throughput is very poor but I have not been able to determine a course of action to correct the issue.

Throughput seems to be less than 10MB/s for local file copies (in a windows VM file copy). It 'feels' like the disk read/write cache is not on although this is likely not the issue. ESXi in not reporting high demands for i/o, it is just 'poor'.

Two SATA disks are attached to the B120i controller as a mirror. ESXi web interface is not flagging any errors related to the disks (although the embedded flash card device has failed it is not in use). There are no abnormal sounds coming from the server.

Firmware versions (from iLO):

Firmware Version Info
Dynamic Smart Array B120i Controller, 5.00, Embedded
HP Ethernet 1Gb 2-port 332i Adapter 5720-v1.39 NCSI, v1.3.16.0 ,Embedded
iLO, 2.76 Oct 31 2020, System Board
Intelligent Platform Abstraction Data, 3.02, System Board
Intelligent Provisioning N/A System Board
Redundant System ROM P78, 03/28/2014, System Board
Server Platform Services (SPS) Firmware, 3.0.5.80.C, System Board
System Programmable Logic Device, Version 0x05, System Board
System ROM, P78 03/28/2014, System Board
System ROM Bootblock, 02/04/2012, System Board

VMware package for the controller (hpvsa):

Hewlet-Packard scsi-hpvsa 5.5.0.100-1OEM.550.0.0.1331820 2015-06-26 20:19:33.000

Any guidance on how to diagnose this would be appreciated. You may safely presume I have very limited experience managing ESXi.

[cac3a]

How are you measuring the throughput ? Is is it on the host itself or VMs ?

[WalkingWounded]

VM: a VM running Server 2012 R2 Essentials, with current updates.

What was done was to copy a few GB of files from one folder to another. That said, I also have seen poor write performance moving files to/from this VM over the network. VM has the vmware tools installed for this version.

It seems to me a disk to same disk file copy should be faster than 10mb/s.

The VM was not being used at the time of the file copy. This is the only VM on this box.

[WalkingWounded]

I ran Crystaldiskmark on the system today (to try to get a reproducible measurement), I attached it here.

The results seem to be in line with what I should expect from spinning rust. I will try a different test shortly (file copy), which I expect to be about half these numbers.

[WalkingWounded]

Used robocopy to do 2 tests: 1GB of random files, 2GB of large files. (these were just some folders I found with enough data or large files to do a test with).

Test 1: directory of about 1GB of files, random sizes

------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras
Dirs : 28 27 0 0 0 0
Files : 100 100 0 0 0 0
Bytes : 1.074 g 1.074 g 0 0 0 0
Times : 0:00:27 0:00:27 0:00:00 0:00:00

Speed : 42274932 Bytes/sec.
Speed : 2418.991 MegaBytes/min.
Ended : May 24, 2021 10:17:14 AM


Test 2: directory with 4 large files totaling about 2GB
------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 1 0 0 0 0
Files : 4 4 0 0 0 0
Bytes : 1.928 g 1.928 g 0 0 0 0
Times : 0:00:49 0:00:49 0:00:00 0:00:00

Speed : 41617436 Bytes/sec.
Speed : 2381.368 MegaBytes/min.
Ended : May 24, 2021 10:19:19 AM

The results surprised me, they about what one might expect for sata drives. I did an addition test on the first folder using windows explorer and di notice the transfer graph was inconsistent. Image attached. The transfer rate swings from about 50mb/s to 10mb/s.

[WalkingWounded]

For the large files the graph is smoother, but also has large dips.

Note: no one was using the server at the time of these tests (idle).

I may be barking up the wrong tree based on these tests. I will try to perform a network file copy later today. All I can say is I have observed poor throughput recently, but it may be inconsistent.

[WalkingWounded]

Sorry about all those typos. I hope it still makes sense.

[WalkingWounded]

I made some adjustments to the antivirus program, it may help in the future. I am abandoning this post at this time however as I cannot reliably reproduce the problem (may be subjective or situational).