ESXi 6.0U3 disk performance extremely slow

[slickoket]

Hi there

I have recently installed and setup a new installation of ESXi 6.0U3 on a PowerEdge R740 and am having issues with the disks speeds being appallingly slow.

I have made a datastore on the host and have tried it with both a RAID 5 configuration and non-RAID as well but both are very slow.  When doing an SCP transfer to the disk is usually sits around 20-50 K/s and trying to run a VM to install an OS loading the Windows 10 ISO takes over an hour.

The host is using a Lewisburg SATA AHCI Controller.

Can anyone help me troubleshoot this issue?

Thanks in advance.

[RickVerstegen]

Are you using the latest vmw_ahci driver version?

You can check by using the command:

esxcli system module get -m vmw_ahci

Can you also verify if the Native AHCI driver is disabled? You can check if its been disabled by running the following command:

esxcli system module list | grep vmw_ahci

Check the second column to see if it shows "false"

[slickoket]

Hey Rick thanks for the reply!

The first command returns an unknown module error and the second returns nothing at all.  So I'm guessing that module is not enabled or even installed.

But I have checked our other ESXi hosts that are also running 6.0U3 with normal disk performance and none of them have this driver either.

All of them seem to use a driver thats just called "ahci"

[SupreetK]

Share more details about the controller and the driver name/version it is using -

Step-1 --> Run the command <esxcfg-scsidevs -a> to list the storage adapters.

Step-2 --> Run the command <vmkchdev -l | grep -i vmhbaX> for the AHCI controller to get the hardware IDs (Replace X with the AHCI controller's number).

Step-3 --> Run the command <vmkload_mod -s driver_name | grep -i version> to get the driver version of the AHCI controller. 'driver_name' can be obtained from the second column of the output from step-1.

Step-4 --> Run the command <esxcli software vib list | grep -i ahci> to get the list of installed AHCI related drivers.

Cheers,

Supreet

[slickoket]

Hi Supreet thanks for the reply.  Please find attached a screenshot of the results of these commands:

From what I can see, this is the newest version of the AHCI driver going from this webpage: https://esxi-patches.v-front.de/vm-6.0.0.html

I think the vmw_ahci was not introduced until 6.5 and we are running 6.0U3.

Any more thoughts based on these outputs?

Thanks!

[SupreetK]

Just to confirm, the disk is being mapped via the SATA controller and not the Avago PERC controller right? Can you run the command <esxcfg-scsidevs -A> and share the output?

Cheers,

Supreet

[slickoket]

Yes it's on the SATA controller and not the PERC.  Here is the output from that command:

[SupreetK]

My doubt was right The disk is mapped via the PERC controller and not the AHCI controller. As we can see from the above output, the disk naa.6d0946.... is mapped to vmhba2 which, is the PERC controller. Can you run the below commands and share the output?

vmkchdev -l | grep -i vmhba2

vmkload_mod -s lsi_mr3 | grep -i version

Also, check the firmware version of the PERC controller from Dell iDRAC console. It should be listed under Storage --> Controllers.

Cheers,

Supreet

[slickoket]

Oh sorry didn't realise the PERC was the RAID controller.  Yes, the datastore is set up through a RAID 5 using 8 separate disks.

Here is the output from those commands:

Firmware of the PERC is 25.5.3.0005.

Thanks for your continued help!

[SupreetK · Accepted solution]

As we can see from the below link, the PERC controller driver is not on the latest version and the firmware is not on a supported version. Let us start with getting this thing sorted -

VMware Compatibility Guide - I/O Device Search​

Download the latest driver from the below link and involve your hardware vendor to get the controller firmware upgraded to the latest version. Latest firmware as per VMware HCL is 25.5.5.0005. However, you can go to any version above this. -

Download VMware vSphere

Upgrade the driver and firmware of the controller and then see if there is a difference in the performance.

Cheers,

Supreet

[slickoket]

Bingo!

Updated the driver from that link and used the lifecycle controller to update the PERC firmware and now disk and transfer speeds are vastly improved.

Thanks so much for all the help and a good learning experience too!