NVMe SSD Fast, but SATA SSD slow !

[jpbourke]

Hi,

I have a Supermicro X10SDV-8C-TLN4F (Xeon D1541 3Ghz 8 Core, 16 Threads) with 64Gb RAM running ESXI 6.5 U1.

1x SATA DOM for ESXi

4x SATA hard disks of various sizes

1x Samsung SATA SSD 850 PRO 2TB

1x Samsung NVMe SSD 960 PRO 2Tb

I am doing a write test to a 10G file.  I don't believe the results.  The NVMe seems too "good to be true" and the SATA SSD seems "too bad to be true".  I suspect that the SATA SSD is not performing.

I have done the esxcli system module set –enabled=false –module=vmw_ahci and rebooted, and I got a small performance increase.  But still not what I thought.

Test is to write a 10G file.

NVMe SSD 960 PRO 2Tb

[root@localhost:/vmfs/volumes/599bd559-ff0022f4-2abf-0cc47a7c09b4] time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

real    0m 14.32s

Samsung say "2,100MB/s Seq. Write"- ESXi disk monitor says about 1,100MB/sec.

SATA SSD 850 PRO 2TB

[root@localhost:/vmfs/volumes/5984ddf8-0e518991-7de5-0cc47a7c09b4] time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

real    5m 32.65s

Samsung say "Up to 520 MB/sec Sequential Write" - ESXi disk monitor says about 65MB/sec.

SATA Disk

[root@localhost:/vmfs/volumes/589566b4-b40e0cf2-8cac-0cc47a7c09b4] time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

real    13m 29.23s

So are these numbers real ?

Thanks

john

[roman79]

Hi jpbourke​,

There might be different reasons why a sequential write speed is at 65MB/sec.

Anthony Spitery published a blog post about similar issue with a SATA controller on the Supermicro motherboard and poor SSD performance - https://anthonyspiteri.net/homelab-supermicro-5020d-tnt4-storage-driver-performance-issues-and-fix/.

Following Anthony's article, Paul Braren did a really good job testing different system configurations / drivers / etc. - Easy fix for Supermicro and other Xeon D systems experiencing SATA3/AHCI slowdown on ESXi 6.5 | Tink....

Assuming your system is on the latest BIOS / firmware and ESXi 6.5 U1, I would recommend reaching out those two people with your test results.

Hope you find this information helpful.

[jpbourke]

Quick update,

I believe I have the right driver now.  I'll look at the BIOS later today

[jpbourke]

All changes with ESXi 6.,5 U1

# esxcli system version get

   Product: VMware ESXi

   Version: 6.5.0

   Build: Releasebuild-5969303

   Update: 1

   Patch: 26

# esxcli software vib list | grep ahci

sata-ahci                      3.0-26vmw.650.1.26.5969303            VMW     VMwareCertified   2017-08-22 

vmw-ahci                       1.0.0-39vmw.650.1.26.5969303          VMW     VMwareCertified   2017-08-22

# esxcli storage core path list

(edited)

sata.vmhba0-sata.0:5-t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____

   UID: sata.vmhba0-sata.0:5-t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____

   Runtime Name: vmhba0:C0:T5:L0

   Device: t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____

   Device Display Name: Local ATA Disk (t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____)

   Adapter: vmhba0

   Channel: 0

   Target: 5

   LUN: 0

   Plugin: NMP

   State: active

   Transport: sata

   Adapter Identifier: sata.vmhba0

   Target Identifier: sata.0:5

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 33554432

# esxcli system module set --enabled=false --module="vmw_ahci"

# reboot

After reboot

# esxcli system module list | grep ahci

ahci                                true        true

vmw_ahci                           false       false

# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

655360+0 records in

655360+0 records out

real    6m 20.89s

user    0m 0.00s

sys     0m 0.00s

# esxcli system module set --enabled=true --module=vmw_ahci

#reboot

After reboot

# esxcli system module list | grep ahci

ahci                                true        true

# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"

655360+0 records in

655360+0 records out

real    5m 20.66s

user    0m 0.00s

sys     0m 0.00s

[roman79]

Hi jpbourke​,

Thank you for sharing this data.

It looks like your environment is on the latest version.

VMware confirmed that an issue with poor performance of the native driver (vmw-ahci) has been resolved in vSphere 6.5 U1 - http://www.virtuallyghetto.com/2017/07/ahci-vmw_ahci-performance-issue-resolved-in-esxi-6-5-update-1....

However, the results of your tests show only 32MB/s sequential write. Whereas, independent tests in Windows OS show approximately 500MB/s sequential write - Sequential Performance - The 2TB Samsung 850 Pro & EVO SSD Review and Samsung 850 PRO SSD 2TB Review | StorageReview.com - Storage Reviews.

The only difference I see comparing your results is that you use 16K blocks instead of 4K ones. Is it any particular reason to do this? Can you run a new test with 4K blocks?

Regards,