VMware Cloud Community
logan892
Contributor
Contributor

LSI SAS 2008 RAID 1 always thrashing

I'm attempting to use an LSI 9212-4i4e in IR mode with ESXi 6.0. I have configured two Western Digital RE 320 GB drives in RAID 1.

Firmware on the controller is P20, and I also loaded the P20 drives from the website.

There is no on-board cache on this controller.

Drive activity lights were blinking while the drives were initializing, and of course write performance was horrendous.

Writing anything caused the drives to thrash, and sequential writing speeds (testing with dd) reached no more than 2 MB/s.

After several hours (drives left to their initialization over night) the initialization appears to have finished and the drive activity has quieted down. Drive activity lights are no longer constantly blinking, and remain mostly quiet unless I initiate a write myself.

However, the sequential write performance is still absolutely terrible. Write speed has only doubled compared with when the drives were initializing, reaching approximately 4-5 MB/s.

The drives are still thrashing constantly when writing to the volume.

I've tried to remove and recreate the vmfs volume. The result is the same.

Writing 150 MB (from /dev/zero) takes 30+ seconds. Reading back the same 150 MB takes 4 seconds.

Writes: 4.5 MB/s

Writes: 37 MB/s

[root@esxi6:~] date; time dd if=/dev/zero of=/vmfs/volumes/vmstore2_raid1_wd_re/out.tmp bs=1048576 count=150

Tue Mar 22 10:04:45 UTC 2016

150+0 records in

150+0 records out

real    0m 33.20s

user    0m 0.08s

sys     0m 0.00s

[root@esxi6:~] date; time dd of=/dev/null if=/vmfs/volumes/vmstore2_raid1_wd_re/out.tmp bs=1048576 count=150

Tue Mar 22 10:06:23 UTC 2016

150+0 records in

150+0 records out

real    0m 4.05s

user    0m 0.10s

sys     0m 0.00s

[root@esxi6:~] date; time dd if=/dev/zero of=/vmfs/volumes/vmstore2_raid1_wd

_re/out.tmp bs=1048576 count=300

Tue Mar 22 10:07:35 UTC 2016

300+0 records in

300+0 records out

real    1m 12.24s

user    0m 0.15s

sys     0m 0.00s

[root@esxi6:~] date; time dd of=/dev/null if=/vmfs/volumes/vmstore2_raid1_wd

_re/out.tmp bs=1048576 count=300

Tue Mar 22 10:08:55 UTC 2016

300+0 records in

300+0 records out

real    0m 7.63s

user    0m 0.22s

sys     0m 0.00s

cmd: esxcli storage core device list

naa.600508e00000000071e6339d7559b40d

   Display Name: LSI Serial Attached SCSI Disk (naa.600508e00000000071e6339d7559b40d)

   Has Settable Display Name: true

   Size: 304222

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.600508e00000000071e6339d7559b40d

   Vendor: LSI

   Model: Logical Volume

   Revision: 3000

   SCSI Level: 6

   Is Pseudo: false

   Status: degraded

   Is RDM Capable: true

   Is Local: false

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unknown

   Other UIDs: vml.0200000000600508e00000000071e6339d7559b40d4c6f67696361

   Is Shared Clusterwide: true

   Is Local SAS Device: false

   Is SAS: true

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 128

   No of outstanding IOs with competing worlds: 32

   Drive Type: logical

   RAID Level: RAID1

   Number of Physical Drives: 2

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

[root@esxi6:~] esxcli storage core device raid list -d naa.600508e0000000007

1e6339d7559b40d

   Physical Location: enclosure 0, slot 6

   Physical Location: enclosure 0, slot 7

cmd: esxcli storage core device vaai status get

naa.600508e00000000071e6339d7559b40d

   VAAI Plugin Name:

   ATS Status: unsupported

   Clone Status: unsupported

   Zero Status: supported

   Delete Status: unsupported

[root@esxi6:~] esxcli storage core device partition list

Device                                                                      Partition  Start Sector  End Sector  Type           Size

--------------------------------------------------------------------------  ---------  ------------  ----------  ----  -------------

t10.ATA_____WDC_WD1003FBYX2D50Y7B1________________________WD2DWCAW36KXSCNY          0             0  1953525168     0  1000204886016

t10.ATA_____WDC_WD1003FBYX2D50Y7B1________________________WD2DWCAW36KXSCNY          1          2048  1953523712    fb  1000203091968

mpx.vmhba33:C0:T0:L0                                                                0             0    61457664     0    31466323968

mpx.vmhba33:C0:T0:L0                                                                1            64        8192     0        4161536

mpx.vmhba33:C0:T0:L0                                                                5          8224      520192     6      262127616

mpx.vmhba33:C0:T0:L0                                                                6        520224     1032192     6      262127616

mpx.vmhba33:C0:T0:L0                                                                7       1032224     1257472    fc      115326976

mpx.vmhba33:C0:T0:L0                                                                8       1257504     1843200     6      299876352

mpx.vmhba33:C0:T0:L0                                                                9       1843200     7086080    fc     2684354560

t10.ATA_____Samsung_SSD_850_PRO_256GB_______________S1SUNSAG346333H_____            0             0   500118192     0   256060514304

t10.ATA_____Samsung_SSD_850_PRO_256GB_______________S1SUNSAG346333H_____            1          2048   419432448    fb   214748364800

naa.600508e00000000071e6339d7559b40d                                                0             0   623046656     0   318999887872

naa.600508e00000000071e6339d7559b40d                                                1          2048   623046623    fb   318998822400

My main VM store is currently on a single Samsung SSD, connected to the internal Intel controller. While performance is not stellar here either, it is able to reach above 100 MB/s writes which is much more acceptable.

What may be causing this constant HDD seek/thrashing while writing to the RAID 1 array, and is there a way to mitigate it in ESXi 6.0?

Should I try downgrading the firmware to P19 and also reverting the VMware drivers to P19?

Can I activate or deactivate some VMFS related features?

Can I from ESXi modify the drive cache behavior?

Am I s**t-out-of-luck with seeing any decent write performance using this controller, while retaining some semblance of data integrity in the event of power failure, for example?

Reply
0 Kudos
3 Replies
JarryG
Expert
Expert

There is no on-board cache on this controller.

^ This is your problem: you are trying to use controller which is unsuitable for ESXi. Basically, you are running your disks un-cached, because ESXi (unlike any other modern OS) does *NOT* do disk-caching.

What may be causing this constant HDD seek/thrashing while writing to the RAID 1 array

When you write to raid-1 array, controller must write data-chunk to both drives, then read them back (from both disks), calculate crc and compare it (before going on to next data-chunk). This is trully terrible without cache.

...and is there a way to mitigate it in ESXi 6.0?

Yes, there is one: buy some serious raid-controller, with on-board cache (the bigger&faster the better).

Should I try downgrading the firmware to P19 and also reverting the VMware drivers to P19?

Can I activate or deactivate some VMFS related features?

No, because it does not make any sense.

Can I from ESXi modify the drive cache behavior?

No, and it does not make any sense either. Controller can not use disk-cache for its operatioins. In fact, some raid-controllers even turn the disk-cache (not controller-cache) off, to prevent data-loss in case of power-outage and to avoid double-caching (keeping the same data both in controller-cache and disk-cache)...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! :winking_face:
Reply
0 Kudos
logan892
Contributor
Contributor

That's too bad. These two WD RE drives have served me well in software RAID 1 under CentOS up until a year ago when I still ran VMware Server.

The LSI controller was initially intended for a FreeNAS setup. It so happens that I acquired two, and so came about my attempt to use one of them in IR mode for local VMware storage.

I have been considering getting two Samsung SM863 drives to use in RAID 1. I would assume that SSDs, while likely not being able to reach their full potential in such a non-cached hardware scenario, would fare much better than mechanical drives in this same situation. If anyone has experience with this I'd love to hear about it!

The SM863 have super capacitor secured on-drive cache. Would this SSD cache also be completely disabled, and/or performance crippled, when connected to an LSI 9212-4i4e without cache and configured in a RAID 1 array?

The cost of two SM863 would be similar to that of a new RAID controller with cache and BBU, which makes the SSD alternative much more appealing.

Strictly speaking I don't think my home server / hobby project absolutely requires RAID 1. I'd be better off with a proper backup solution. The potential failure of a single SSD is something I can live with, as long as my backups are in order.

Reply
0 Kudos
JarryG
Expert
Expert

I'm affraid even in case of SSDs missing controller-cache will have negative impact on performance, with solo/raid ratio being similar, as for HDD (so instead of ~500MB you will get maybe ~50MB/s from SM863). If you buy such an expencive SSDs, it is a pitty not to use them properly. Two options come to my mind:

A: get some 2nd-hand/used controller. I got mine M5016 on eBay, for ~200$, including cache and super-cap. One year later I found one more for ~60$ (some ISP was selling spare-parts).

B: pass-through your disks/SSD to one VM, and create NAS with software-raid, for hosting other VMs. NAS-OS will do disk-caching and with vmxnet3-nics on local vswitch you'd get quite good network performance...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! :winking_face:
Reply
0 Kudos