VMware Cloud Community
FrostyatCBM
Enthusiast
Enthusiast

Dell MD3200/MD1200 6Gb/sec Shared SAS performance tests

We recently acquired a Dell MD3200 and MD1200 6Gb/sec shared SAS storage system and I have spent a couple of days setting up some tests to see how it compares to our old iSCSI EMC Clariion. Wasn't really a fair fight, given that the EMC had 500GB 7200rpm SATA disks in it, whereas the new systems had 450GB 15000rpm SAS in the MD3200 and 2TB 7200rpm nearline SAS in the MD1200.

Big caveat on the following test results ... all are the result of testing a single VM ... I do not (yet) have any results with a multipleVM shared workloads on shared LUNs, so I haven't tested the impact of contention, locking, etc.

Tests were done by running simple IOMETER jobs. Results per job were cross-checked with 'esxtop' to ensure that they roughly matched (they did). I installed a Windows Server 2008 R2 VM and created 100GB vmdk's on each storage tier. I tested with 5-disk and 6-disk RAID5 as well as a few tests with a 6-disk RAID10.

I had a guess at constructing some sample workloads to simulate SQL data (reading/writing 64KB chunks of data) as well as simulating retrieval of data for backups. My guess as to how these might work could be way off! If anyone thinks so and can suggest improvements to the IOMETER test workloads, I am happy to receive that feedback and can probably repeat the tests with different parameters.

Some of the conclusions I reached (some of which came as a surprise to me, and some which didn't!):

  • 5-disk RAID5 is almost indistinguishable from 6-disk RAID (between 2% and 4% better performance with the 6-disk config)

edit: this is only under light workload ... see updated stats below

  • 5-disk RAID5 out-performed a 6-disk RAID10 by a fair margin (about 20%) on the "all in one" IOMETER tests

edit: this is only under light workload ... see updated stats below

  • 450GB 15k SAS outperformed 2TB 7.2k nearline SAS by 50%-100% in a 5-disk RAID5 configuration

  • its possible to pull around 700-800MB/sec of real data out of the shared SAS storage given optimal conditions, which blows away a small iSCSI config (unless you have masses of 1Gb/sec NICs teamed, or are blessed with 10Gb/sec ethernet)

Here's the some of the test results, in case they're of interest. Feedback welcomed.

IOMETER standard "all in one" workload

Storage

Physical

RAID

Number

TEST RESULTS

% IMPROVEMENT

Tier

Disk Type

Type

of Disks

IOPS

MB/sec

Avg Seek

IOPS

MB/sec

Avg Seek

MD3200

15000rpm 450GB SAS

RAID5

5

2015

25.8

0.50

294%

291%

74%

MD3200

15000rpm 450GB SAS

RAID5

6

2065

26.5

0.48

303%

302%

75%

MD3200

15000rpm 450GB SAS

RAID10

6

1640

21.0

0.61

220%

218%

69%

MD1200

7200rpm 2TB nearline SAS

RAID5

5

1710

21.9

0.58

234%

232%

70%

MD1200

7200rpm 2TB nearline SAS

RAID5

6

1733

22.2

0.58

238%

236%

70%

MD1200

7200rpm 2TB nearline SAS

RAID10

6

1175

15.1

0.85

129%

129%

56%

iSCSI SAN

7200rpm 500GB SATA

RAID5

5

512

6.6

1.95

"simulated SQL workload" workload (reading/writing 64KB blocks in 80:20 ratio, random/sequential in 90:10 ratio)

Storage

Physical

RAID

Number

TEST RESULTS

% IMPROVEMENT

Tier

Disk Type

Type

of Disks

IOPS

MB/sec

Avg Seek

IOPS

MB/sec

Avg Seek

MD3200

15000rpm 450GB SAS

RAID5

5

240

14.9

4.18

229%

224%

69%

MD3200

15000rpm 450GB SAS

RAID5

6

250

15.6

4.00

242%

239%

71%

MD1200

7200rpm 2TB nearline SAS

RAID5

5

119

7.4

8.43

63%

61%

38%

MD1200

7200rpm 2TB nearline SAS

RAID5

6

126

7.9

7.90

73%

72%

42%

iSCSI SAN

7200rpm 500GB SATA

RAID5

5

73

4.6

13.60

"simulated backup workload" workload (reading 64KB blocks, 100% sequential)

Storage

Physical

RAID

Number

TEST RESULTS

% IMPROVEMENT

Tier

Disk Type

Type

of Disks

IOPS

MB/sec

Avg Seek

IOPS

MB/sec

Avg Seek

MD3200

15000rpm 450GB SAS

RAID5

5

2004

125.3

0.50

309%

309%

75%

MD3200

15000rpm 450GB SAS

RAID5

6

2017

126.1

0.49

312%

312%

76%

MD1200

7200rpm 2TB nearline SAS

RAID5

5

1967

123.0

0.51

301%

302%

75%

MD1200

7200rpm 2TB nearline SAS

RAID5

6

2008

125.5

0.50

310%

310%

75%

iSCSI SAN

7200rpm 500GB SATA

RAID5

5

490

30.6

2.04

Reply
0 Kudos
35 Replies
abkabou
Contributor
Contributor

I have connected

Here is how i have configured my San

I have Set Vmware ISCSI Prefered path to MRU

I have only used Used 1GB Port on each Controler.

I have dedicated one Network card by Network

Checked that the Lun was accessed using the prefered Path, if it's not the case the MD3200i give you an alert Message.

I have Created one VMKNIC per Network Card

I have Set Jumbo Frame to 6500 on both element, network card

I have set one Target by Initiator.

I have created two VSWITCH.

VSWITCH1 ==> VMK1 ==> Physical Network Card 1 ==> Jumbo 6500 on SAN and on network Card and vswitch ==> accessing Controler 1 (accessing controler 2 if controller one 1 fail)

VSWITCH2 ==> VMK2 ==> Physical Network Card 2 ==> Jumbo 6500 on SAN and on network Card and vswitch ==> accessing Controler 2 (accessing controler 1 if controller one 1 fail)

Abdellah KABOU

Reply
0 Kudos
Drovec
Contributor
Contributor

Hi,

Thank you for your hints. I have more questions for you.

You say that you use ISCSI Preferred path MRU. Is it faster than Round Robin?

You say that you use 1GB Port on each Controller. On MD3200i each controller has 4 ports. Are you really using only 1 port from each controller? I’m not sure that it’s good idea, what happens if one controller fails?

You say that you dedicated one Network card by Network. Do you mean 1 physical Ethernet port to one iSCSI VMkernel? Or 1 physical Ethernet card to one iSCSI VMkernel?

You use Jumbo frame value 6500, why this number?

You say that you set one Target by Initiator. Do you mean one target as one MD3200i or one IP address?

I also want to ask if you use SAN switches. If yes what type and configuration.

Thank you.

Reply
0 Kudos
abkabou
Contributor
Contributor

Hi,

You say that you use ISCSI Preferred path MRU. Is it faster than Round Robin?

     - i was using MRU but i have change it now to Round Robin, and it's faster when using multiple PATH

You  say that you use 1GB Port on each Controller. On MD3200i each  controller has 4 ports. Are you really using only 1 port from each  controller? I’m not sure that it’s good idea, what happens if one  controller fails?

I was using one port on each controler but both port on each controler could access the LUn, the first port on controler 0 was the prefered path.

If controler 0 fails it will switch to controler 1 port 1

You say that you dedicated one  Network card by Network. Do you mean 1 physical Ethernet port to one  iSCSI VMkernel? Or 1 physical Ethernet card to one iSCSI VMkernel?

1 physical Ethernet port to one  iSCSI VMkernel

You use Jumbo frame value 6500, why this number?

This is the best value that i have found during my tests, i tried increasing and decreasing this value

You say that you set one Target by Initiator. Do you mean one target as one MD3200i or one IP address?

ISCSI target discovered from one target on each VMHBA

I also want to ask if you use SAN switches. If yes what type and configuration.

I'm using Alcatel-lucent switch 6450

You can't change the MTU SIZE in the configuration, beacuase according to some other alcatel topics it detects the mtu and just forward it.

I have now Connect my esxi using Three ISCSI VMKERNEL, to 2 iscsi port of the MD3200i

- ISCSI 01 192.168.131.99

- ISCSI02 192.168.131.100

- ISCSI03 192.168.132.100

I put you my last results.

last_best_result_one_ctrl_3_link_esxi.JPG

Thanks Abdellah KABOU

System and Network Engineer

Reply
0 Kudos
Drovec
Contributor
Contributor

Thank you for your tips Smiley Happy

Reply
0 Kudos
abkabou
Contributor
Contributor

Youare welcome. Smiley Happy

Thanks Abdellah KABOU

System and Network Engineer

Reply
0 Kudos
pbsenv
Contributor
Contributor

I’d be interested if anyone else is getting the purple screen of death on esxi now with 5.0 update 1. I’ve now had 3 in 2 months on one host and all 3 times vmware support has said its the Dell (LSI) SAS HBA causing it. They have said they are going to roll out a new driver soon in a future esxi update. This is obviously not good!

Reply
0 Kudos
abkabou
Contributor
Contributor

I have tried to update my esxi to the latest update esxi 5.0 update 1 using vmware iso,

After upgrade isaw a crash screen.

I'm back to the dell latest version after rebooting.

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

I have been running ESXi 5.0 U1 (3 hosts) for a several months now with our MD3200/MD1220/MD1200 array and have not had any crash problems at all. Everything has been remarkably stable actually.

Reply
0 Kudos
pbsenv
Contributor
Contributor

Here is the official word from the VMware support engineer about the Dell HBA card (LSI):

"

Thank you for your Support Request.

I wanted to follow up with you regarding the support request # 12207949208 registered regarding the issue with PSOD on the ESX/ESXi host.

As was informed earlier, we had found that the cause of the issue is due to the compatibility issues of the mptasas drivers ( controller drivers) with ESXi 5.0 hosts.

You could either downgrade the host to a lower version till we have drivers to fix this issue or we could check to see if we could get a debug driver ( test drivers) for you to install of the host and check if it helps in resolving the issue.

Looking forward to hearing from you.

"

Reply
0 Kudos
alexxdavid
Contributor
Contributor

Just let me know if you want some advice regarding the md3220i or the md3200i as they are quite capable when configured correctly.

I can get up to 400 MBps on 100% read on iometer.

David

Reply
0 Kudos
alexxdavid
Contributor
Contributor

And those are not good results, there is no way that the system would be faster in 50% read/write than in 100% read.

something got be wrong in this config

Reply
0 Kudos
radimf
Contributor
Contributor

Hi,

actually scenario like this might be fairly common (100% read slower than mixed seq read/write) 

and actually it might be correct numbers for his current config.

i have seen simillar behaviour before

last time i have seen this was VM residing on SAN and fast storage behind it.

reason for this behaviour - as VM/ ESXi host/SAN was configured - test used only 1 port for 100% sequential read (no MPIO etc), and due to high number of spindles on SAN storage the cap was the 1x 8gbit FC port bandwith and not the storage itself.

In that test read/write scenario was able to use 2nd port as well and resulted in higher bandwith.

looks to me as this might be simillar case :

- 1gbit maxing for 100% seq read (125mb/s? - looks like too good to be true number for single 1gig link)

- more throughput delivered via both ports during read/write seq access  (188mb/s?)

Reply
0 Kudos
Starman20111014
Contributor
Contributor

I ran some perf tests on my production environment on both my md3200's. The test ran for 24 hours (including Veeam backups). I'll post some more findings on a few arrays:

host 1

6 x 600gig 15K 3.25 raid 5               166.7 mb/sec    peak iops: 2355.3     avg io 71.4kb / 30.0kb

8 x 900gig 10K 2.5 inch raid 10:     296.9 mb/sec     peak iops: 316     avg io 645.5kb / 6.1kb

8 x 900gig 10K 2.5 inch raid 10:     415.9 mb/sec     peak iops: 1023.7 avg io 585kb / 0.0kb

8 x 300 gig 10K 2.5 inch raid 6:     126.8 mb/sec     peak iops: 1560.2     avg io 68.5bk /1 7.1 kb

8 x 300 gig 10K 2.5 inch raid 6:     244.1 mb/sec     peak iops: 3891.8     avg io 187.6kb / 20.8 kb

8 x 1TB 7.2K 2.5 Raid 6:                199.6 mb/sec     peak iops: 590     avg io 97.0kb / 13.0 kb

host 2

8 x 900gig 10K 2.5 inch raid 10:     65.4 mb/sec     peak iops: 3882.3     avg io 119/2kb / 31.2kb

8 x 300 gig 10K 2.5 inch raid 6:     87.4 mb/sec     peak iops: 4499.6     avg io 45.7kb / 18.9 kb

host 3:

8 x 900gig 10K 2.5 inch raid 10:     174.4 mb/sec     peak iops: 5012.7     avg io 47.1kb / 40kb

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

I thought it might be interesting to dredge up this old thread of mine, now that we are about 7 years down the track.

Our MD3200/MD1220/MD1200 setup has served us really, really well.

Not a single regret about going with Direct Attached Storage (with 3 x ESXi hosts); it has performed brilliantly.

But now I am about to embark on our Infrastructure v2 project and have just purchased:

3 x Dell R740 hosts (will run vSphere 6.5)

Dell SC5020 storage with 20 x 1.92TB SSDs (direct attached storage again!).

Some time in the next 4-6 weeks we will be firing these puppies up and I'll be repeating my performance testing (see first page of this thread).

Looking forward to seeing how much better the SSD performs!

Frosty.

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

Well I have to say that I am rapt with the performance of our new R740's with the SC5020 storage loaded with 20 x SSDs.  I re-ran basically the same set of performance tests on the SC5020 as I ran on the MD3200 originally (see start of this thread) and both IOPS and MB/sec results are through the roof.  Here's a brief explanation of the config and initial test results:

SC5020 has 2 disk groups of 10 x 1.92TB SSDs each, only a single tier (no spinning disks) so therefore no "smarts" in the SC5020 like auto-tiering being utilised.  We are using direct attached storage, giving 2 cables per host, each with 4 x 12Gb/sec channels).  We are using only single redundancy RAID5 and RAID10 volumes.  I ran up a Windows Server 2008 R2 VM with 2 x vCPU and 4GB RAM.  Gave it an extra 100GB HDD for use with IOMeter.  Ran a series of tests to try to simulate both Exchange and SQL workloads, comparing with the MD3200, and was like night and day.

* data throughput / performance for VMs should be anywhere from 20x - 40x better (measured in MB/sec and in IOPS)

* back-end disk write performance (measured via Dell Storage Manager) was seen to peak at:

-- 3250MB/sec writing to RAID5 on some tests (e.g. Storage vMotion, so volume to volume copy)

-- 4250MB/sec writing to RAID10 on some tests (ditto)

DSM also showed performance on some tests peaking in excess of 100,000 IOPS.

IOMETER test results

1.  simulated SQL database workload RAID10

(64K blocks, 66:34 read/write, 100% random, 16 I/Os per target)

Old MD3200 RAID10 = 1243 IOPS and 78 MB/sec

New SC5020 RAID10 = 38973 IOPS (31x) and 2554 MB/sec (33x)

2.  simulated SQL database workload RAID5

(64K blocks, 66:34 read/write, 100% random, 16 I/Os per target)

Old MD3200 RAID5 = 794 IOPS and 50 MB/sec

New SC5020 RAID5 = 33692 IOPS (42x) and 2208 MB/sec (44x)

3.  simulated Exchange mail server workload RAID10

(8K blocks, 55:45 read/write, 80% random, 64 I/Os per target)

Old MD3200 RAID10 = 2004 IOPS and 16 MB/sec

New SC5020 RAID10 = 45414 IOPS (23x) and 372 MB/sec (23x)

4.  simulated Exchange mail server workload RAID5

(8K blocks, 55:45 read/write, 80% random, 64 I/Os per target)

Old MD3200 RAID5 = 1145 IOPS and 9 MB/sec

New SC5020 RAID5 = 42282 IOPS (37x) and 346 MB/sec (38x)

I'm pretty sure we could coax more performance out of this with a different config, however I've made some conscious trade-offs in order to get some other benefits (e.g. reduced rebuild time and minimising number of volumes/VMs affected if an SSD fails).

Reply
0 Kudos
MagnusQITJ
Contributor
Contributor

Good Morning,

I know the topic is old, but I still use an MD3200i equipment.
My question is the following, if I connect the MD3200i to the same esxi HOST, with 8 Network Interface, will I get a 4Gbps connection? (4x1GB), and the other 4 ports would be redundant?
I ask which setting I get the best performance?

Sorry for my English, I'm using a translator.

Reply
0 Kudos