New !! Open unofficial storage performance thread

[christianZ]

Hello everybody,

the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.

Oliver will make a few links between the old and the new one and then he will close the old thread.

Thanks for joining in.

Reg

Christian

[SteveEsx]

Hi Gabriel,

I don't use DAS anywhere I have tested local server disks and SAN boxes. I guess with DAS you are refering to the internal raid with 6 disks on the R710 server?  Is 22k iops not normal for internal raid 10 disks with 6 spindles? I don't have enough experience with this test to know that, I would love to know if I have done something wrong in the test

I have followed what I think is the normal procedure here, created a virtual machine with different versions of windows server and installed iometer 2006 then run the test. The numbers are from the logfiles parsed by the web page http://vmktree.org/iometer/

I am not posting here to have the "best numbers" I want to find what is normal performance with a test that is repeated by other people so I can use that test as a reference for finding issues with i/o at other sites.

This is the procedure I have done:

1. created 3 virtual machines "iometer01", "iometer02", and "iometer03" (using 3 different VM scsi types)
2. installed windows server and patched them with windows update
3. installed iometer-2006.07.27.win32.i386-setup.exe (I'm not sure if this is the version everyone is using or not?)
4. Opened the configuration file that contains the tests, added them to "assigned access specifications"
5. Clicked the green flag to start test and created log file
6. Parsed the logfile with  http://vmktree.org/iometer/

If there are some other tests you would like me to do to verify the numbers I'm happy to do so.

Why some tests say 0% cpu load I have no idea I'm not an iometer expert, I have attatched the logfile from one test "iometer02-san01.csv" which is the test that seems to have the lowest cpu load so maybe you can see if something is not right there?

Thanks for your input, interresting to know if the test numbers are sane.

[EllettIT]

My last day with Ellett Brothers will be March 25th, please call 803-345-3751 to be directed to the appropriate person.

Thanks!

[larstr]

Steve,

When using iometer inside a VM there are several things that may affect your results. One is disk and disk controller cache that may affect the results and give higher numbers. Using a larger iometer test file will solve this problem.

Another thing that may affect the results is if the clock inside the guest is not correct. This happens especially with high cpu load. In 3.5 the solution to this issue was the descheduled time service. This service is however not available anymore on newer ESX versions.

You're also running your tests on the local C: drive which typically is the drive where the OS is installed. It would be better to use a dedicated disk drive.

Regarding the cpu load, by studying the csv file we can see that the load is indeed higher than 0%, but since you're using multiple workers (2 cpus), only the load of the second cpu is reported by that page.

Lars

[EllettIT]

My last day with Ellett Brothers will be March 25th, please call 803-345-3751 to be directed to the appropriate person in the IT department.

Thanks!

[SteveEsx]

Hi Lars,

Yes I agree with all you are saying but I think this is the most efficient way to have a simple iometer reference test (a "standard" test that it is easy to teach others to do). I would follow your suggestions if my goal were to have a competition between SAN products in performance, but my goal is simply to see systems that have major issues with either random or sequential i/o performance at other sites. My company frequently install heavy database solutions on old SANs that often perform very badly which is a waste of time and money, nice to have some numbers to show whats wrong in an efficent way.

I guess one option would be to have a virtual machines and add extra VMDK disks on other LUNs to test them, that would also be efficient. But what do most people do when they post results here I wonder? I did my first tests here on a server with no other VMs on a LUN with no other VMs, my goal was to get the best result on a non busy environment so I can see the effects later when the environment is busy.

I think the test file is large enough in this test, the VM has 8 gb of ram and the PERC controller and MD3000i controllers does not have more than 1 gb cache. However I'm rather new at using iometer before I used simpler tools like HDtune, HDtach. Those tools are not so good at showing random i/o performance as I think iometer is, and it is very nice to have some reference numbers and comments from this community. It might also have an effect on my own storage purchases in the future I guess

Do you think 22k iops is impossible with 6 drives like Gabriel says here? For comparison I did a test on a physical PE2900 host with 8 drives and it got these numbers:

W2k8 r2 enterprise

Raid 10

8 x SAS Seagate 300 gb sas 15k

Perc 6/i

SEP11 AV installed

SERVER TYPE:    Dell PowerEdge 2900 III CPU TYPE    / NUMBER: HOST TYPE: Dell PowerEdge 2900 III STORAGE TYPE / DISK NUMBER / RAID LEVEL: R10    8xSAS 15K Seagate 3.5" Perc 6/i
Test name	Latency	Avg iops	Avg MBps	cpu load
Max Throughput-100%Read	2.98	18170	567	2%
RealLife-60%Rand-65%Read	15.79	3079	24	0%
Max Throughput-50%Read	3.06	19046	595	3%
Random-8k-70%Read	17.55	2813	21	0%

However on this host the test file might have been too small I guess as the server has 32gb of ram and w2k8r2 does a lot of strange i/o caching.

Just for fun I also tested a PE2900 server with mainstream intel 80gb g2 ssd drives and then it had an awesome performance in the random I/O as expected:

W2k8 r2 ent

Raid 5

8 x intel ssd 80gb gen2

Perc 6/i

no AV

SERVER    TYPE: Dell PowerEdge 2900 III CPU TYPE    / NUMBER: HOST TYPE: Dell PowerEdge 2900 III STORAGE TYPE / DISK NUMBER / RAID LEVEL: R5    8xIntel 80gb SSD gen2 Perc 6/i
Test name	Latency	Avg iops	Avg MBps	cpu load
Max Throughput-100%Read	3.04	19950	623	2%
RealLife-60%Rand-65%Read	4.59	12480	97	1%
Max Throughput-50%Read	3.12	18678	583	3%
Random-8k-70%Read	5.21	10396	81	0%

I know its impossible to make the perfect test but at least I hope to avoid doing some really bad mistakes that makes the numbers meaningless. And I'm hoping these numbers can be compared to others in this forum.

Didn't know about the results only showing one cpu thats interresting.

Thanks for your input,

S

[larstr]

Steve,

When I setup a new system I usually do the same thing as you and run a few tests before it gets any load just to get a baseline of that system. If I later get performance issues I can compare with my initial results in the troubleshooting process.

The sequential tests are often highly affected by the cache and that's why you get such high nimbers on those tests. The disk controllers can predict sequential reads and will often read larger blocks than the ones requested to get maximum performance on such operations. The disks also have their own cache that helps a bit. These things help are especially helpful for operations such as backup and large file copy operations.

In a VMware environment with many VMs sharing the same storage, these numbers are not as important as the random workload tests. Random and RealLife show a workload more similar to the one you can expect during heavy load and are also the ones worth paying most attention to. As your tests show, the SSD tests show really good performance.

Lars

[Jaseliu]

Hmm, I am getting pretty bad result from my StarWind iSCSI setup.

I have my SAN setup as Raid 60, it seem like when it is doing RealLife random access, I suffered the most. Only averageing 3Mbps..

This is a Dell R710 server connect to a Chenbro 72TB box running raid 60 thru gig nic.

Maybe raid 60 is not that great for random access, but 3Mbps..

Anyone have any insight ? thanks so much !

Max Throughput-100% Read
IOps - 44145.3
MBps- 1379.5

RealLife-60%Rand-65%Read
IOps - 3664.4
MBps- 28.62

Through iSCSI .

Max Throughput-100% Read
IOps - 2967.95
MBps- 92.74

RealLife-60%Rand-65%Read
IOps - 400.4
MBps- 3.12

[EllettIT]

I am no longer employed with Ellett Brothers, please call 803-345-3751 to be directed to the appropriate person in the IT department.

Thanks!

[larstr]

Jaseliu,

For random tests you should not pay as close attention to the MBps as the IOps. Having 3664 IOps on rge Random test is quite good, but your number over iSCSI (400) is quite bad. How many disks do you use in your setup? What type of disks?

Lars

[Jaseliu]

Hi, thanks for the reply.

I am running 24 7200rpm SATA HDs Raid 60 setup.

I think I sort of figure out why I am getting the bad numbers.

Here is why, not sure what's the difference, but it's kinda weird to me.

I ran the first test without a filesystem in Iometer, and that's the crappy result I got back.

Then, I formatted the drive to be NTFS, and let IOMeter created a temp file there, and here is my result.

Max Throughtput 100% Read

IOPs: 2995.9

MBps 93.6

RealLife 60% Rand

IOps: 2271.2 (Read: 1474 Write 796)

MBps 17.7

Max Throughput 50%

IOps: 1323.5

MBps: 41.3

Random 8k 70% Read

IOps: 2670.7

Mbps: 20.8

That's a good amount of increase of the raw test that I did, but why is that ? and why is MBps not as important in random test ?

Thanks !

[dougdavis22]

SERVER TYPE: Windows 2008 R2 Enterprise, 64Gb RAM (all RAM reserved)
CPU TYPE / NUMBER: vCPU / 8 (CPU affinity set for CPU cores 1-9)
HOST TYPE: HP BL465c, 128GB RAM, 2 x AMD 6174 (12-Core), 2.2Ghz
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP MDS600 DAS, 1 x 2TB SATA, RAID0

Test name	Avg. Resp Time /ms	Avg IOps	Avg MBps
Max Throughput-100%Read	5.022	12206	381
RealLife-60%Rand-65%Read	235.3	125	0.97
Max Throughput-50%Read	190.2	155	4.8
Random-8k-70%Read	234.5	125	0.97

[mry]

Kontakt drift@eogs.dk eller +45 3330 7587

[dougdavis22]

SERVER TYPE: Windows 2008 R2 Standard, 8Gb RAM CPU TYPE / NUMBER: vCPU / 2 HOST TYPE: IBM HS22v, 128GB RAM, 2 x Intel Xeon X5680 (6-Core), 3.33Ghz STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS5300 via SVC, 7+1 600GB FC, RAID5
Test name	Avg. Resp Time /ms	Avg IOps	Avg MBps
Max Throughput-100%Read	5.696	10673	333
RealLife-60%Rand-65%Read	14.06	2063	16
Max Throughput-50%Read	19.50	1491	46
Random-8k-70%Read	78.60	377	3

[joeraman]

Its informative one provides a lots of information on topic also provide more data on topic.

[legaultp1]

here are my insane specs

it's a 2 server configuration.

1 file server, 1x 10GBps infiniband interconnect, 1 vm server

file server config:

8gb ram

quad core 3ghz@1333fsb

adaptec 5805 SAS controller running 16 sata drives WD 1TB black edition with BBU

Mellanox Connect-X infiniband card

6 Arrays to of 2tb raid 6 ( 1 array is built from 7x 1tb disks raid 6, 3 per drives)

ESXI Server: ESXI 4.1 U1

16GB ram

8gb compact flash disk

CPU: 2x quad core 3ghz @1333fsb

network card 10GB infiniband to connect to my file server

Infiniband switch: Topspin 90

-------------

Test	average io response(ms)	Maximum IO response(ms)	cpu utilization	IOPS	MBPS
4K 100% read 0% random, 1 queue	0.1235	73.1755	22.74%	8113.81	31.51
16K 100% read, 0% random, 1 queue	0.1559	12.5543	21.27%	6386.21	99.91
4K 75% read, 0% random, 1 queue	0.1316	25.0660	20.85%	7565.66	29.63
16K 75% read, 0% random, 1 queue	0.2561	109.4688	13.62%	3903.28	61.95
4K, 100% read, 100% random 1 queue	6.3223	63.3976	0.65%	158.97	0.62
16K, 100% read, 100% random 1 queue	6.6691	77.2706	0.53%	151.92	2.35
4K, 75% read, 100% random, 1 queue	6.5592	106.5505	0.60%	152.16	0.56
16K 75% read, 100% random, 1 queue	7.3537	62.5770	0.44%	135.34	2.12
same tests with 64 queue length
4K 100% read 0% random, 64 queue	1.5147	86.9491	96.93%	42083.38	164.65
16K 100% read, 0% random, 64 queue	2.9452	79.2967	54.66%	20837.64	328.29
4K 75% read, 0% random, 64 queue	3.1938	132.7558	44.08%	20049.75	78.04
16K 75% read, 0% random, 64 queue	18.0842	464.8004	9.62%	3497.97	54.66
4K, 100% read, 100% random 64 queue	122.8217	594.2059	1.54%	583.22	2.27
16K 100% read, 100% random, 64 queue	106.5224	591.4899	1.72%	532.28	8.31
4K, 75% read, 100% random, 64 queue	218.6931	660.9712	0.90%	283.34	1.11
16K 75% read, 100% random, 64 queue	233.4813	621.7400	1.04%	271.87	4.24

anyway, i ran some tests as i just finished configuring it a few days ago after pain and pain of configuration

[chiprock]

All, We just purchased a NetApp SAN and I am looking for feedback on whether or not the performance is where it should be.  Help is appreciated.  I need to go into production shortly.

SERVER TYPE: 2008 64bit R2 VM ON ESXi 4.1.0

CPU TYPE / NUMBER: 4 VCPU - 8GB Ram

HOST TYPE: IBM x3690 X5 - Xecon x7560 2.27Ghz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Netapp FAS2040 / two disk aggregates - see below for details / RAID DP

SAN TYPE / HBAs: 1GB ISCSI - Server has two 1GB ports set to go round robin to the Netapp SAN.  Cisco 2960 swiches.  Netapp 2040 has 2 trunked ports comprised of two 1GB ports.  So it has two 2GB trunks.

Note:  Jumbo Frames are in use

Here is the results from an array made up of 12 300GB - 15k SAS disks.  This is on controller A within the FAS2040.

Here is the results from the second array on a seperate DS4243 which attached to the 2040 via a SAS channel.  It is comprised of 24 - 1TB SATA Disk.  I believe they are 7200 RPM.

Lastly here are the results from some local disk.  This is two 300GB 15k SAS 2.5" SAS disks that are local in the IBM3690.

Any feedback is appreciated!  Again I am going live and need to know if these results are acceptable for this type of configuration.  I have dug thru the three threads on this site and just want to say they are awesome.  I have found a ton of good info to compare but still would like feedback if anyone has any.

Thanks in advance,

[cnenzel01]

SERVER TYPE: Intel Modular Server
CPU TYPE / NUMBER: Xeon 2.4GHz/2
HOST TYPE: ESX 4.1
STORAGE TYPE / DISK NUMBER / RAID LEVEL: 
Intel Modular Server / 6 / 50
 

Test name
Latency
Avg iops
Avg MBps
cpu load

 


Max Throughput-100%Read
0.00
7617
238
31%




RealLife-60%Rand-65%Read
5.33
1855
14
47%




Max Throughput-50%Read
38.90
2374
74
68%




Random-8k-70%Read
4.06
1656
12
45%

Any thoughts?  does this look about right for this type of server (blade) with 6 discs in a RAID 50 array?

[PinkishPanther]

Hello All,

I've found this a useful thread!

Please note that when the latency numbers report 0 for Max Throughput Latency it probably means you've used the latest version of IOMeter and parsed it via - http://vmktree.org/iometer/

I have discovered that the first data point have moved from column O to column R and the 4th data point has gone from AT to AW.

This means:

The first field in the table reports write MBps   (Decimal) instead of Average Response Time

The fourth field is not CPU but I'm not certain which one it is, doesn't seem to be the data in AT which I'd expect.

I expect to be posting some numbers for our environment later, I'll be interested in feedback.

One other thing, we believe that our Jumbo Frames via ESX 4.1 is not working.  We are able to send an unfragmented ping of 8972 bytes yet when doing a wireshark capture within an image using our iSCSI interface the largest packets are 1490 - we also see no Jumbo frames being counted at the SAN or our switches.  (Cisco 3750's)  Does anyone know if the count of giants using the show interface command will increase when Jumbo frames work?

Thanks,

John

[kopper27]

hi guys

this is mine

using iometer-2008-06-22-rc2.win.32 and http://www.mez.co.uk/OpenPerformanceTest.icf

formated at http://vmktree.org/iometer/

too late I realize I did theses test on a Windows 7 box

I executed 2. Only difference is the amount of Hard Drives first is 3 disk RAID 5 and the second is 9 Disk RAID 5. Based on that what do you think is much better for my Storage and Vmware Infraestructure Less disk or more Disk.

your inputs good or bad? is there a web page that says that value is low because .....

BTW why RealLife and Random Are so low 6 MBps?

thanks a lot

--------------- Update -----------------

I asked to one of our pre-sale storage guy about this since I really don't know what those test are or mean.

Max Throughput-100%Read = All the bandwith only for READ. It uses all the Bandwith available. This is secuential.

RealLife-60%Rand-65%Read = A common or RealLife metric which says Reading 65 percent but in a random way it means that blocks that are being read are not secuencial.

Max Throughput-50%Read = This means 50% read 50 % write secuencial block.

Random-8K-70%Read = Using 8K blocks and doing 70% read on random blocks.

is that above is correct much disks better performance or at least less latency

[minifo]

SERVER TYPE: Windows 2008 R2 on ESXi 4.1
CPU TYPE / NUMBER: 2 vCPU
HOST TYPE: HP DL380 G7
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP P2000i 1Gb G3 MSA / 16x146gb 10k / RAID10

ESXi software iSCSI MPIO

 

Test name
Latency
Avg iops
Avg MBps
cpu load

 


Max Throughput-100%Read
5.86
10151
317
0%




RealLife-60%Rand-65%Read
22.44
2095
16
14%




Max Throughput-50%Read
5.65
10475
327
0%




Random-8k-70%Read
18.04
2732
21
10%

Is the random IO performance a bit low?