VMware Cloud Community
christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
0 Kudos
457 Replies
UMinventor
Contributor
Contributor

As InsaneGeek points out, there are numerous mistakes one can make to improve or degrade an array's performance. I'd like to add:

\* run the tests for 15 minutes and throw away the results. Now record.

\* if one had enough time, it would be interesting to allow the tests to continue capturing results every 30 minutes over a couple days - but no one has that kind of time. If one did, report the first and last window recorded.

\* ensure the files accessed by the test realistically fill the available storage. That is, if there is 1TB of disk space available, the files should fill at least 60% (for example) of the available space. Then the random access should be over this entire space. Sequential files that simulate log files should be much smaller - say 1GB.

Does anybody have a NetApp FAS? EVA8000? CX3? Sun6540?

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS VM ON ESX / SAN-MELODY 2.0.1

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM xSeries 235, 4 GB RAM; 2x XEON HT 2,4 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody on IBM xSeries 235 (as above) /

5 x SCSI U160 disks (r/w caching on) / NMV Volume with raw disks, 3 GB cache, 1 x 1Gb NIC for iSCSI

SAN TYPE / HBAs : iSCSI, ESX software initiator, 1 x 1Gb NIC

\##################################################################################

TEST NAME--


Av. Resp. Time ms
Av. IOs/sek

Av. MB/sek
----

\##################################################################################

Max Throughput-100%Read........__32______..........__1780____.........___56_____

RealLife-60%Rand-65%Read......___43_____..........___1360___.........____11____

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: Host CPU Util. IN VM ~75 % (by 1); Test file = 15 GB on disk 😧

##################################################################################

Results verified over "Perfmon" on San-Melody server (checked the ios on disks ~ 200 ios/sek).

Tests over 15 minutes got the same numbers.

0 Kudos
christianZ
Champion
Champion

Well I didn't want to make this test a "Mega Test" but a small verifying of owns results and more details should be of course helpful.

The cache of storage subsystems has of course influence on the results but one should say that the cache is one of more important properties of such systems. By systems with e.g. 32 GB cache it is not practicable to create e.g. 100 GB test file - by such systems one should create the load over many clients - and by us the "Response Time" is the most significant(together with IOs/sek)- in this case both values will be generate on cache ios - that's true.

The DS8000 and EMC Symetrix series are playing of course in other league here.

Now I myself saw by testing of SanMelody that the "Cache Algorithm" seems to function very well and even with 15 GB test file - with 3 GB cache ( and testing time of 15 min.) I could see only small disk's activities - so tested many times until I could see a lot of activieties on disks and all these were of course verified over "perfmon" on the SanMelody server.

0 Kudos
mcwill
Expert
Expert

I'm not so sure about the SanMelody cache, but maybe it depends on the disk controller. I did a test recently comparing SanMelody Lite to SanMelody on the same hardware so the only variable that changed was the 128MB cache on the Lite version compared to approx 1.3GB cache on the full version.

The difference was at most a 10% increase in speed.

I'll dig out the values and post them.

0 Kudos
mcwill
Expert
Expert

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS VM ON ESX / SAN-MELODY 2.0.1

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE 1950 4GB

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody Full on Xeon 5130 whitebox /

RAID 10 - 4 x SATA 10k disks (r/w caching on) - 256KB Stripe/ 1.3 GB cache

SAN TYPE / HBAs : iSCSI, ESX software initiator, 2 x 1Gb Team NIC

\##################################################################################

TEST NAME--


Av. Resp. Time ms
Av. IOs/sek

Av. MB/sek
----

\##################################################################################

Max Throughput-100%Read........__18______..........__3133____.........___99_____

RealLife-60%Rand-65%Read......___72_____..........___834___.........____6.3____

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS:

0 Kudos
mcwill
Expert
Expert

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS VM ON ESX / SAN-MELODY LITE 2.0.1

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE 1950 4GB

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody Lite on Xeon 5130 whitebox /

RAID 10 - 4 x SATA 10k disks (r/w caching on) - 256KB Stripe/ 128 MB cache

SAN TYPE / HBAs : iSCSI, ESX software initiator, 2 x 1Gb Team NIC

\##################################################################################

TEST NAME--


Av. Resp. Time ms
Av. IOs/sek

Av. MB/sek
----

\##################################################################################

Max Throughput-100%Read........__19______..........__3058____.........___95_____

RealLife-60%Rand-65%Read......___78_____..........___772___.........____6.0____

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS:

0 Kudos
christianZ
Champion
Champion

Thanks for that - so we can see now that probably the numbers from Pauliew1978 are not optimal (as I guessed) and my test box with 5 scsi disks and 3 GB cache runs fast (although old hardware here).

Pauliew1978 - you must check your configuration - the numbers should be better by your hardware - I think.

0 Kudos
CWedge
Enthusiast
Enthusiast

Yes indeed it seems to be the fastest for now (and

everlasting ?).

TEST NAME-----------------Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

IBM DS8300:

RealLife-60%Rand-65%Read......___1.42__..........__260

3__.........__31.30___

EMC DMX3000:

RealLife-60%Rand-65%Read......___1.8_____..........\___

2170___.........___17_____

The question here could be which system could serve

more guests with such performance .

They both are playing of course in other league - can

profit from the large cache (32 GB).

I notice the MB/s number of 31.10 - that seems to be

a bug (by block size 8kB).

I was running 180 VM's over 9TB's while that test was running....

Up till a little while ago I was getting errors "Low I/O Detected" lololol....

I love this thing...No matter how hard I push the Vm's it's still falling asleep.

Which I think makes the ESX hosts life a lot easier as they don't have to buffer so much I/O's which keeps CPU down.

I just keeping running into low memory errors, never any cpu or i/o errors.

One other side note. I only have 144 of the 572 Drives it can Hold 😜

Although i think it'd be neck and neck running these two boxes at full capacity.

Also keep in mind now our DMX3000 is only DMX-2 Archietecture...

The DMX3 is even faster and more scalable...Hopefully I'll have one of those this year...

Message was edited by:

CWedge@Amsa

0 Kudos
christianZ
Champion
Champion

Woh - you are lucky men with such system.

Of course the two systems couldn't be compared with this test !!

As I mentioned they both are playing in other league - or maybe the EMC DMX3000 plays in its own league ?

0 Kudos
rievax
Enthusiast
Enthusiast

Here are my results on 3 different LUN types:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 2.5.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 3 x 250GB 7.5K SATA + 1 / R5

SAN TYPE / HBAs : FC , QLA4050 HBA

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek-

\##################################################################################

Max Throughput-100%Read....._______21________......____2792___.......____87____...

RealLife-60%Rand-65%Read...._______364_______......____123____.......____0.96__...

Max Throughput-50%Read......_______14________......____553____.......____17____...

Random-8k-70%Read..........._______411_______......____109____.......____0.85__...

EXCEPTIONS: VCPU Util. 51-25-15-18 %;

##################################################################################

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 2.5.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 6 x 146GB 15K FASTT + 1 / R5

SAN TYPE / HBAs : FC , QLA4050 HBA

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek-

\##################################################################################

Max Throughput-100%Read....._______21________......____2179___.......____68____...

RealLife-60%Rand-65%Read...._______35________......____501____.......____3.91__...

Max Throughput-50%Read......_______16________......____499____.......____15.6__...

Random-8k-70%Read..........._______25________......____501____.......____3.91__...

EXCEPTIONS: VCPU Util. 96-36-41-32 %;

##################################################################################

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 2.5.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 6 x 146GB 15K FASTT + 1 / R10

SAN TYPE / HBAs : FC , QLA4050 HBA

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek-

\##################################################################################

Max Throughput-100%Read....._______25________......____2356___.......____73____...

RealLife-60%Rand-65%Read...._______49________......____1100___.......____8.59__...

Max Throughput-50%Read......_______32________......____1816___.......____28.4__...

Random-8k-70%Read..........._______46________......____1222___.......____9.54__...

EXCEPTIONS: VCPU Util. 48-40-46-37 %;

0 Kudos
christianZ
Champion
Champion

Thanks for that - only one hint:

SAN TYPE / HBAs : FC , QLA4050 HBA

I suppose you have a FC hba - so that could be e.g. the qla24xx (qla4050 is the iscsi hba).

0 Kudos
joachims1
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM on ESX 3.01

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: BL25P, 16GB RAM; 2 x opteron 280 DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Netapp FAS3020 / 12+2 Disks / Raid DP (r6)

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........___10_____..........__5106___.........___159_______

RealLife-60%Rand-65%Read......____80____..........____642______.........___3______

Max Throughput-50%Read..........___7_______..........___7918_______.........____124___

Random-8k-70%Read.................___80_______..........__652________.........___4______

EXCEPTIONS: CPU Util.-XX%;

0 Kudos
rievax
Enthusiast
Enthusiast

Yes, sorry, it is a QLA2340 HBA.

0 Kudos
pauliew1978
Enthusiast
Enthusiast

Hi Christianz,

I am rechecking my setup as we speak. There was a problem with multipathing io set up on my sanmelody config. SO maybe it will be slightly quicker now. i am running some tests as we speak. Also my disks were not set to advanced performance in wondows disk manager so It may have stopped sanmelody using the faster features that the disk can do.

cheers and thanks for organising this test it has helped me get some real life figures on other setups.

Paul

0 Kudos
pauliew1978
Enthusiast
Enthusiast

Hi Christianz,

Just out of interest is there a way of controlling the amount of cache within sanmelody?

thanks,

Paul

0 Kudos
christianZ
Champion
Champion

Until now I haven't found it.

0 Kudos
christianZ
Champion
Champion

Wow - the first NetApp here. Thanks.

Have you sata or fc disks in it ?

Are you using FC or iSCSI (or maybe NFS) as connection type - when iSCSI then iscsi hba or software initiator ?

0 Kudos
cperdereau
Enthusiast
Enthusiast

Basic question:

1) where are you running this IOmeter ? on a VM or on the ESX server itself

2) I tried to compile the last version of iometer-2006_07_27.linux.i386-bin

\- copied the .tgz file in /usr/iometer

\- untar with : tar xvfz something.tar.gz

\- go to the subfolder named src/iomtr_kstat

\- cp Makefile-Linux24 Makefile

\- make

and it gave me this result

\[root@svr-vmesx-1 iomtr_kstat]# make

/usr/bin/gcc -O2 -Wall -D__KERNEL__ -DMODULE -DIOMTR_SETTING_KSTAT_PERCPU -DIOMTR_SETTING_NO_CPU_KHZ -mcmodel=kernel -isystem /lib/modules/`uname -r`/build/include -c -o iomtr_kstat.o iomtr_kstat.c

cc1: code model `kernel' not supported in the 32 bit mode

iomtr_kstat.c: In function `imkstat_ioctl':

iomtr_kstat.c:104: warning: unused variable `khz'

make: *** \[iomtr_kstat.o] Error 1

I sure you understand my Linux knowledge is basic Smiley Sad

0 Kudos
mcwill
Expert
Expert

I used the windows binary release in a Win2k3 VM for my testing.

As I understand it disk access for the service console isn't very optimised so normally runs slower than disk access within VMs.

0 Kudos
larstr
Champion
Champion

There's really no point in running this in the ESX server's Service Console. The Service Console VM is not tuned to give you any good storage performance. You will get much more usable results running it either from native linux or from a virtual machine.

Lars

0 Kudos