christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
0 Kudos
457 Replies
christianZ
Champion
Champion

First the tests results are more theoretical - shows what the storage could achieve - all here tested in one vm, that means one vm could run so fast.

In practice you won't see e.g. 300 MB/s by seq. read from one vm (real is 50 MB/s e.g. by backup) - but as more MB/s you see here as more you can achieve in reality.

The most real test seems to be the "RealLife-60%Rand-65%Read" - in normal life you have random and sequential connections mixed (often 60/40 %).

I have tested here e.g. one physical server with Infortrend subsystem - that gives you comparison what such system can reach (that subsystem could serve many servers simultan - but isn't any "enterprise" one; by backup of this Windows server over 1Gb network I reach ~3 GB/min reading - bottle-neck here is the network and the write speed on the backup server / b2d here) - you can compare it to results on your vm.

One doesn't forget that there are subsystem with active/active storage processors (e.g. EVA4000/6000, EMC, Infortrend too but not EQL), so your system can theoretical reach more when you have enough disk spindles behind. The disk spindles are another breake here - as more you have as faster you can work - especially the "Response time" will be impacted thereby (one says the 30 ms or 80 ms shouldn't be overstepped)

Hope that's clear.

0 Kudos
christianZ
Champion
Champion

Happyhammer,

maybe you have an issue with Jumbo and FlowControl simultan enabled - I saw that by us (HP Procurve 2848 - Jumbo and Flowcontroll enabled - bad performance by seq. r/w) - after recommendation from EQL we enabled only flowcontrol - it is better as you can see below.

0 Kudos
Gabrie1
Commander
Commander

SERVER TYPE: W2003 VM on ESX3.0.1.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL585 G1, 32GB RAM; 4x dual core AMD 885, 2,66 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM SVC on IBM DS4100 / 6+1 Disks / R10

VMFS 100GB LUN, 1MB block size, HP/QLogic QLA2300 HBA.

All on SATA disks \!!! IBM DS4100 - Testlab configuration

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........___10.4370.........._5611.15__........._175.35___

RealLife-60%Rand-65%Read......___3.3357__..........__299.69__.........__2.34___

Max Throughput-50%Read..........__3.8708__..........__258.21__........._8.07__

Random-8k-70%Read.................___5.2375__.........._190.03___.........__1.48__

http://www.GabesVirtualWorld.com
0 Kudos
happyhammer
Hot Shot
Hot Shot

christian

thanks for that, my next set of tests are with normal ethernet frames with both ESX initiator andHBA's, i will run your icf as well and will post results

0 Kudos
conyards
Expert
Expert

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - VM VDI on 1MB Block Size VMFS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: XP VDI VM on ESX3.0.1.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM LS41, 32GB RAM; 4x dual core AMD OPTERON, 2.4 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS8300 / 7+1 Disks / R5

VMFS3 300GB LUN, 1MB block size, QLogic QLA 231X/2340 HBA.

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........___9.59.........._6130__........._191.59___

RealLife-60%Rand-65Read......___1.42__..........__2603__.........__31.30___

Max Throughput-50%Read..........__6.27__..........__3415__........._213.59__

Random-8k-70%Read.................___1.2__.........._2727___.........__30.4__

\##################################################################################

Number of active VMs on LUN = 40.

https://virtual-simon.co.uk/
0 Kudos
christianZ
Champion
Champion

Thanks for input -

you have a large cache in your DS8xxx - that has influence on random iops and response time - have you increased the testfile size (for now 8000000 sectors=4GB)??

0 Kudos
conyards
Expert
Expert

I kept the testfile size set at 8000000 sectors 4 GB, and used the test template that was included in a post at the start of this thread.

https://virtual-simon.co.uk/
0 Kudos
happyhammer
Hot Shot
Hot Shot

tests re run with MTU at 1500 and not jumbos,

appears tests RealLife-60%Rand-65%Read and Random-8k-70%Read are similar to before, the other 2 MAX tests have increased substantially.

Switch was Cisco 3750g does not make a lot of sense that jumbos are in fact slower!!!

Another observation from my tests im running is that on a physical BOX to the PS100 im getting FAR better results from a NIC than the 4052, although the HBA is using less CPU

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 8GB RAM; 2x XEON 5355(Quad), 2,66 GHz,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS100e x 1 / 14+2 SATA / R50

SAN TYPE / HBAs : iSCSI, QLA4052 HBA Flow Control

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read...............__16______..........___3569___.........___111____

RealLife-60%Rand-65%Read......___37_____..........___1302___.........____10____

Max Throughput-50%Read................____9____..........___6386___.........___199____

Random-8k-70%Read.................____42____..........___1183___.........____9____

EXCEPTIONS: VCPU Util. 29-36-42-32 %;

0 Kudos
christianZ
Champion
Champion

I saw your results, especially MaxThrouput50%Read - it seems to be very low in comparison to the others number, I think (VM on VMFS).

0 Kudos
davidbarclay
Virtuoso
Virtuoso

I noticed that too..but after I destroyed all the LUNs and deleted the VM Smiley Sad

I would like to retest, but this project had to move on!

There wasn't any other activity on the SAN at all either...

I wonder???

Dave

0 Kudos
christianZ
Champion
Champion

That is better (by sequential r/w) - as by me - it seems that "jumbos" don't do it faster - or by some switches "jumbo and flow control" shouldn't be enabled simultan.

Do you mean the performance from vm with iscsi hba is slower then physical server with nics ??

0 Kudos
happyhammer
Hot Shot
Hot Shot

sorry i meant just on Physical server(win 2003) to PS100, the nic's were outperforming the HBA's(tried all drivers, initiators, and combos and cant get nowhere near NIC performance) The HBA'a 4052 have latest firmware

On a 4K SEQ read was getting 102MBps/17%cpu on NIC and only 90MBps/8% on HBA

I would expect the HBA's to match or beat the nics in throughput and have less CPU overhead

0 Kudos
christianZ
Champion
Champion

I guess if you have strong server cpu - it can do the iscsi capsulation job (ip->tcp->iscsi->scsi) faster than the cpu on hba.

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS[b] VM ON ESX / VM SNAPSHOT ON[/b]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 7020, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 SAS10k / R50

SAN TYPE / HBAs : iSCSI, QLA4050 HBA

\##################################################################################

TEST NAME--


Av. Resp. Time ms
Av. IOs/sek

Av. MB/sek
----

\##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......___22_____..........___1230___.........____10____

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: VCPU Util. IN VM ~100%; IN VC 55%

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS VM ON ESX / LUN SNAPSHOT ON (EQL)[/b]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 7020, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 SAS10k / R50

SAN TYPE / HBAs : iSCSI, QLA4050 HBA

\##################################################################################

TEST NAME--


Av. Resp. Time ms
Av. IOs/sek

Av. MB/sek
----

\##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......___31_____..........___1760___.........____14____

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: VCPU Util. IN VM ~37%

0 Kudos
christianZ
Champion
Champion

Somebody asks if unload from vmfs2 makes any differences - I tested it and can't see any.

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS PHYS. WIN2003 / MS ISCSI INITIATOR[/b]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Win2003 Std. PHYS.

CPU TYPE / NUMBER: CPU / 2

HOST TYPE: Dell PE1950, 4GB RAM, 2x XEON 5130, 2,0 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 SAS10k / R50[/b]

SAN TYPE / HBAs : iSCSI; iSCSI softw. initiator in WIN2003, 1XGb NIC for iSCSI

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........____17____..........___3560___.........___112____

RealLife-60%Rand-65%Read......____13____..........___3550___.........____28____

Max Throughput-50%Read..........____10____..........___5950___.........____186___

Random-8k-70%Read.................____14____..........___3340___.........____26____

EXCEPTIONS:

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS PHYS. WIN2003 / MS ISCSI INITIATOR / 2 X PS3600X[/b]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Win2003 Std. PHYS.

CPU TYPE / NUMBER: CPU / 2

HOST TYPE: Dell PE1950, 4GB RAM, 2x XEON 5130, 2,0 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 2 / 2x14+2x2 SAS10k / R50 / LUN STRIPPED[/b]

SAN TYPE / HBAs : iSCSI; iSCSI softw. initiator in WIN2003, 1XGb NIC for iSCSI

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........____17____..........___3570___.........___112____

RealLife-60%Rand-65%Read......_____8____..........___5919___.........____46____

Max Throughput-50%Read..........____12____..........___4810___.........____150___

Random-8k-70%Read................._____8____..........___5720___.........____45____

EXCEPTIONS: Seq. r/w not as high as by single box

0 Kudos
christianZ
Champion
Champion

My comments:

\- as we can see here vm on iscsi compared to phys. server loses more throughput as its counterparts on fc

\- iscsi can't equal fc by sequential r/w (as expected)

\- iscsi "Response times" aren't higher than by fc

\- VM on VMWare server reaches round 50% of VM on ESX

0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS CORRECTION (LAST TEST BY CONCURRENT BACKUP)[/b]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Win2003 Std. PHYS.[/b]

CPU TYPE / NUMBER: CPU / 2

HOST TYPE: Dell PE2650, 4GB RAM, 2x XEON, 2,8 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: INFORTREND S12F-R1420 / 6+1 SAS15k / R5[/b]

SAN TYPE / HBAs : FC, QLA2460

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........____5_____..........___11090__.........___346____

RealLife-60%Rand-65%Read......____38____..........___1522___.........____12____

Max Throughput-50%Read.........._____8____..........___7160___.........____224___

Random-8k-70%Read.................____39____..........___1512___.........____12____

EXCEPTIONS:

0 Kudos