VMware Cloud Community
christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
Reply
0 Kudos
457 Replies
bjselman
Contributor
Contributor

I am out of the office today, Thursday, August 7, attending a virtualization conference. If you have an IT emergency, please contact Scott Holmes.

Thanks,

BJ

Reply
0 Kudos
zbenga
Enthusiast
Enthusiast

i had the same errors on Clariion it's to do with the kernel in linux guests, will try latest kernel

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

This my result for my new environment. I will do another test later with juste one array.

Are these result good for an Equallogic Product?

SERVER TYPE: PHYS.

CPU TYPE / NUMBER: XEON 3.2GHz / 1

HOST TYPE: Proliant ML350 G3, 4GB RAM; 1x XEON, 3.2 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 2 / 2 x (14+2) 1TB Disks SATA / R10

MS Initiator, flow control enable and jumbo frame enable. 2 x 1GB Nics

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........___9.54___..........__5542.56__.........___170.39___

RealLife-60%Rand-65%Read......____4.8___.........._10945.23__.........___85.51___

Max Throughput-50%Read..........___8.6____..........__5660.31__.........__176.88__

Random-8k-70%Read.................___4.5____.........._11656.56__.........__91.07___

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

Reply
0 Kudos
wpatton
Expert
Expert

Chrisfmss,

You can do an easy compare and search of other results using this online doc:

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

*Disclaimer: VMware Employee* If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".
Reply
0 Kudos
larstr
Champion
Champion

This my result for my new environment. I will do another test later with juste one array.

Are these result good for an Equallogic Product?

It looks like you're having some timing problems. Your mixed IO results are twice as high as your max throughput tests and that's not normal behavior. Either you have a too little test file (so you only test against RAM instead of disk) or you have some timing problems. How much RAM are there on your SAN controllers?

Lars

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

There is my test with only one array.

SERVER TYPE: PHYS.

CPU TYPE / NUMBER: XEON 3.2GHz / 1

HOST TYPE: Proliant ML350 G3, 4GB RAM; 1x XEON, 3.2 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 1 / (14+2) 1TB Disks SATA / R10

MS Initiator, flow control enable and jumbo frame enable. 2 x 1GB Nics

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........___9.4___..........__5091.55__.........___159.11___

RealLife-60%Rand-65%Read......___11.8___..........__4314.28__.........___33.71___

Max Throughput-50%Read..........___9.0____..........__5925.95__.........__185.19__

Random-8k-70%Read.................___12.2____..........__4186.12__.........__32.70___

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

Reply
0 Kudos
christianZ
Champion
Champion

+++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS / PRODUCTION INFRASTRUCTURE

+++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.5 Upd. 1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 16GB RAM; 2x XEON X5450, 3,0 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 SAS10k / R50 (FW 3.3.1 w/o thinprov.)

SAN TYPE / HBAs : iSCSI, QLA4050 HBA

TEST NAME-----------------Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........__17______..........___3540___.........___110____

RealLife-60%Rand-65%Read......___36_____..........___1590___.........____12____

Max Throughput-50%Read..........____11____..........___5620___.........___175____

Random-8k-70%Read.................____39____..........___1468___.........____12____

EXCEPTIONS: VCPU Util. <22%, all tests running on prod. systems; storage unit (1) cap. utilization > 50 %;

runnings machines: vms ca. 45/(ca. 100 configured), ph. ca. 5, in production since 03.2007

These tests are running on not stripped volume (only one member) - it is a large difference

to the first test on fresh system. I wonder.

First what I noticed here was if I make the test on a new (free) esx host w/o any vm running on it then

I can reach much more (~2300 ios/sec by Reallife) - i.e. the performance degradation comes with

more running vms (on esx), not with storage saturation.

And I see a lot of new entries here from:

dalepa - interesting nfs analysis

saqp - test of Xtravirt xvs virtual san

happyhammer (again) - test of cx3-40

chrisfmss - test of Eql PS5000 with satas

Thanks to all for keeping this thread alive.

Reply
0 Kudos
christianZ
Champion
Champion

Ok. I wrote that your results are not so good but now I can see the results from happyhammer (the same box cx3-40) and can see much more as by your tests (and think these are much more realistic numbers for such storage).

Maybe there is some configuration option to make it faster - should be possible, I think.

Reply
0 Kudos
christianZ
Champion
Champion

>RealLife-60%Rand-65%Read......___11.8___..........__4314.28__.........___33.71___

I wonder about your results here - that all with satas. I'm a little sceptical, if it possible with satas.

Have here one new box with SAS 15k disks, want to test it as well.

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

I use MS initiator included in Hit Kit 3.1.1 with MPIO on Windows 2003 SP2 64 bits. Can the switch make a difference ? I use 2 Cisco 3750-E in one stack. Jumbo frame and flowcontrol on. Another thing i see from your setup, you are using Raid 50. I am using the new firmware 4.0.1.

Reply
0 Kudos
christianZ
Champion
Champion

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS / PRODUCTION INFRASTRUCTURE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: PH. Win 2003/64 sp2

CPU TYPE / NUMBER: PH. CPU / 2

HOST TYPE: Dell PE2950, 16GB RAM; 2x XEON X5450, 3,0 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000 x 1 / 14+2 SAS15k / R10 (FW 3.3.1 w/ thinprov.)

SAN TYPE / HBAs : iSCSI, QLA4050 HBA X 2, LB W/O JUMBO

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........____9,7____..........___6195___.........___194____

RealLife-60%Rand-65%Read......___12,8____..........__4032___.........___31,5____

Max Throughput-50%Read..........____9,7____...........___6152____.........___192____

Random-8k-70%Read.................____13,4___..........___3975___.........____31____

EXCEPTIONS: CPU Util. <10%,

##################################################################################

Could not reach the numbers of PS5000 with sata (new FW 4.0.1 here). Is that the new Fw that makes such difference (or maybe jumbos)? I wonder.

Reply
0 Kudos
bjselman
Contributor
Contributor

I am out of the office all week, September 15-19, attending VMworld 2008. Please contact Scott Holmes by email: smholmes@travelhost.com, or extension 115.

Thanks,

BJ

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

Have you try with MS Initiator ? What the size of the lun?

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

I have made another test with 4 nic and 2 ps5000e. It doesn't make such diffirence.

SERVER TYPE: PHYS.

CPU TYPE / NUMBER: XEON 3.2GHz / 1

HOST TYPE: Proliant ML350 G3, 4GB RAM; 1x XEON, 3.2 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 2 / 2 x (14+2) 1TB Disks SATA / R10

MS Initiator, flow control enable and jumbo frame enable. 4 x 1GB Nics

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........___8.4___..........__6180.46__.........___193.14___

RealLife-60%Rand-65%Read......____4.6___.........._11118.87__.........___86.87___

Max Throughput-50%Read..........___8.3____..........__6288.04__.........__196.50__

Random-8k-70%Read.................___4.3____.........._12044.71__.........__94.10___

EXCEPTIONS: CPU Util.-XX%;

Reply
0 Kudos
christianZ
Champion
Champion

Have you try with MS Initiator ?

There are 2 iscsi hbas (Qlogic) and the ms iscsi initiator is used only for multipathing/load balancing

What the size of the lun?

200 GB

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

My test was 10 GB. Can this make a difference ?

Reply
0 Kudos
christianZ
Champion
Champion

Woh. I'm still a little sceptical - that is better than the DS4800. Are you sure your testfile is 4GB and have you verified your results with e.g. "esxtop / d" direct on the console.

Reply
0 Kudos
chrisfmss
Enthusiast
Enthusiast

I use the IOmeter config file provided in this tread. What do you mean for the test file ?

Reply
0 Kudos
christianZ
Champion
Champion

I use the IOmeter config file provided in this tread. What do you mean for the test file ?

During the first test a 4 GB test file will be created. Please check on c: or d: if you have one.

Sorry for my mistrust Smiley Wink.

Reply
0 Kudos
asp24
Enthusiast
Enthusiast

SERVER TYPE: W2003 VM on ESX3.5.0.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Tyan Transport TA26, 32GB RAM; 2x quad core AMD 2352, 2,1 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Open-E DSS,

u160 external sata-scsi Infortrend A16U-G1A3 / 12+1 Disks / R5

VMFS 1900GB LUN, 1MB block size, SW Iscsi init.

Samsung F1 750GB disks, Dual Opteron 2 GHz, 4GB mem, LSI U320 controller on Open-E server.

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........___17.23.........._3334.96__........._104.21___

RealLife-60%Rand-65%Read......___76.39__..........__747.13__.........__5.84___

Max Throughput-50%Read..........__13.33__..........__4087.47__........._127.73__

Random-8k-70%Read.................___89.29__.........._643.06___.........__5.02__

This will soon be used for DR storage.

I have ordered an INFORTREND 3U 16-Bay iSCSI-to-SAS RAID ( S16E-G1430 ). I have also ordered

9x Seagate Cheetah 15k.5 300GB SAS drives and 9x Samsung F1 1TB drives (one spare of each type)

I hope I will be satisfied with the new Infortrend box. 4 x Gigabit's for ISCSI traffic. I think I will use

2x dedicated gigabit NICs on each host for iscsi traffic (6 hosts total, about 60 VM's,

mix of web, sql, exchange and terminal servers (RDP) )

I have not decided on RAID level for the 8 SAS-drives yet. Raid 5, 10 or 50. Comments?

Message was edited by: oreeh

Reason: fixed formatting to prevent thread display issues

Oliver Reeh

VMware Communities User Moderator

Reply
0 Kudos