VMware Cloud Community
christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
Reply
0 Kudos
457 Replies
doubleH
Expert
Expert

here are my results. it seems that they are somewhat inline with happyhammer, but i'm surprised that AnthonyM has better performance using nics and an iSCSI initiator.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: W2K3 SP1 VM (1gb RAM) on ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL385 g2, 8GB RAM; 2 x AMD Opteron 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS100E x 1 / 12+2 SATA Disks / R50

SAN TYPE / HBAs : iSCSI, QLA4052C HBA Jumbo's and Flow Control

VMFS: 400GB LUN, 2MB Block Size

NETWORK TYPE: HP Procurve 5406 Jumbo's and Flow Control

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........____23____..........___2611___.........___81___

RealLife-60%Rand-65%Read.......____40____..........___1185___.........___9____

Max Throughput-50%Read.........____14____..........___4132___.........___129__

Random-8k-70%Read..............____48____..........___1045___.........___8____

EXCEPTIONS: VCPU Util.-32%;

##################################################################################

spelling

Message was edited by:

doubleH

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
christianZ
Champion
Champion

... and you have probably the same issue like happyhammer (jumbo and flow control simultaneously) - try the enable only flow control - you should see better results by sequential reads/writes.

Indeed with ms iscsi initiator you can get much better results - saw this too.

Thanks.

Reply
0 Kudos
doubleH
Expert
Expert

did you say this was a recommendation from EQL? if so could you provide a url to the document or if it was a support request could you please pass the sr# along to me so i can reference this with support.

thanks

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
christianZ
Champion
Champion

Here is the original answer from EQL:

Good morning,

With the Procurve 28xxx series we have found that running Jumbo frames + flow control doesn’t work well. The switch lacks sufficient buffers. Flow control is much more effective at providing the best performance. Also, you can increase the available buffers by enabling the ‘qos-passthrough-mode’ option. This will change the number of QOS queues from 4 to 2. However, it will require a reboot of the switch to take effect.

I’ve attached our network guidelines which shows our suggested method of configuring multiple switches. In the diagram there is a small line between the switches. This represents a set of trunked ports between them. At a minimum there should be three and we typically suggest four for best performance.

I’ve also attached the current draft of our FAQ for VMware. Are you using Qlogic HBAs? If so there are some settings you should check as well. They are listed in the FAQ.

Lastly is the Window disk timeout settings document. In the guest you should extend the Windows disk timeout to 60 seconds.

Please let me know if this answers your question. If not, please reply-all to this e-mail.

Regards,

Don[/i]

The case nr. was: 9528

And indeed changing of it gets more throughput by seq. r/w.

Reply
0 Kudos
doubleH
Expert
Expert

thank you very much. i'll see if they recommend this on the 5406's as well.

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
doubleH
Expert
Expert

i callled support and they confirmed what you have said (not that i doubted you). there is a 2nd reason in addition to the buffer reason you provided. it is that ESX doesn't support jumbo frames in 3.0.1.

i'll go ahead and turn off jumbo's on my storage vlan. the only problem is that i'll have to take down the esx servers to take jumbo's off of the HBA's in the bios. i'll run the test again and see what difference it makes.

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
doubleH
Expert
Expert

here are the new stats after disabling jumbos on the switch ports and HBA's....

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: W2K3 SP1 VM (1gb RAM) on ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL385 g2, 8GB RAM; 2 x AMD Opteron 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS100E x 1 / 12+2 SATA Disks / R50

SAN TYPE / HBAs : iSCSI, QLA4052C HBA

VMFS: 400GB LUN, 2MB Block Size

NETWORK TYPE: HP Procurve 5406 with Flow Control

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........____16____..........___3561___.........___111___

RealLife-60%Rand-65%Read.......____39____..........___1188___.........___9____

Max Throughput-50%Read.........____9____...........___6303___.........___197__

Random-8k-70%Read..............____47____..........___1050___.........___8____

EXCEPTIONS: VCPU Util.-32%

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
redmini
Contributor
Contributor

It was a PS3800. Sorry for the mistake.

Reply
0 Kudos
Atamido
Contributor
Contributor

So, am I to understand from redmini's results that running the 64 bit version of W2K3 R2 in a VM instead of the 32 bit version can give you a 1-30% performance hit on reads?

Reply
0 Kudos
murreyaw
Enthusiast
Enthusiast

I was seeing a 20% hit on my 64bit vms.

Reply
0 Kudos
Atamido
Contributor
Contributor

Wow, that is ugly. So unless you need 64bit for some reason, should one avoid it like the plague in a VM?

Reply
0 Kudos
mitchellm3
Enthusiast
Enthusiast

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Win2k3 SP2 VM w/ 1GB RAM on ESX 3.0.1

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: DL585G2, 32GB RAM; 4x Opteron 2.8GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS6800 x 1 / 7+1 Disks / R5

350GB VMFS LUN

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........_10.53_____..........___5586___.........____174___

RealLife-60%Rand-65%Read......___25.56___..........___1971___.........____15.4___

Max Throughput-50%Read..........__6.39____..........__8067____.........___252____

Random-8k-70%Read................._24.26____..........___1999___.........__15.62____

EXCEPTIONS: CPU Util.- 54, 41, 82, 43

##################################################################################

Will post results from DS4800 soon.

Message was edited by: mitchellm3

mitchellm3

Reply
0 Kudos
christianZ
Champion
Champion

That's great, those two models weren't there.

Thanks.

Reply
0 Kudos
Remko_N
Contributor
Contributor

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - vRAID1

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz

STORAGE TYPE / INFRA: HP EVA 4000 / 4 Gbps

DISK NUMBER / RAID LEVEL: 14 x 146GB FC / vRAID 1

VMFS: 200GB LUN, 1 MB Block Size

#####################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\#####################################################################################

Max Throughput-100%Read.....___5.0____..........__11168___.........__349.0___

RealLife-60%Rand-65%Read....__11.8____..........___2808___.........___21.9___

Max Throughput-50%Read......__14.6____..........___2974___.........___92.9___

Random-8k-70%Read...........__11.9____..........___2408___.........___18.8___

EXCEPTIONS: CPU Util. 68-69-57-74%;

Reply
0 Kudos
Remko_N
Contributor
Contributor

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - vRAID5

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz

STORAGE TYPE / INFRA: HP EVA 4000 / 4 Gbps

DISK NUMBER / RAID LEVEL: 14 x 146GB FC / vRAID 5

VMFS: 300GB LUN, 1 MB Block Size

#####################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\#####################################################################################

Max Throughput-100%Read.....___4.7____..........__11158___.........__348.6___

RealLife-60%Rand-65%Read....__13.5____..........___2149___.........___16.8___

Max Throughput-50%Read......__41.6____..........___1203___.........___37.6___

Random-8k-70%Read...........__13.5____..........___1855___.........___14.5___

EXCEPTIONS: CPU Util. 82-75-35-79%;

Reply
0 Kudos
Remko_N
Contributor
Contributor

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - DAS

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz

STORAGE TYPE: DAS (HP SmartArray P400 512MB)

DISK NUMBER / RAID LEVEL: 2 x 72GB 15K SAS / RAID 1+0

VMFS: 40GB, 1 MB Block Size

#####################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\#####################################################################################

Max Throughput-100%Read.....___5.7____..........__10067___.........__314.6___

RealLife-60%Rand-65%Read....__59.5____..........____818___.........____6.4___

Max Throughput-50%Read......___5.7____..........___9954___.........__311.0___

Random-8k-70%Read...........__61.1____..........____812___.........____6.3___

EXCEPTIONS: CPU Util. 50-38-50-31%;

Reply
0 Kudos
christianZ
Champion
Champion

Thanks for that.

What I observed was the difference by "Max Throughput-50%Read" on Vraid1 and Vraid5 - really large.

Reply
0 Kudos
ahasen
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VMWARE ESX 3.01 - VM Windows Server 2003R2 Standard

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: 3 x Dell PE2900, 16GB RAM; 2x XEON 5345, 2,33 GHz, QuadCore

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Datacore SANMelody 2.02 / 7+1 Disk / R5

STORAGE HARDWARE: 2 x Dell PE2900 / internal SAS Harddisks 300GB/10K / FC attached to ESX (Redundant)

FC Switch: 2 x Brocade 200E / FC HBA: Qlogic 2462

VMFS: 300GB / 1GB Blocksize

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........_____4.50___.........._11974__........ ____374____

RealLife-60%Rand-65%Read......._____0.59___.........._18125__.........____142____

Max Throughput-50%Read........._____2.01___.........._15181__.........____474____

Random-8k-70%Read.............._____0.48___.........._18834__.........____147____

EXCEPTIONS: CPU Util.-98%;

Reply
0 Kudos
larstr
Champion
Champion

How much memory on your storage server?

However, I suspect that your results are not as accurate as we would prefer (as we see others (including me) with high cpu load).

Is it possible to get any stats from the SanMelody side while doing these tests?

Lars

Reply
0 Kudos
christianZ
Champion
Champion

Agreeded

either cpu high utilisation causes incorrect time cycles or the whole test file sits in host cache - maybe possible to verify the results by san melody statistics (to check the disks activities, response time,...)

Reply
0 Kudos