VMware Cloud Community
christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
Reply
0 Kudos
457 Replies
christianZ
Champion
Champion

No don't think so.

In addition - I have here a small city county running all servers (2 NW, 4 vms/VMServer with W2003, 300 users)over 10 SATAs (small SAN/FC with Infortend system), but having not any DB - IMHO the six SATAs would be little to few for any DB.

By DB the mean values are the "Response time" and "IOs/sek" - all a little sluggish by you.

Just my thought.

Reply
0 Kudos
acr
Champion
Champion

christianZ, when it comes to IOMeter you can get some bizare numbers..

This almost always happens when your dealing with .vmdk files.. and depends on your Storage and Cache setup..

Weve stopped using IOMeter for .vmdk tests (especially for iSCSI).

Attach a RAW Disk to the test VM (initailize it but dont format it), then throw the same tests at the VMs RAW Disk, the numbers always come in line and make much more sense..

Reply
0 Kudos
christianZ
Champion
Champion

Thanks for that

\- what is with perfmon (while running iometer load) - will it show incorrect values too ?

In addition I can always see the numbers in EQL statistics and up to now they were always equal (MB/s).

Reply
0 Kudos
admin
Immortal
Immortal

Borrowed an HP MSA1510i for a couple of weeks. First tests don't look too promising...this was with the SATA enclosure (MSA20) attached, also have an SCSI enclosure (MSA30) with a similar number of spindles (10k I believe) so will post how much improvement the better disks give.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Windows 2003 R2 VM 1024MB RAM 10GB VMDK.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell SC1430, 2GB RAM; 1 XEON 5120, 1,86 GHz, DualCore

STORAGE TYPE : MSA1510i Read Cache:128Mb Write Cache 128MB RAID5 7+1 SATA Disks (MSA20 cabinet). Single Controller.

CONNECTION: 2 x 1GBe, dedicated Cisco 3750 switch.

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

Max Throughput-100%Read........16.93511..........3473.355..........108.5424

RealLife-60%Rand-65%Read.....192.419944.........306.6018.......2.395327

Max Throughput-50%Read.......... 28.906212.......... 2054.455.......64.20172

Random-8k-70%Read................. 196.450229.........295.3122.......2.307127

CPU Util.-XX%; 65.449696% - 15.945951% - 30.388721% - 17.881071%

4GB IOmeter Datafile

Reply
0 Kudos
eliot
Enthusiast
Enthusiast

christianZ, when it comes to IOMeter you can get some

bizare numbers..

This almost always happens when your dealing with

.vmdk files.. and depends on your Storage and Cache

setup..

Weve stopped using IOMeter for .vmdk tests

(especially for iSCSI).

Attach a RAW Disk to the test VM (initailize it but

dont format it), then throw the same tests at the VMs

RAW Disk, the numbers always come in line and make

much more sense..

I cant say that ive seen that - performance is very similar between RDM and VMDK (i connect both into the same vm and test).

Reply
0 Kudos
christianZ
Champion
Champion

As acr mentioned I have seen such bizar numbers too - therefore I try confirm my results over other sources (e.g. perfmon, or statistics direct on storage subsystem).

But your are right - normally you won't see any significant differences, I think.

Reply
0 Kudos
acr
Champion
Champion

Im not to sure, but again it can depend on the SAN and workload tests..

If your testing sequencial loads then it can get done it cache which will impact true readings..?

Reply
0 Kudos
christianZ
Champion
Champion

Thanks for that -

in the past I had to select an entry level storage for one of our customer -

there were MSA1000 and Infortrend as candidates - I choosed the Infortrend - it is much faster than hp.

The Infortrend systems (http://www.infortrend.com/europe/main/2_product/products.asp) could be interesting as entry level systems - when they get the VMware certifications - maybe next months.

Their management is not on "Enterprise level" but they are fast and there are models with redundant controllers too (active/active).

Reply
0 Kudos
larstr
Champion
Champion

SERVER TYPE: VM @ ESX3.01, MS Windows 2003R2,

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM X3755, 16GB RAM; 2x Opteron 8218, 2,6 GHz, DC,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM ServeRAID, 2xSAS, RAID1 (local storage)

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sec---Av. MB/sec----

\##################################################################################

Max Throughput-100%Read........______19__..........____3086__.........___96.46__

RealLife-60%Rand-65%Read......._____133__.........._____382__.........____2.98__

Max Throughput-50%Read.........______91__.........._____662__.........___20.69__

Random-8k-70%Read..............______91__.........._____598__.........____4.67__

EXCEPTIONS: CPU Util. 27%;

Reply
0 Kudos
larstr
Champion
Champion

SERVER TYPE: VM @ VMware Server 1.0.1, MS Windows 2003,

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP ML350G5, 8GB RAM; 2x Xeon 5130, 2 GHz, DC,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP Smart Array E200+BBWC, 6x250GB SATA, RAID5 (local storage)

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sec---Av. MB/sec----

\##################################################################################

Max Throughput-100%Read........____0.49__..........___10713__.........___334.8__

RealLife-60%Rand-65%Read......._____170__.........._____342__.........____2.67__

Max Throughput-50%Read.........____0.63__..........___10856__.........___339.2__

Random-8k-70%Read.............._____182__.........._____313__.........____2.44__

EXCEPTIONS: CPU utilization on both max throughput tests were close to 100% so I'm not

sure that those results are accurate. Cpu util on the other tests were ~18%.

Reply
0 Kudos
christianZ
Champion
Champion

Thanks for that.

I would say the cpu utilization is ok - your system reached imposing ios numbers by seq. r/w - the vm must perform them.

When you see my first test with vm you will see a high cpu utilization too, especially by seq. tests where the ios were high.

Reply
0 Kudos
meistermn
Expert
Expert

Till now the ibm DS8300 is the fasted SAN or not?

Reply
0 Kudos
christianZ
Champion
Champion

Yes indeed it seems to be the fastest for now (and everlasting ?).

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

IBM DS8300:

RealLife-60%Rand-65%Read......___1.42__..........__2603__.........__31.30___

EMC DMX3000:

RealLife-60%Rand-65%Read......___1.8_____..........___2170___.........___17_____

The question here could be which system could serve more guests with such performance .

They both are playing of course in other league - can profit from the large cache (32 GB).

I notice the MB/s number of 31.10 - that seems to be a bug (by block size 8kB).

Reply
0 Kudos
larstr
Champion
Champion

I would say the cpu utilization is ok - your system

reached imposing ios numbers by seq. r/w - the vm

must perform them.

Keep in mind that the clock inside the VM becomes very unreliable under high cpu load. That's why I don't think these IOPS are too reliable. The VM surely performed these IOs, but the guest OS doesn't really have an accurate answer to how long time this took. If the cpu load is high the timing inside the guests is typically more wrong than during light loads. Looking at the load inside the guests from the hosts perspective normally gives a better picture. Looking at these numbers as provided by the guests will give us an idea about the performance, but it's not the 100% answer.

The document Timekeeping in VMware Virtual Machines[/url] gives some further details on this issue.

Lars

Reply
0 Kudos
larstr
Champion
Champion

The following test is run from the host os where the previous VMware Server VM was tested. The file system tested is the one storing that virtual machine.

SERVER TYPE: Host with VMware Server 1.0.1, CentOS x64, 2 running VMs

CPU TYPE / NUMBER: pCPU / 4

HOST TYPE: HP ML350G5, 8GB RAM; 2x Xeon 5130, 2 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP Smart Array E200+BBWC, 6x250GB SATA, RAID5 (local storage)

\#################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sec---Av. MB/sec----

\##################################################################################

Max Throughput-100%Read........____18.3__..........____3148__.........____98.4__

RealLife-60%Rand-65%Read......._____333__.........._____178__........._____1.4__

Max Throughput-50%Read.........______24__..........____2374__.........____74.2__

Random-8k-70%Read.............._____417__.........._____142__........._____1.1__

EXCEPTIONS:

Lars

Reply
0 Kudos
christianZ
Champion
Champion

Yes, you are right - in this test

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS VM ON ESX / VM SNAPSHOT ON

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I saw it too (util. vcpu ~100 %; cpu in VC ~55%).

What I always do is correcting the ios/sek in VC statistics or directly in storage (you can see there only MB/sek but when one knows the block size it is possible to calculate the ios/sek).

Reply
0 Kudos
murreyaw
Enthusiast
Enthusiast

Has anyone created a virtual appliance for this yet?

Reply
0 Kudos
pauliew1978
Enthusiast
Enthusiast

Ok I have tweaked my set up from 1 6 spindle raid 10 to a 10 spindle raid 10 set. The differences are quite big!. Hopefully I should be able to run a few vms on this set up. The only thing I am not sure of those is what the percentage means on these stats (i.e. 60 percent read). How does this percentage relate to number of requests and bandwidth etc....

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM (ms 2000 server) 2gb vram.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: hp dl385, 6GB RAM for host esx server; 2x amd opteron (dual core), 2.4 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: sanmelody server on win2003 r2 raid 10 (1tb lun) 2gb ram (used as cache by sanmelody)

iscsi, sata disks 10 spindles

\##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

\##################################################################################

Max Throughput-100%Read........___17.2_______..........___3357_______.........____105______

RealLife-60%Rand-65%Read......_____82.7_____.........._____696_____.........____5.43______

Max Throughput-50%Read.........._____18.46_____..........______3117____.........____97.4______

Random-8k-70%Read.................___77_______..........____762______.........___5.95_______

EXCEPTIONS: CPU Util.-XX%;

Reply
0 Kudos
christianZ
Champion
Champion

>RealLife-60%Rand-65%Read<

In this test are 60 % of ios random - 40 % sequential, 65 % of ios are reads the rest writes - all that is similar to conventional disk's access.

Hope that's clear.

Reply
0 Kudos
InsaneGeek
Enthusiast
Enthusiast

I don't want to make it that much more complicated than it already is but I'd really suggest some additional notations to make the numbers more comparable (there are more but I think these tend to cover most bases).

Additionally the length of time & data size probably both need to be significantly larger, as >8gig of cache is in many storage arrays and is likely to have a significant portion of all transaction happen within the cache in the storage array if the test is only ran for 5 minutes.

1) Amount of memory used in the guest (I'd probably suggest using a fixed number like 256mb)

2) Amount of cache used in the storage array

3) Whether or not the storage array is shared or dedicated

4) If array shared report # hosts sharing the array, and average number of IOPs and or throughput to array prior to and after the test

5) Whether or not the physical spindles are shared or dedicated

6) If spindles are shared report # hosts sharing the spindles average number of IOPs and or throughput to spindles prior, during and after

7) Which part of the disk platter the storage is on as outter to inner speeds can be upto 2x difference (outter 25%, outter middle 25%, inner middle 25%, inner 25%)

😎 Any known tweeks (dedicated cache toward specific spindles, short-stroking drives, QOS settings for luns in array, etc)

9) Guest & Host OS version + patch number

10) Drive make & model

Don't want to sound like I'm thinking this isn't a good idea (because I do), just want to add a little bit more detail and make the numbers more appropriate.

Reply
0 Kudos