Attention!
Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.
The new thread is available here:
[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]
My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any
warranty.
If anybody shouldn't be agreed with some results then he can make own tests and presents
his/her results too.
I hope this way to classify the different systems and give a "neutral" performance comparison.
Additionally I will mention that the performance is one of many aspects to choose the right system.
The others could be e.g.
\- support quality
\- system management integration
\- distribution
\- self made experiences
\- additional features
\- costs for storage system and infrastructure, etc.
There are examples of IOMETER Tests:
=====================================
\######## TEST NAME: Max Throughput-100%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
\######## TEST NAME: RealLife-60%Rand-65%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
\######## TEST NAME: Max Throughput-50%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
\######## TEST NAME: Random-8k-70%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
The global options are:
=====================================
Worker
Worker 1
Worker type
DISK
Default target settings for worker
Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
Disk maximum size,starting sector
8000000,0
Run time = 5 min
For testing the disk C is configured and the test file (8000000 sectors) will be created by
first running - you need free space on the disk.
The cache size has direct influence on results. By systems with cache over 2GB the test
file should be increased.
LINK TO IOMETER:
Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s
To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks
Here the config file *.icf
\####################################### BEGIN of *.icf
Version 2004.07.30
'TEST SETUP ====================================================================
'Test Description
IO-Test
'Run Time
' hours minutes seconds
0 5 0
'Ramp Up Time (s)
0
'Default Disk Workers to Spawn
NUMBER_OF_CPUS
'Default Network Workers to Spawn
0
'Record Results
ALL
'Worker Cycling
' start step step type
1 5 LINEAR
'Disk Cycling
' start step step type
1 1 LINEAR
'Queue Depth Cycling
' start end step step type
8 128 2 EXPONENTIAL
'Test Type
NORMAL
'END test setup
'RESULTS DISPLAY ===============================================================
'Update Frequency,Update Type
4,WHOLE_TEST
'Bar chart 1 statistic
Total I/Os per Second
'Bar chart 2 statistic
Total MBs per Second
'Bar chart 3 statistic
Average I/O Response Time (ms)
'Bar chart 4 statistic
Maximum I/O Response Time (ms)
'Bar chart 5 statistic
% CPU Utilization (total)
'Bar chart 6 statistic
Total Error Count
'END results display
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
Max Throughput-100%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
'Access specification name,default assignment
RealLife-60%Rand-65%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
'Access specification name,default assignment
Max Throughput-50%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
'Access specification name,default assignment
Random-8k-70%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
'END access specifications
'MANAGER LIST ==================================================================
'Manager ID, manager name
1,PB-W2K3-04
'Manager network address
193.27.20.145
'Worker
Worker 1
'Worker type
DISK
'Default target settings for worker
'Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
'Disk maximum size,starting sector
8000000,0
'End default target settings for worker
'Assigned access specs
'End assigned access specs
'Target assignments
'Target
C:
'Target type
DISK
'End target
'End target assignments
'End worker
'End manager
'END manager list
Version 2004.07.30
\####################################### ENDE of *.icf
TABLE SAMPLE
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM or PHYS.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50
##################################################################################
TEST NAME--
##################################################################################
Max Throughput-100%Read........__________..........__________.........__________
RealLife-60%Rand-65%Read......__________..........__________.........__________
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS: CPU Util.-XX%;
##################################################################################
I hope YOU JOIN IN !
Regards
Christian
A Google Spreadsheet version is here:
Message was edited by:
ken.cline@hp.com to remove ALL CAPS from thread title
Message was edited by:
RDPetruska
Added link to Atamido's Google Spreadsheet
As InsaneGeek points out, there are numerous mistakes one can make to improve or degrade an array's performance. I'd like to add:
\* run the tests for 15 minutes and throw away the results. Now record.
\* if one had enough time, it would be interesting to allow the tests to continue capturing results every 30 minutes over a couple days - but no one has that kind of time. If one did, report the first and last window recorded.
\* ensure the files accessed by the test realistically fill the available storage. That is, if there is 1TB of disk space available, the files should fill at least 60% (for example) of the available space. Then the random access should be over this entire space. Sequential files that simulate log files should be much smaller - say 1GB.
Does anybody have a NetApp FAS? EVA8000? CX3? Sun6540?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS VM ON ESX / SAN-MELODY 2.0.1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: IBM xSeries 235, 4 GB RAM; 2x XEON HT 2,4 GHz
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody on IBM xSeries 235 (as above) /
5 x SCSI U160 disks (r/w caching on) / NMV Volume with raw disks, 3 GB cache, 1 x 1Gb NIC for iSCSI
SAN TYPE / HBAs : iSCSI, ESX software initiator, 1 x 1Gb NIC
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........__32______..........__1780____.........___56_____
RealLife-60%Rand-65%Read......___43_____..........___1360___.........____11____
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS: Host CPU Util. IN VM ~75 % (by 1); Test file = 15 GB on disk 😧
##################################################################################
Results verified over "Perfmon" on San-Melody server (checked the ios on disks ~ 200 ios/sek).
Tests over 15 minutes got the same numbers.
Well I didn't want to make this test a "Mega Test" but a small verifying of owns results and more details should be of course helpful.
The cache of storage subsystems has of course influence on the results but one should say that the cache is one of more important properties of such systems. By systems with e.g. 32 GB cache it is not practicable to create e.g. 100 GB test file - by such systems one should create the load over many clients - and by us the "Response Time" is the most significant(together with IOs/sek)- in this case both values will be generate on cache ios - that's true.
The DS8000 and EMC Symetrix series are playing of course in other league here.
Now I myself saw by testing of SanMelody that the "Cache Algorithm" seems to function very well and even with 15 GB test file - with 3 GB cache ( and testing time of 15 min.) I could see only small disk's activities - so tested many times until I could see a lot of activieties on disks and all these were of course verified over "perfmon" on the SanMelody server.
I'm not so sure about the SanMelody cache, but maybe it depends on the disk controller. I did a test recently comparing SanMelody Lite to SanMelody on the same hardware so the only variable that changed was the 128MB cache on the Lite version compared to approx 1.3GB cache on the full version.
The difference was at most a 10% increase in speed.
I'll dig out the values and post them.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS VM ON ESX / SAN-MELODY 2.0.1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell PE 1950 4GB
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody Full on Xeon 5130 whitebox /
RAID 10 - 4 x SATA 10k disks (r/w caching on) - 256KB Stripe/ 1.3 GB cache
SAN TYPE / HBAs : iSCSI, ESX software initiator, 2 x 1Gb Team NIC
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........__18______..........__3133____.........___99_____
RealLife-60%Rand-65%Read......___72_____..........___834___.........____6.3____
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS VM ON ESX / SAN-MELODY LITE 2.0.1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell PE 1950 4GB
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Win2003/SanMelody Lite on Xeon 5130 whitebox /
RAID 10 - 4 x SATA 10k disks (r/w caching on) - 256KB Stripe/ 128 MB cache
SAN TYPE / HBAs : iSCSI, ESX software initiator, 2 x 1Gb Team NIC
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........__19______..........__3058____.........___95_____
RealLife-60%Rand-65%Read......___78_____..........___772___.........____6.0____
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS:
Thanks for that - so we can see now that probably the numbers from Pauliew1978 are not optimal (as I guessed) and my test box with 5 scsi disks and 3 GB cache runs fast (although old hardware here).
Pauliew1978 - you must check your configuration - the numbers should be better by your hardware - I think.
Yes indeed it seems to be the fastest for now (and
everlasting ?).
TEST NAME----------------
-Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----IBM DS8300:
RealLife-60%Rand-65%Read......___1.42__..........__260
3__.........__31.30___
EMC DMX3000:
RealLife-60%Rand-65%Read......___1.8_____..........\___
2170___.........___17_____
The question here could be which system could serve
more guests with such performance .
They both are playing of course in other league - can
profit from the large cache (32 GB).
I notice the MB/s number of 31.10 - that seems to be
a bug (by block size 8kB).
I was running 180 VM's over 9TB's while that test was running....
Up till a little while ago I was getting errors "Low I/O Detected" lololol....
I love this thing...No matter how hard I push the Vm's it's still falling asleep.
Which I think makes the ESX hosts life a lot easier as they don't have to buffer so much I/O's which keeps CPU down.
I just keeping running into low memory errors, never any cpu or i/o errors.
One other side note. I only have 144 of the 572 Drives it can Hold 😜
Although i think it'd be neck and neck running these two boxes at full capacity.
Also keep in mind now our DMX3000 is only DMX-2 Archietecture...
The DMX3 is even faster and more scalable...Hopefully I'll have one of those this year...
Message was edited by:
CWedge@Amsa
Woh - you are lucky men with such system.
Of course the two systems couldn't be compared with this test !!
As I mentioned they both are playing in other league - or maybe the EMC DMX3000 plays in its own league ?
Here are my results on 3 different LUN types:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 2.5.2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 3 x 250GB 7.5K SATA + 1 / R5
SAN TYPE / HBAs : FC , QLA4050 HBA
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read....._______21________......____2792___.......____87____...
RealLife-60%Rand-65%Read...._______364_______......____123____.......____0.96__...
Max Throughput-50%Read......_______14________......____553____.......____17____...
Random-8k-70%Read..........._______411_______......____109____.......____0.85__...
EXCEPTIONS: VCPU Util. 51-25-15-18 %;
##################################################################################
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 2.5.2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 6 x 146GB 15K FASTT + 1 / R5
SAN TYPE / HBAs : FC , QLA4050 HBA
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read....._______21________......____2179___.......____68____...
RealLife-60%Rand-65%Read...._______35________......____501____.......____3.91__...
Max Throughput-50%Read......_______16________......____499____.......____15.6__...
Random-8k-70%Read..........._______25________......____501____.......____3.91__...
EXCEPTIONS: VCPU Util. 96-36-41-32 %;
##################################################################################
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM ON ESX 2.5.2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: IBM xSeries 365, 16GB RAM; 4x XEON MP, 3,00 GHz, HT
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4100 / 6 x 146GB 15K FASTT + 1 / R10
SAN TYPE / HBAs : FC , QLA4050 HBA
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read....._______25________......____2356___.......____73____...
RealLife-60%Rand-65%Read...._______49________......____1100___.......____8.59__...
Max Throughput-50%Read......_______32________......____1816___.......____28.4__...
Random-8k-70%Read..........._______46________......____1222___.......____9.54__...
EXCEPTIONS: VCPU Util. 48-40-46-37 %;
Thanks for that - only one hint:
SAN TYPE / HBAs : FC , QLA4050 HBA
I suppose you have a FC hba - so that could be e.g. the qla24xx (qla4050 is the iscsi hba).
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM on ESX 3.01
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: BL25P, 16GB RAM; 2 x opteron 280 DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Netapp FAS3020 / 12+2 Disks / Raid DP (r6)
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........___10_____..........__5106___.........___159_______
RealLife-60%Rand-65%Read......____80____..........____642______.........___3______
Max Throughput-50%Read..........___7_______..........___7918_______.........____124___
Random-8k-70%Read.................___80_______..........__652________.........___4______
EXCEPTIONS: CPU Util.-XX%;
Yes, sorry, it is a QLA2340 HBA.
Hi Christianz,
I am rechecking my setup as we speak. There was a problem with multipathing io set up on my sanmelody config. SO maybe it will be slightly quicker now. i am running some tests as we speak. Also my disks were not set to advanced performance in wondows disk manager so It may have stopped sanmelody using the faster features that the disk can do.
cheers and thanks for organising this test it has helped me get some real life figures on other setups.
Paul
Hi Christianz,
Just out of interest is there a way of controlling the amount of cache within sanmelody?
thanks,
Paul
Until now I haven't found it.
Wow - the first NetApp here. Thanks.
Have you sata or fc disks in it ?
Are you using FC or iSCSI (or maybe NFS) as connection type - when iSCSI then iscsi hba or software initiator ?
Basic question:
1) where are you running this IOmeter ? on a VM or on the ESX server itself
2) I tried to compile the last version of iometer-2006_07_27.linux.i386-bin
\- copied the .tgz file in /usr/iometer
\- untar with : tar xvfz something.tar.gz
\- go to the subfolder named src/iomtr_kstat
\- cp Makefile-Linux24 Makefile
\- make
and it gave me this result
\[root@svr-vmesx-1 iomtr_kstat]# make
/usr/bin/gcc -O2 -Wall -D__KERNEL__ -DMODULE -DIOMTR_SETTING_KSTAT_PERCPU -DIOMTR_SETTING_NO_CPU_KHZ -mcmodel=kernel -isystem /lib/modules/`uname -r`/build/include -c -o iomtr_kstat.o iomtr_kstat.c
cc1: code model `kernel' not supported in the 32 bit mode
iomtr_kstat.c: In function `imkstat_ioctl':
iomtr_kstat.c:104: warning: unused variable `khz'
make: *** \[iomtr_kstat.o] Error 1
I sure you understand my Linux knowledge is basic
I used the windows binary release in a Win2k3 VM for my testing.
As I understand it disk access for the service console isn't very optimised so normally runs slower than disk access within VMs.
There's really no point in running this in the ESX server's Service Console. The Service Console VM is not tuned to give you any good storage performance. You will get much more usable results running it either from native linux or from a virtual machine.
Lars