Attention!
Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.
The new thread is available here:
[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]
My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any
warranty.
If anybody shouldn't be agreed with some results then he can make own tests and presents
his/her results too.
I hope this way to classify the different systems and give a "neutral" performance comparison.
Additionally I will mention that the performance is one of many aspects to choose the right system.
The others could be e.g.
\- support quality
\- system management integration
\- distribution
\- self made experiences
\- additional features
\- costs for storage system and infrastructure, etc.
There are examples of IOMETER Tests:
=====================================
\######## TEST NAME: Max Throughput-100%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
\######## TEST NAME: RealLife-60%Rand-65%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
\######## TEST NAME: Max Throughput-50%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
\######## TEST NAME: Random-8k-70%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
The global options are:
=====================================
Worker
Worker 1
Worker type
DISK
Default target settings for worker
Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
Disk maximum size,starting sector
8000000,0
Run time = 5 min
For testing the disk C is configured and the test file (8000000 sectors) will be created by
first running - you need free space on the disk.
The cache size has direct influence on results. By systems with cache over 2GB the test
file should be increased.
LINK TO IOMETER:
Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s
To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks
Here the config file *.icf
\####################################### BEGIN of *.icf
Version 2004.07.30
'TEST SETUP ====================================================================
'Test Description
IO-Test
'Run Time
' hours minutes seconds
0 5 0
'Ramp Up Time (s)
0
'Default Disk Workers to Spawn
NUMBER_OF_CPUS
'Default Network Workers to Spawn
0
'Record Results
ALL
'Worker Cycling
' start step step type
1 5 LINEAR
'Disk Cycling
' start step step type
1 1 LINEAR
'Queue Depth Cycling
' start end step step type
8 128 2 EXPONENTIAL
'Test Type
NORMAL
'END test setup
'RESULTS DISPLAY ===============================================================
'Update Frequency,Update Type
4,WHOLE_TEST
'Bar chart 1 statistic
Total I/Os per Second
'Bar chart 2 statistic
Total MBs per Second
'Bar chart 3 statistic
Average I/O Response Time (ms)
'Bar chart 4 statistic
Maximum I/O Response Time (ms)
'Bar chart 5 statistic
% CPU Utilization (total)
'Bar chart 6 statistic
Total Error Count
'END results display
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
Max Throughput-100%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
'Access specification name,default assignment
RealLife-60%Rand-65%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
'Access specification name,default assignment
Max Throughput-50%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
'Access specification name,default assignment
Random-8k-70%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
'END access specifications
'MANAGER LIST ==================================================================
'Manager ID, manager name
1,PB-W2K3-04
'Manager network address
193.27.20.145
'Worker
Worker 1
'Worker type
DISK
'Default target settings for worker
'Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
'Disk maximum size,starting sector
8000000,0
'End default target settings for worker
'Assigned access specs
'End assigned access specs
'Target assignments
'Target
C:
'Target type
DISK
'End target
'End target assignments
'End worker
'End manager
'END manager list
Version 2004.07.30
\####################################### ENDE of *.icf
TABLE SAMPLE
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM or PHYS.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50
##################################################################################
TEST NAME--
##################################################################################
Max Throughput-100%Read........__________..........__________.........__________
RealLife-60%Rand-65%Read......__________..........__________.........__________
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS: CPU Util.-XX%;
##################################################################################
I hope YOU JOIN IN !
Regards
Christian
A Google Spreadsheet version is here:
Message was edited by:
ken.cline@hp.com to remove ALL CAPS from thread title
Message was edited by:
RDPetruska
Added link to Atamido's Google Spreadsheet
No don't think so.
In addition - I have here a small city county running all servers (2 NW, 4 vms/VMServer with W2003, 300 users)over 10 SATAs (small SAN/FC with Infortend system), but having not any DB - IMHO the six SATAs would be little to few for any DB.
By DB the mean values are the "Response time" and "IOs/sek" - all a little sluggish by you.
Just my thought.
christianZ, when it comes to IOMeter you can get some bizare numbers..
This almost always happens when your dealing with .vmdk files.. and depends on your Storage and Cache setup..
Weve stopped using IOMeter for .vmdk tests (especially for iSCSI).
Attach a RAW Disk to the test VM (initailize it but dont format it), then throw the same tests at the VMs RAW Disk, the numbers always come in line and make much more sense..
Thanks for that
\- what is with perfmon (while running iometer load) - will it show incorrect values too ?
In addition I can always see the numbers in EQL statistics and up to now they were always equal (MB/s).
Borrowed an HP MSA1510i for a couple of weeks. First tests don't look too promising...this was with the SATA enclosure (MSA20) attached, also have an SCSI enclosure (MSA30) with a similar number of spindles (10k I believe) so will post how much improvement the better disks give.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: Windows 2003 R2 VM 1024MB RAM 10GB VMDK.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell SC1430, 2GB RAM; 1 XEON 5120, 1,86 GHz, DualCore
STORAGE TYPE : MSA1510i Read Cache:128Mb Write Cache 128MB RAID5 7+1 SATA Disks (MSA20 cabinet). Single Controller.
CONNECTION: 2 x 1GBe, dedicated Cisco 3750 switch.
TEST NAME--
Max Throughput-100%Read........16.93511..........3473.355..........108.5424
RealLife-60%Rand-65%Read.....192.419944.........306.6018.......2.395327
Max Throughput-50%Read.......... 28.906212.......... 2054.455.......64.20172
Random-8k-70%Read................. 196.450229.........295.3122.......2.307127
CPU Util.-XX%; 65.449696% - 15.945951% - 30.388721% - 17.881071%
4GB IOmeter Datafile
christianZ, when it comes to IOMeter you can get some
bizare numbers..
This almost always happens when your dealing with
.vmdk files.. and depends on your Storage and Cache
setup..
Weve stopped using IOMeter for .vmdk tests
(especially for iSCSI).
Attach a RAW Disk to the test VM (initailize it but
dont format it), then throw the same tests at the VMs
RAW Disk, the numbers always come in line and make
much more sense..
I cant say that ive seen that - performance is very similar between RDM and VMDK (i connect both into the same vm and test).
As acr mentioned I have seen such bizar numbers too - therefore I try confirm my results over other sources (e.g. perfmon, or statistics direct on storage subsystem).
But your are right - normally you won't see any significant differences, I think.
Im not to sure, but again it can depend on the SAN and workload tests..
If your testing sequencial loads then it can get done it cache which will impact true readings..?
Thanks for that -
in the past I had to select an entry level storage for one of our customer -
there were MSA1000 and Infortrend as candidates - I choosed the Infortrend - it is much faster than hp.
The Infortrend systems (http://www.infortrend.com/europe/main/2_product/products.asp) could be interesting as entry level systems - when they get the VMware certifications - maybe next months.
Their management is not on "Enterprise level" but they are fast and there are models with redundant controllers too (active/active).
SERVER TYPE: VM @ ESX3.01, MS Windows 2003R2,
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: IBM X3755, 16GB RAM; 2x Opteron 8218, 2,6 GHz, DC,
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM ServeRAID, 2xSAS, RAID1 (local storage)
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........______19__..........____3086__.........___96.46__
RealLife-60%Rand-65%Read......._____133__.........._____382__.........____2.98__
Max Throughput-50%Read.........______91__.........._____662__.........___20.69__
Random-8k-70%Read..............______91__.........._____598__.........____4.67__
EXCEPTIONS: CPU Util. 27%;
SERVER TYPE: VM @ VMware Server 1.0.1, MS Windows 2003,
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP ML350G5, 8GB RAM; 2x Xeon 5130, 2 GHz, DC,
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP Smart Array E200+BBWC, 6x250GB SATA, RAID5 (local storage)
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........____0.49__..........___10713__.........___334.8__
RealLife-60%Rand-65%Read......._____170__.........._____342__.........____2.67__
Max Throughput-50%Read.........____0.63__..........___10856__.........___339.2__
Random-8k-70%Read.............._____182__.........._____313__.........____2.44__
EXCEPTIONS: CPU utilization on both max throughput tests were close to 100% so I'm not
sure that those results are accurate. Cpu util on the other tests were ~18%.
Thanks for that.
I would say the cpu utilization is ok - your system reached imposing ios numbers by seq. r/w - the vm must perform them.
When you see my first test with vm you will see a high cpu utilization too, especially by seq. tests where the ios were high.
Till now the ibm DS8300 is the fasted SAN or not?
Yes indeed it seems to be the fastest for now (and everlasting ?).
\##################################################################################
TEST NAME--
\##################################################################################
IBM DS8300:
RealLife-60%Rand-65%Read......___1.42__..........__2603__.........__31.30___
EMC DMX3000:
RealLife-60%Rand-65%Read......___1.8_____..........___2170___.........___17_____
The question here could be which system could serve more guests with such performance .
They both are playing of course in other league - can profit from the large cache (32 GB).
I notice the MB/s number of 31.10 - that seems to be a bug (by block size 8kB).
I would say the cpu utilization is ok - your system
reached imposing ios numbers by seq. r/w - the vm
must perform them.
Keep in mind that the clock inside the VM becomes very unreliable under high cpu load. That's why I don't think these IOPS are too reliable. The VM surely performed these IOs, but the guest OS doesn't really have an accurate answer to how long time this took. If the cpu load is high the timing inside the guests is typically more wrong than during light loads. Looking at the load inside the guests from the hosts perspective normally gives a better picture. Looking at these numbers as provided by the guests will give us an idea about the performance, but it's not the 100% answer.
The document Timekeeping in VMware Virtual Machines[/url] gives some further details on this issue.
Lars
The following test is run from the host os where the previous VMware Server VM was tested. The file system tested is the one storing that virtual machine.
SERVER TYPE: Host with VMware Server 1.0.1, CentOS x64, 2 running VMs
CPU TYPE / NUMBER: pCPU / 4
HOST TYPE: HP ML350G5, 8GB RAM; 2x Xeon 5130, 2 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP Smart Array E200+BBWC, 6x250GB SATA, RAID5 (local storage)
\#################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........____18.3__..........____3148__.........____98.4__
RealLife-60%Rand-65%Read......._____333__.........._____178__........._____1.4__
Max Throughput-50%Read.........______24__..........____2374__.........____74.2__
Random-8k-70%Read.............._____417__.........._____142__........._____1.1__
EXCEPTIONS:
Lars
Yes, you are right - in this test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS VM ON ESX / VM SNAPSHOT ON
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I saw it too (util. vcpu ~100 %; cpu in VC ~55%).
What I always do is correcting the ios/sek in VC statistics or directly in storage (you can see there only MB/sek but when one knows the block size it is possible to calculate the ios/sek).
Has anyone created a virtual appliance for this yet?
Ok I have tweaked my set up from 1 6 spindle raid 10 to a 10 spindle raid 10 set. The differences are quite big!. Hopefully I should be able to run a few vms on this set up. The only thing I am not sure of those is what the percentage means on these stats (i.e. 60 percent read). How does this percentage relate to number of requests and bandwidth etc....
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM (ms 2000 server) 2gb vram.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: hp dl385, 6GB RAM for host esx server; 2x amd opteron (dual core), 2.4 GHz
STORAGE TYPE / DISK NUMBER / RAID LEVEL: sanmelody server on win2003 r2 raid 10 (1tb lun) 2gb ram (used as cache by sanmelody)
iscsi, sata disks 10 spindles
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........___17.2_______..........___3357_______.........____105______
RealLife-60%Rand-65%Read......_____82.7_____.........._____696_____.........____5.43______
Max Throughput-50%Read.........._____18.46_____..........______3117____.........____97.4______
Random-8k-70%Read.................___77_______..........____762______.........___5.95_______
EXCEPTIONS: CPU Util.-XX%;
>RealLife-60%Rand-65%Read<
In this test are 60 % of ios random - 40 % sequential, 65 % of ios are reads the rest writes - all that is similar to conventional disk's access.
Hope that's clear.
I don't want to make it that much more complicated than it already is but I'd really suggest some additional notations to make the numbers more comparable (there are more but I think these tend to cover most bases).
Additionally the length of time & data size probably both need to be significantly larger, as >8gig of cache is in many storage arrays and is likely to have a significant portion of all transaction happen within the cache in the storage array if the test is only ran for 5 minutes.
1) Amount of memory used in the guest (I'd probably suggest using a fixed number like 256mb)
2) Amount of cache used in the storage array
3) Whether or not the storage array is shared or dedicated
4) If array shared report # hosts sharing the array, and average number of IOPs and or throughput to array prior to and after the test
5) Whether or not the physical spindles are shared or dedicated
6) If spindles are shared report # hosts sharing the spindles average number of IOPs and or throughput to spindles prior, during and after
7) Which part of the disk platter the storage is on as outter to inner speeds can be upto 2x difference (outter 25%, outter middle 25%, inner middle 25%, inner 25%)
😎 Any known tweeks (dedicated cache toward specific spindles, short-stroking drives, QOS settings for luns in array, etc)
9) Guest & Host OS version + patch number
10) Drive make & model
Don't want to sound like I'm thinking this isn't a good idea (because I do), just want to add a little bit more detail and make the numbers more appropriate.