Attention!
Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.
The new thread is available here:
[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]
My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any
warranty.
If anybody shouldn't be agreed with some results then he can make own tests and presents
his/her results too.
I hope this way to classify the different systems and give a "neutral" performance comparison.
Additionally I will mention that the performance is one of many aspects to choose the right system.
The others could be e.g.
\- support quality
\- system management integration
\- distribution
\- self made experiences
\- additional features
\- costs for storage system and infrastructure, etc.
There are examples of IOMETER Tests:
=====================================
\######## TEST NAME: Max Throughput-100%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
\######## TEST NAME: RealLife-60%Rand-65%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
\######## TEST NAME: Max Throughput-50%Read
size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
\######## TEST NAME: Random-8k-70%Read
size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
The global options are:
=====================================
Worker
Worker 1
Worker type
DISK
Default target settings for worker
Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
Disk maximum size,starting sector
8000000,0
Run time = 5 min
For testing the disk C is configured and the test file (8000000 sectors) will be created by
first running - you need free space on the disk.
The cache size has direct influence on results. By systems with cache over 2GB the test
file should be increased.
LINK TO IOMETER:
Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s
To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks
Here the config file *.icf
\####################################### BEGIN of *.icf
Version 2004.07.30
'TEST SETUP ====================================================================
'Test Description
IO-Test
'Run Time
' hours minutes seconds
0 5 0
'Ramp Up Time (s)
0
'Default Disk Workers to Spawn
NUMBER_OF_CPUS
'Default Network Workers to Spawn
0
'Record Results
ALL
'Worker Cycling
' start step step type
1 5 LINEAR
'Disk Cycling
' start step step type
1 1 LINEAR
'Queue Depth Cycling
' start end step step type
8 128 2 EXPONENTIAL
'Test Type
NORMAL
'END test setup
'RESULTS DISPLAY ===============================================================
'Update Frequency,Update Type
4,WHOLE_TEST
'Bar chart 1 statistic
Total I/Os per Second
'Bar chart 2 statistic
Total MBs per Second
'Bar chart 3 statistic
Average I/O Response Time (ms)
'Bar chart 4 statistic
Maximum I/O Response Time (ms)
'Bar chart 5 statistic
% CPU Utilization (total)
'Bar chart 6 statistic
Total Error Count
'END results display
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
Max Throughput-100%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
'Access specification name,default assignment
RealLife-60%Rand-65%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
'Access specification name,default assignment
Max Throughput-50%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
'Access specification name,default assignment
Random-8k-70%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
'END access specifications
'MANAGER LIST ==================================================================
'Manager ID, manager name
1,PB-W2K3-04
'Manager network address
193.27.20.145
'Worker
Worker 1
'Worker type
DISK
'Default target settings for worker
'Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
'Disk maximum size,starting sector
8000000,0
'End default target settings for worker
'Assigned access specs
'End assigned access specs
'Target assignments
'Target
C:
'Target type
DISK
'End target
'End target assignments
'End worker
'End manager
'END manager list
Version 2004.07.30
\####################################### ENDE of *.icf
TABLE SAMPLE
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM or PHYS.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50
##################################################################################
TEST NAME--
##################################################################################
Max Throughput-100%Read........__________..........__________.........__________
RealLife-60%Rand-65%Read......__________..........__________.........__________
Max Throughput-50%Read..........__________..........__________.........__________
Random-8k-70%Read.................__________..........__________.........__________
EXCEPTIONS: CPU Util.-XX%;
##################################################################################
I hope YOU JOIN IN !
Regards
Christian
A Google Spreadsheet version is here:
Message was edited by:
ken.cline@hp.com to remove ALL CAPS from thread title
Message was edited by:
RDPetruska
Added link to Atamido's Google Spreadsheet
here are my results. it seems that they are somewhat inline with happyhammer, but i'm surprised that AnthonyM has better performance using nics and an iSCSI initiator.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: W2K3 SP1 VM (1gb RAM) on ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL385 g2, 8GB RAM; 2 x AMD Opteron 2,66 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS100E x 1 / 12+2 SATA Disks / R50
SAN TYPE / HBAs : iSCSI, QLA4052C HBA Jumbo's and Flow Control
VMFS: 400GB LUN, 2MB Block Size
NETWORK TYPE: HP Procurve 5406 Jumbo's and Flow Control
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........____23____..........___2611___.........___81___
RealLife-60%Rand-65%Read.......____40____..........___1185___.........___9____
Max Throughput-50%Read.........____14____..........___4132___.........___129__
Random-8k-70%Read..............____48____..........___1045___.........___8____
EXCEPTIONS: VCPU Util.-32%;
##################################################################################
spelling
Message was edited by:
doubleH
... and you have probably the same issue like happyhammer (jumbo and flow control simultaneously) - try the enable only flow control - you should see better results by sequential reads/writes.
Indeed with ms iscsi initiator you can get much better results - saw this too.
Thanks.
did you say this was a recommendation from EQL? if so could you provide a url to the document or if it was a support request could you please pass the sr# along to me so i can reference this with support.
thanks
Here is the original answer from EQL:
With the Procurve 28xxx series we have found that running Jumbo frames + flow control doesnt work well. The switch lacks sufficient buffers. Flow control is much more effective at providing the best performance. Also, you can increase the available buffers by enabling the qos-passthrough-mode option. This will change the number of QOS queues from 4 to 2. However, it will require a reboot of the switch to take effect.
Ive attached our network guidelines which shows our suggested method of configuring multiple switches. In the diagram there is a small line between the switches. This represents a set of trunked ports between them. At a minimum there should be three and we typically suggest four for best performance.
Ive also attached the current draft of our FAQ for VMware. Are you using Qlogic HBAs? If so there are some settings you should check as well. They are listed in the FAQ.
Lastly is the Window disk timeout settings document. In the guest you should extend the Windows disk timeout to 60 seconds.
Please let me know if this answers your question. If not, please reply-all to this e-mail.
Regards,
Don[/i]
The case nr. was: 9528
And indeed changing of it gets more throughput by seq. r/w.
thank you very much. i'll see if they recommend this on the 5406's as well.
i callled support and they confirmed what you have said (not that i doubted you). there is a 2nd reason in addition to the buffer reason you provided. it is that ESX doesn't support jumbo frames in 3.0.1.
i'll go ahead and turn off jumbo's on my storage vlan. the only problem is that i'll have to take down the esx servers to take jumbo's off of the HBA's in the bios. i'll run the test again and see what difference it makes.
here are the new stats after disabling jumbos on the switch ports and HBA's....
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: W2K3 SP1 VM (1gb RAM) on ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL385 g2, 8GB RAM; 2 x AMD Opteron 2,66 GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS100E x 1 / 12+2 SATA Disks / R50
SAN TYPE / HBAs : iSCSI, QLA4052C HBA
VMFS: 400GB LUN, 2MB Block Size
NETWORK TYPE: HP Procurve 5406 with Flow Control
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........____16____..........___3561___.........___111___
RealLife-60%Rand-65%Read.......____39____..........___1188___.........___9____
Max Throughput-50%Read.........____9____...........___6303___.........___197__
Random-8k-70%Read..............____47____..........___1050___.........___8____
EXCEPTIONS: VCPU Util.-32%
It was a PS3800. Sorry for the mistake.
So, am I to understand from redmini's results that running the 64 bit version of W2K3 R2 in a VM instead of the 32 bit version can give you a 1-30% performance hit on reads?
I was seeing a 20% hit on my 64bit vms.
Wow, that is ugly. So unless you need 64bit for some reason, should one avoid it like the plague in a VM?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: Win2k3 SP2 VM w/ 1GB RAM on ESX 3.0.1
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: DL585G2, 32GB RAM; 4x Opteron 2.8GHz, DC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS6800 x 1 / 7+1 Disks / R5
350GB VMFS LUN
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........_10.53_____..........___5586___.........____174___
RealLife-60%Rand-65%Read......___25.56___..........___1971___.........____15.4___
Max Throughput-50%Read..........__6.39____..........__8067____.........___252____
Random-8k-70%Read................._24.26____..........___1999___.........__15.62____
EXCEPTIONS: CPU Util.- 54, 41, 82, 43
##################################################################################
Will post results from DS4800 soon.
Message was edited by: mitchellm3
mitchellm3
That's great, those two models weren't there.
Thanks.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS - vRAID1
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz
STORAGE TYPE / INFRA: HP EVA 4000 / 4 Gbps
DISK NUMBER / RAID LEVEL: 14 x 146GB FC / vRAID 1
VMFS: 200GB LUN, 1 MB Block Size
#####################################################################################
TEST NAME--
\#####################################################################################
Max Throughput-100%Read.....___5.0____..........__11168___.........__349.0___
RealLife-60%Rand-65%Read....__11.8____..........___2808___.........___21.9___
Max Throughput-50%Read......__14.6____..........___2974___.........___92.9___
Random-8k-70%Read...........__11.9____..........___2408___.........___18.8___
EXCEPTIONS: CPU Util. 68-69-57-74%;
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS - vRAID5
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz
STORAGE TYPE / INFRA: HP EVA 4000 / 4 Gbps
DISK NUMBER / RAID LEVEL: 14 x 146GB FC / vRAID 5
VMFS: 300GB LUN, 1 MB Block Size
#####################################################################################
TEST NAME--
\#####################################################################################
Max Throughput-100%Read.....___4.7____..........__11158___.........__348.6___
RealLife-60%Rand-65%Read....__13.5____..........___2149___.........___16.8___
Max Throughput-50%Read......__41.6____..........___1203___.........___37.6___
Random-8k-70%Read...........__13.5____..........___1855___.........___14.5___
EXCEPTIONS: CPU Util. 82-75-35-79%;
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE OF RESULTS - DAS
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP ProLiant DL385G2, 14GB RAM; 2x Dual Core AMD Opteron 2220, 2,80 GHz
STORAGE TYPE: DAS (HP SmartArray P400 512MB)
DISK NUMBER / RAID LEVEL: 2 x 72GB 15K SAS / RAID 1+0
VMFS: 40GB, 1 MB Block Size
#####################################################################################
TEST NAME--
\#####################################################################################
Max Throughput-100%Read.....___5.7____..........__10067___.........__314.6___
RealLife-60%Rand-65%Read....__59.5____..........____818___.........____6.4___
Max Throughput-50%Read......___5.7____..........___9954___.........__311.0___
Random-8k-70%Read...........__61.1____..........____812___.........____6.3___
EXCEPTIONS: CPU Util. 50-38-50-31%;
Thanks for that.
What I observed was the difference by "Max Throughput-50%Read" on Vraid1 and Vraid5 - really large.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VMWARE ESX 3.01 - VM Windows Server 2003R2 Standard
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: 3 x Dell PE2900, 16GB RAM; 2x XEON 5345, 2,33 GHz, QuadCore
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Datacore SANMelody 2.02 / 7+1 Disk / R5
STORAGE HARDWARE: 2 x Dell PE2900 / internal SAS Harddisks 300GB/10K / FC attached to ESX (Redundant)
FC Switch: 2 x Brocade 200E / FC HBA: Qlogic 2462
VMFS: 300GB / 1GB Blocksize
\##################################################################################
TEST NAME--
\##################################################################################
Max Throughput-100%Read........_____4.50___.........._11974__........ ____374____
RealLife-60%Rand-65%Read......._____0.59___.........._18125__.........____142____
Max Throughput-50%Read........._____2.01___.........._15181__.........____474____
Random-8k-70%Read.............._____0.48___.........._18834__.........____147____
EXCEPTIONS: CPU Util.-98%;
How much memory on your storage server?
However, I suspect that your results are not as accurate as we would prefer (as we see others (including me) with high cpu load).
Is it possible to get any stats from the SanMelody side while doing these tests?
Lars
Agreeded
either cpu high utilisation causes incorrect time cycles or the whole test file sits in host cache - maybe possible to verify the results by san melody statistics (to check the disks activities, response time,...)