VMware Cloud Community
christianZ
Champion
Champion

Open unofficial storage performance thread

Attention!

Since this thread is getting longer and longer, not to mention the load times, Christian and I decided to close this thread and start a new one.

The new thread is available here:

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any

warranty.

If anybody shouldn't be agreed with some results then he can make own tests and presents

his/her results too.

I hope this way to classify the different systems and give a "neutral" performance comparison.

Additionally I will mention that the performance is one of many aspects to choose the right system.

The others could be e.g.

\- support quality

\- system management integration

\- distribution

\- self made experiences

\- additional features

\- costs for storage system and infrastructure, etc.

There are examples of IOMETER Tests:

=====================================

\######## TEST NAME: Max Throughput-100%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

\######## TEST NAME: RealLife-60%Rand-65%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

\######## TEST NAME: Max Throughput-50%Read

size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

\######## TEST NAME: Random-8k-70%Read

size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

The global options are:

=====================================

Worker

Worker 1

Worker type

DISK

Default target settings for worker

Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

Disk maximum size,starting sector

8000000,0

Run time = 5 min

For testing the disk C is configured and the test file (8000000 sectors) will be created by

first running - you need free space on the disk.

The cache size has direct influence on results. By systems with cache over 2GB the test

file should be increased.

LINK TO IOMETER:

Significant results are: Av. Response time, Av. IOS/sek, Av. MB/s

To mention are: what server (vm or physical), Processor number/type; What storage system, How many disks

Here the config file *.icf

\####################################### BEGIN of *.icf

Version 2004.07.30

'TEST SETUP ====================================================================

'Test Description

IO-Test

'Run Time

' hours minutes seconds

0 5 0

'Ramp Up Time (s)

0

'Default Disk Workers to Spawn

NUMBER_OF_CPUS

'Default Network Workers to Spawn

0

'Record Results

ALL

'Worker Cycling

' start step step type

1 5 LINEAR

'Disk Cycling

' start step step type

1 1 LINEAR

'Queue Depth Cycling

' start end step step type

8 128 2 EXPONENTIAL

'Test Type

NORMAL

'END test setup

'RESULTS DISPLAY ===============================================================

'Update Frequency,Update Type

4,WHOLE_TEST

'Bar chart 1 statistic

Total I/Os per Second

'Bar chart 2 statistic

Total MBs per Second

'Bar chart 3 statistic

Average I/O Response Time (ms)

'Bar chart 4 statistic

Maximum I/O Response Time (ms)

'Bar chart 5 statistic

% CPU Utilization (total)

'Bar chart 6 statistic

Total Error Count

'END results display

'ACCESS SPECIFICATIONS =========================================================

'Access specification name,default assignment

Max Throughput-100%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,100,0,0,1,0,0

'Access specification name,default assignment

RealLife-60%Rand-65%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,65,60,0,1,0,0

'Access specification name,default assignment

Max Throughput-50%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

32768,100,50,0,0,1,0,0

'Access specification name,default assignment

Random-8k-70%Read,ALL

'size,% of size,% reads,% random,delay,burst,align,reply

8192,100,70,100,0,1,0,0

'END access specifications

'MANAGER LIST ==================================================================

'Manager ID, manager name

1,PB-W2K3-04

'Manager network address

193.27.20.145

'Worker

Worker 1

'Worker type

DISK

'Default target settings for worker

'Number of outstanding IOs,test connection rate,transactions per connection

64,ENABLED,500

'Disk maximum size,starting sector

8000000,0

'End default target settings for worker

'Assigned access specs

'End assigned access specs

'Target assignments

'Target

C:

'Target type

DISK

'End target

'End target assignments

'End worker

'End manager

'END manager list

Version 2004.07.30

\####################################### ENDE of *.icf

TABLE SAMPLE

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM or PHYS.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE6850, 16GB RAM; 4x XEON 51xx, 2,66 GHz, DC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS3600 x 1 / 14+2 Disks / R50

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__________..........__________.........__________

RealLife-60%Rand-65%Read......__________..........__________.........__________

Max Throughput-50%Read..........__________..........__________.........__________

Random-8k-70%Read.................__________..........__________.........__________

EXCEPTIONS: CPU Util.-XX%;

##################################################################################

I hope YOU JOIN IN !

Regards

Christian

A Google Spreadsheet version is here:

Message was edited by:

ken.cline@hp.com to remove ALL CAPS from thread title

Message was edited by:

RDPetruska

Added link to Atamido's Google Spreadsheet

Tags (1)
Reply
0 Kudos
457 Replies
larstr
Champion
Champion

Here are some more results. This time I've tested ESX. I guess using the descheduled time service will give more accurate results due to the timing issues. Still, it doesn't make 100% sense that we get better performance on RealLife in a vm on ESX than on a physical install.

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service disabled

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

5.3

9711

303

RealLife-60%Rand-65%Read

43

786

6.1

Max Throughput-50%Read

6.4

8796

274

Random-8k-70%Read.

55

778

6

EXCEPTIONS: CPU Util. 73% 55% 56% 41%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

5.4

9887

308

RealLife-60%Rand-65%Read

42

777

6.0

Max Throughput-50%Read

6

8987

280

Random-8k-70%Read.

53

539

6

EXCEPTIONS: CPU Util. 67% 56% 67% 43%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled, arrayaccelerator=disable

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB arrayaccelerator=disable / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

24

2384

74.5

RealLife-60%Rand-65%Read

96

607

4.7

Max Throughput-50%Read

76

758

23.7

Random-8k-70%Read.

87

671

5.2

EXCEPTIONS: CPU Util. 27% 17% 20% 17%

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast

Sorry for the format change... :smileydevil:

Summary: it appears that most of the numbers are about the same across Netapp Heads.

*

OS

*

*

CPUxMemxGhz

*

*

Storage Vendor

*

*

Model

*

*

Protocol

*

*

  1. Disks

*

*

RAID

*

*

Results

*

*

Max Throughput

100%Read.

*

*

RealLife

60%Rand-65%Read

*

*

Max Throughput

50%Read

*

*

Random

8k-70%Read.

*

Win 2003R2sp2

8x16x2.6

Netapp

FAS3070

NFS/1G

40

RAID-DP













Av. RTime ms

16.8

12.9

4.7

12.8








Av. IOs/sek

3465

500

1135

506








Av. MB/sek

108

2.5

17.8

2.7








CPU

50

34

37

29

Win 2003r2sp2

8x16x2.6

Netapp

FAS6070

NFS/1G

28

RAID-DP













Av. Rtime ms

17.86

8.9

6.13

8.9








Av. IOs/sek

3304

506

1056

501








Av. MB/sek

103

3.9

33

3.9








CPU

66

31

35

27

Win 2003r2sp2

8x16x2.6

Netapp

FAS6070

ISCSI/1G

28

RAID-DP













Av. Rtime ms

17.86

18.39

5.6

20.7








Av. IOs/sek

3310

502

974

501








Av. MB/sek

103

3.9

30.4

3.9








CPU

58

31

33

35

Win 2003r2sp2

8x16x2.6

Netapp

FAS3050

NFS/1G

32

RAID-DP













Av. Rtime ms

17.80

18.6

6.2

20.8








Av. IOs/sek

3309

502

1189

501








Av. MB/sek

103

3.9

37.1

3.9








CPU

60

30

36

21

Win 2003r2sp2

8x16x2.6

Netapp

R200

NFS/1G

27

RAID DP













Av. Rtime ms

17.23

41

5.9

43.2








Av. IOs/sek

3412

502

1237

501








Av. MB/sek

106

3.9

38.6

3.9








CPU

54

36

38

40

Reply
0 Kudos
lasswellt
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS - VM on 1MB Block Size VMFS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM on VMware ESX 3.0.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 32GB RAM; 2x XEON, 2.0 GHz, Quad-Core

STORAGE TYPE / DISK NUMBER / RAID LEVEL: NetApp FAS3070 / 14+2 Disks / 144GB 15k

SAN TYPE / HBAs : FC, QLA2432

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........_____4.9__.........___11206___.........____350____

RealLife-60%Rand-65%Read......_____4_____..........___642___........._____5_____

Max Throughput-50%Read..........____1______..........__2114___.........___66____

Random-8k-70%Read.................____2.7____..........____922___.........____7______

EXCEPTIONS: CPU / 60%, 6%, 15%, 8%

##################################################################################

Message was edited by: lasswellt

Added CPU Util.

http://b2.techcolumnist.com
Reply
0 Kudos
ericdaly
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - VM on 1MB Block Size VMFS - IBM DS4800 (RAID 1)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM x3650, 36GB RAM; 2x XEON 5355 (Quadcore), 2,66 GHz,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4800 / 30 x 146GB 15k FC HDD on RAID1

VMFS: 500GB LUN, 1MB Block Size

SAN TYPE / HBAs : 4GB FC, QLogic QLA2432 Dual HBAs, Dual Cisco MDS 9216i Switches

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read.......___4.93______.......___11339.62___....___354.36____

RealLife-60%Rand-65%Read......___6.41_____......._____7859.46___....____61.40____

Max Throughput-50%Read........___2.49______.......___17374.72___....___542.96____

Random-8k-70%Read.............___6.43_____.......____ 7783.61___....____60.81____

Reply
0 Kudos
ericdaly
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS - VM on 1MB Block Size VMFS (RAID 5) IBM DS4800

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.0.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM x3650, 36GB RAM; 2x XEON 5355 (Quadcore), 2,66 GHz,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4800 / 30 x 146GB 15k FC HDD on RAID5

VMFS: 500GB LUN, 1MB Block Size

SAN TYPE / HBAs : 4GB FC, QLogic QLA2432 Dual HBAs, Dual Cisco MDS 9216i Switches

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read.......___4.97______.......___11343.74___....___354.49____

RealLife-60%Rand-65%Read......___7.15_____......._____6450.95___....____50.40____

Max Throughput-50%Read........___2.83______.......___17314.57___....___541.08____

Random-8k-70%Read.............___6.93_____.......____ 6454.94___....____50.43____

Reply
0 Kudos
christianZ
Champion
Champion

@dalepa

@lasswellt

Thanks for joining in.

Well I wonder why you (both) can only reach 500-600 ios/sek by Reallife-Test. I would expect much more???

@ericdaly

Thanks for that. I begin slowly to regret not to get the DS4800 (it was an alternative for us too) - really brute power in it!!

Reply
0 Kudos
ericdaly
Contributor
Contributor

Here are some more test done on new HP EVA 6000 today. These were the exact same tests I ran earlier in the week on a brand new IBM DS4800 (see previous posts). At the time of running tests on both SAN's there was no other I/O accessing the SAN, just 2 x ESX hosts and 1 active VM running I/O meter. The IBM comes in on top. The only major differnences in tests were the disks used.

HP EVA 6000 disk group made up of 30 x 300GB 10k FC (500GB VMFS LUN created on this)

IBM DS4800 disk group made up of 30 x 146GB 15k FC (500GB VMFS LUN created on this)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++* **TABLE OF RESULTS - VM on 1MB Block Size VMFS (vRAID 1) HP EVA 6000 **+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ *SERVER TYPE: Windows 2003 STD VM ON ESX 3.0.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL380 G5, 32GB RAM; Dual Intel Quad Core 2GHz E5335,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP EVA6000 / 30 x 300gb 10k FC HDD on vRAID1

VMFS: 500GB LUN, 1MB Block Size

SAN TYPE / HBAs : 4GB FC, HP StorageWorks FC1142SR 4Gb HBA's

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read.......___5.84______.......___9684.25___....___302.63____

RealLife-60%Rand-65%Read......___10.77_____.......____4488.41___....____35.07____

Max Throughput-50%Read........___8.08______.......___5395.06___....___168.60____

Random-8k-70%Read.............___10.64_____.......____4587.93___....___35.84____

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++* **TABLE OF RESULTS - VM on 1MB Block Size VMFS (vRAID 5) HP EVA 600 *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Windows 2003 STD VM ON ESX 3.0.2

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL380 G5, 32GB RAM; Dual Intel Quad Core 2GHz E5335,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP EVA6000 / 30 x 300gb 10k FC HDD on vRAID5

VMFS: 500GB LUN, 1MB Block Size

SAN TYPE / HBAs : 4GB FC, HP StorageWorks FC1142SR 4Gb HBA's

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read.......___5.12______.......___10790.57___....___337.21____

RealLife-60%Rand-65%Read......___11.81_____.......____3870.55___....____30.24____

Max Throughput-50%Read........___25.30______.......___1990.08___....___62.19____

Random-8k-70%Read.............___11.59_____.......____3940.41___....___30.78____

I was suprised to see the differnence on the Max Throughput-50%Read tests that were run on RAID1 disks:

HP EVA 6000 = Max Throughput-50%Read........___8.08______.......___5395.06___....___168.60____

IBM DS4800 = Max Throughput-50%Read........___2.49______.......__17374.72___....___542.96____

I was more suprised to see the differnence on the Max Throughput-50%Read tests that were run on RAID5 disks:

HP EVA 6000 = Max Throughput-50%Read........___25.30______.......___1990.08___....___62.19____

IBM DS4800 = Max Throughput-50%Read........___2.83______.......___17314.57___....___541.08____ (WOW!)

Reply
0 Kudos
larstr
Champion
Champion

I have now also done some more testing. One interesting thing I found now was that enabling cache on the disks will not give much performance difference regarding IOs, but it seems to give you less cpu load in the VM.

This time I'm testing RAID 1+0. My two previous tests were using only RAID 1.

SERVER TYPE: Physical Windows 2003R2sp2

CPU TYPE / NUMBER: 2x quad core

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

2.95

19932

622

RealLife-60%Rand-65%Read

46

1209

9.4

Max Throughput-50%Read

5

11272

352

Random-8k-70%Read.

39

1391

10.8

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

4.3

9976

311

RealLife-60%Rand-65%Read

30

1439

11

Max Throughput-50%Read

5.5

8779

274

Random-8k-70%Read.

30

1431

11

EXCEPTIONS: CPU Util. 92% 46% 89% 45%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled. Cache on individual disks enabled.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

5.4

9681

302

RealLife-60%Rand-65%Read

34

1353

10.5

Max Throughput-50%Read

6.1

8763

273

Random-8k-70%Read.

35

1412

11

EXCEPTIONS: CPU Util. 71% 40% 72% 33%

Reply
0 Kudos
christianZ
Champion
Champion

@ericdaly

@larstr

I like your deeper analyzes and comparisons.Thanks for that.

The DS4800 seems to be one of the best (performance) systems in midrange IMHO.

Reply
0 Kudos
cmanucy
Hot Shot
Hot Shot

I just wanted to whet everyone's appetite... I'm in the midst of some rather in-depth testing on some iSCSI solutions, and have been able to produce some very interesting data.

For example: differences between PCI-E & PCI-X cards, using dual-port vs. 2x single-port NICs, and the impacts these (and other) decisions make on CPU overhead. A little sprinkle of AMD-vs-Intel as well.

I hope to have some good stuff to post shortly. If anyone has any other ideas/requests, I'll see if I can cram it in while I still have the ability to test as well... it's not often you can take production systems and pull cables out just to see "what will happen" to the units...

---- Carter Manucy
Reply
0 Kudos
larstr
Champion
Champion

This time I've done a bit different test. This test has been done on a production system to a SAN that has a few servers connected (windows, pSeries, esx). The tested VM was the only active one in this LUN, but the array was shared with quite a few systems. The general load generated from other systems were however fairly low at this time of the night. I've also attached a few screen shots of the stats collected during the test.

The cpu stats I've reported has always been the load reported by iometer. Here is also attached a graph from the esx side (as collected by a VM that had some timing problems), and also storage stats taken from both the san and the esx server.

SERVER TYPE: Virtual Windows 2003sp2 Enterprise on ESX 3.0.2.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: IBM X3755, 24 GB RAM; 2x Opteron 8218, 2,6 GHz, DC - 22 running VMs

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS4500 / 2xEmulex LPe11000 4Gb FC adapters/ 9x146GB disks / raid 5/ vmfs 2MB block size

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

Max Throughput-100%Read.

10.6

5315

166

RealLife-60%Rand-65%Read

76

606

4.7

Max Throughput-50%Read

7.5

6710

209

Random-8k-70%Read.

53

796

6.2

EXCEPTIONS: CPU Util. 100% 51% 84% 47%

Message was edited by: larstr

Image upload as attachments to the message failed. Thumbnails too, so I had to include pictures directly into the message.Sorry about that.

Reply
0 Kudos
larstr
Champion
Champion

I must admit that for me this thread is one of the most valuable around. Thanks for starting it, Christian! Smiley Happy

During VMworld there was however also a very good session on the topic of storage performance. It showed a few very interesting results regarding queue depths and the impact of connecting different numbers of esx servers to the san controllers (with equal number of VMs).

The pdf is currently only available for VMworld attendies. Information about the session is as follows[/url]:

"

ID: IP42

Session: ESX Storage Performance - A Scalability Study

Presenter: VMware

In this talk, we will present actual performance results of the ESX Server storage subsystem with an emphasis on scalability in distributed environments. The scalability results collected on a 64-host blade cluster with shared storage will be covered for multiple use cases:

Distributed storage performance for IO intensive workloads - Covers performance results of various IO block sizes and access patterns with an aim to understand the steady-state scalability, responsiveness and fairness of ESX Server storage.

Distributed boot-up performance - Covers boot-up performance of a large number of VMs, helping us understand scalability of massively simultaneous file operations and small IO.

Scalable consistency management - Covers performance results of metadata management with advanced distributed locking.

Desktop workload performance - Covers performance of VMFS snapshots which enable Scalable Image management, also covering desktop scenarios.

"

Lars

Reply
0 Kudos
cmanucy
Hot Shot
Hot Shot

First (of a few) posts from my findings from Lefthand's iSCSI solution. This set of results is based upon a single VM running against a single volume on various hardware configurations.

Summary from these posts: HP DL385 seemed to push better numbers thanan IBM x3500. PCI-X cards did MUCH better than PCI-E, and Intel-based NICs beat the 3COM based NICs.

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 1-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

16.892014

3,320.53

103.76661

38.89

RealLife-60%Rand-65%Read

20.443687

2,732.92

21.350962

30.53

Max Throughput-50%Read

16.81863

3,314.06

103.56453

35.93

Random-8k-70%Read.

19.378037

2,844.44

22.22215

32.08

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 1-Way Volume, 1 on-board NIC, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

18.118919

3,139.80

98.118723

40.79

RealLife-60%Rand-65%Read

20.61234

2,684.31

20.971201

34.57

Max Throughput-50%Read

18.00176

3,073.97

96.061636

39.96

Random-8k-70%Read.

19.780298

2,774.99

21.67958

36.35

(Note: difference between this test and one above is using 1 vs 2 on-board NICs on the Lefthand units).

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 2-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

16.813312

3,366.40

105.19988

37.47

RealLife-60%Rand-65%Read

29.34655

1,881.29

14.697543

27.58

Max Throughput-50%Read

18.680861

3,069.82

95.931791

32.16

Random-8k-70%Read.

25.085427

2,176.13

17.001018

29.50

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 2-Way Volume, 1 on-board NIC, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

18.073136

3,156.20

98.631239

39.90

RealLife-60%Rand-65%Read

28.022059

1,980.81

15.475073

30.49

Max Throughput-50%Read

24.293573

2,395.75

74.867232

30.47

Random-8k-70%Read.

24.907659

2,200.19

17.188989

32.73

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 3-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

16.754199

3,344.42

104.51324

41.98

RealLife-60%Rand-65%Read

31.829165

1,766.70

13.802307

26.68

Max Throughput-50%Read

21.750049

2,662.46

83.201824

35.04

Random-8k-70%Read.

28.743989

1,915.24

14.962834

29.32

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 3-Way Volume, 1 on-board NIC, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2x3COM PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

17.250795

3,198.41

99.950318

42.42

RealLife-60%Rand-65%Read

32.346983

1,731.03

13.52365

26.51

Max Throughput-50%Read

30.307202

1,943.30

60.728241

26.01

Random-8k-70%Read.

28.757412

1,914.67

14.958361

28.68

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 2-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: HP DL385, 12GB RAM, 2xOpteron 2.6 DC, 2xIntel e1000 PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

17.12225

3,196.70

99.896827

34.79

RealLife-60%Rand-65%Read

27.626576

2,014.94

15.741728

26.41

Max Throughput-50%Read

20.620568

2,752.33

86.010156

29.10

Random-8k-70%Read.

24.564162

2,238.29

17.486643

28.32

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 2-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2xIntel PCI-E teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

19.8412

2,753.38

86.043215

47.88

RealLife-60%Rand-65%Read

32.484661

1,641.27

12.822457

45.13

Max Throughput-50%Read

26.160281

2,111.47

65.983492

47.71

Random-8k-70%Read.

28.381288

1,841.63

14.387739

46.50

Backend: Lefthand - 3xDL 320s, 36x15K 300GB SAS, 2-Way Volume, 2xbonded on-board NICs, 3COM 4500G

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2xIntel PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

16.790365

3,362.68

105.08382

38.62

RealLife-60%Rand-65%Read

31.189811

1,770.48

13.831869

28.35

Max Throughput-50%Read

19.868966

2,888.53

90.266672

30.79

Random-8k-70%Read.

27.76598

1,937.46

15.136426

30.57

---- Carter Manucy
Reply
0 Kudos
GandhiII
Contributor
Contributor

Many thanks for the test. As far as I can read out of the test, there is no better performance with using 2 NICs instead of one. This means that the network connection in iSCSI is not a bootleneck, if the system is by design okay, like with Lefthands clustered SAN architecture.

Reply
0 Kudos
cmanucy
Hot Shot
Hot Shot

Well, hold onto your thoughts until you see the rest of my data... this is just a single instance, so no, you're not going to fill 1Gbs with one single connection with most of the 'affordable' iSCSI solutions.

However, I've got more data from multi-session (running the same test at the same time against multiple volumes) and this definately pushes the performance numbers. As a matter of fact, that reminds me that I need to try out one of the tests with a downed link to see what happens to the numbers.

-Carter

---- Carter Manucy
Reply
0 Kudos
cmanucy
Hot Shot
Hot Shot

Part 2: Running 4 tests at one time on 2 ESX hosts (each with 2 VM's, each clones of each other). The back-end for this test is common amongst all units. I used 3 LeftHand DL320s, 36 x 15K 300GB SAS drives at RAID 50 (at the host level) and then a one-way (no replication) volume for EACH host. (Eg, each host has its own volume to run off of). All tests run with a 3COM 4500G as the switch.

Frontend: IBM x3500, 2 Xeon 5140's, 5GB RAM, 2xIntel PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator

This front-end hosted two sessions:

Session 1:

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

37.121856

1,561.79

48.805828

47.48

RealLife-60%Rand-65%Read

20.626144

1,820.24

14.220663

68.94

Max Throughput-50%Read

28.656554

2,049.38

64.043037

51.74

Random-8k-70%Read.

19.760432

1,909.62

14.918927

68.40

Session 2:

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

59.564748

988.60

30.893659

41.59

RealLife-60%Rand-65%Read

20.685352

1,833.00

14.320323

68.80

Max Throughput-50%Read

38.861875

1,541.86

48.183002

41.26

Random-8k-70%Read.

19.131433

1,968.08

15.375593

69.67

Frontend 2: Frontend: HP DL385, 12GB RAM, 2xOpteron 2.6 DC, 2xIntel e1000 PCI-X teamed NICs, VMWare ESX 3.0.2, Win2K3 SP2 VM, 1MB BS, ESX iSCSI Initiator.

This front-end also hosted 2 sessions (#'s 3 & 4):

Session 3:

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

38.208753

1,523.27

47.6022

43.32

RealLife-60%Rand-65%Read

20.99765

1,753.75

13.701134

68.56

Max Throughput-50%Read

26.893737

2,165.31

67.665887

54.53

Random-8k-70%Read.

19.724417

1,866.30

14.580491

68.60

Session 4:

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

34.92476

1,686.48

52.702519

42.04

RealLife-60%Rand-65%Read

20.639284

1,746.07

13.641162

69.27

Max Throughput-50%Read

32.786581

1,817.38

56.79308

42.12

Random-8k-70%Read.

20.050742

1,822.10

14.23512

68.91

Totals:

Session Totals - Response Time & CPU Use is AVEAGE, I/O's and MB/sec are SUMS

TEST NAME

Av. Resp. Time ms

Av. IOs/sek

Av. MB/sek

CPU Use

Max Throughput-100%Read.

42.455029

5,760.13

180.00421

43.61

RealLife-60%Rand-65%Read

20.737108

7,153.06

55.883282

68.89

Max Throughput-50%Read

31.799687

7,573.92

236.68501

47.41

Random-8k-70%Read.

19.666756

7,566.10

59.110131

68.89

Interesting data from these sessions - almost everything seemed to run pretty fairly on all the servers. No real show of any 'better' units between the AMD's and Intels here. Nice 55/56MB/sec pulled from all 4 sessions - nice 236MB/sec on the 50% read test. Might have been better with an iSCSI initiator.

---- Carter Manucy
Reply
0 Kudos
meistermn
Expert
Expert

Does the following make the test more comparibel?

1.) Partiton alignment of NTFS in the guest os

2.) Partiton alignment of vmfs

3.) virus scanner

4.) pagefile usage

5.) same os for all

or a linux vm with iometer

Reply
0 Kudos
kenrobertson
Contributor
Contributor

Anyone have any advice for optimizing iSCSI with SANmelody? I seem to be getting only about 1/2 the transfer rate that I figured I'd be getting.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VMware ESX Server 3.0.2 - Windows Server 2003 R2 Standard

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell 1950, 8GB RAM; 2x Intel 5335 (4 cores, 2.0ghz ea)

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell 2950, SANmelody, 6x 300G 15k SAS RAID 5,

500gb LUN, iSCSI 1x 1GB

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........_31.6212_.........._1879.11_........._58.72_

RealLife-60%Rand-65%Read......_50.6439_.........._1139.77_.........__8.90_

Max Throughput-50%Read.........._28.6123_.........._2075.36_........._64.85_

Random-8k-70%Read................._43.4495_.........._1293.32_........._10.10_

EXCEPTIONS:

##################################################################################

Reply
0 Kudos
christianZ
Champion
Champion

My thought is you didn't activate the raid controller cache for writing; but don't forget Sanmelody is using the large cache and you can get

very high random ios, when testing with only one vm (it isn't of course an realistic value in that case).

Reply
0 Kudos
christianZ
Champion
Champion

Thanks for that deep authentic testing - IMHO the ESX iscsi software initiator can be still optimized; maybe the 3.5 comes with better iscsi stack.

Reply
0 Kudos