VMware Cloud Community
christianZ
Champion
Champion

New !! Open unofficial storage performance thread

Hello everybody,

the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.

Oliver will make a few links between the old and the new one and then he will close the old thread.

Thanks for joining in.

Reg

Christian

574 Replies
Dr_Virt
Hot Shot
Hot Shot

Long time FC shop decided to deploy iSCSI for remote office. iSCSI performing surprisingly well.

Physical assets: 2x HP P4300G2 (Lefthand) ver. 8.5, 2x HP 2910AL, 2x HP DL360G6

Network: All 1GbE, 2 Ports each

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM Windows 2008 R2 Standard (Version 7), 1x vCPU, 4 GB RAM, 2x 40GB VMDKs

HOST TYPE: HP Proliant DL360 G6, 36GB RAM, 2x X5650

VMWARE: ESXi 4.1 on USB Drive, Software iSCSI Intiator on 2x VMkernel

SAN Type: HP P4300G2 (LeftHand) / Disks: 450GB 10k SAS / RAID LEVEL: Raid5 / 8 Disks

##################################################################################

Test Name

Average Response Time (ms)

Average I/O per Sec.

Average MB per Sec.

Max Throughput - 100% Read

1.75

25,916.19

809.88

Real Life - 60% Random 65% Read

35.93

1,426.11

11.14

Max Throughput - 50% Read

17.48

3,333.33

104.17

Random - 8K 70% Read

15.59

1,627.12

18.15

Reply
0 Kudos
larstr
Champion
Champion

Christian,

For automatic interpretation of iometer results.csv I've setup a web page here:

http://vmktree.org/iometer/

I also put the .iso file from the older thread in there.

Lars

Reply
0 Kudos
pinkerton
Enthusiast
Enthusiast

Bis zum 13.12. bin ich nicht im Büro. Bitte wenden Sie sich in dieser Zeit an support@mdm.de

Best Regards

Michael Groß

Reply
0 Kudos
larstr
Champion
Champion

SERVER TYPE: Windows Server 2003R2x32 VM running on ESXi 4.1 CPU TYPE / NUMBER: AMD Opteron 6174 HOST TYPE: DL385G7 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Local disks, P410i, 8x15k rpm SAS, RAID5
Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read1.972852589171%
RealLife-60%Rand-65%Read24.3213941059%
Max Throughput-50%Read2.332452376655%
Random-8k-70%Read25.4713921056%
Reply
0 Kudos
JTravers
Contributor
Contributor

Here are my results.  I can't determine why our performance is degraded when using jumbo frames on our MD3000i.  We're using (2) PowerConnect 5424's for our dedicated iSCSI network and jumbo frames are enabled from end-to-end.

SERVER TYPE:  Windows Server 2003R2x32 VM running on ESXi 4.1

CPU TYPE / NUMBER: VCPU x 1, JUMBO FRAMES, MPIO RR

HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5620

STORAGE TYPE / DISK NUMBER / RAID LEVEL: MD3000i, 300GB, 6x15k RPM SAS, RAID 10
Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read19.6130369420%
RealLife-60%Rand-65%Read19.2425581937%
Max Throughput-50%Read19.6230619519%
Random-8k-70%Read18.3427282137%

SERVER TYPE:  Windows Server 2003R2x32 VM running on ESXi 4.1

CPU TYPE / NUMBER: VCPU x 1, JUMBO FRAMES, MPIO RR

HOST TYPE: PowerEdge R710, 24GB RAM, 2xE6520

STORAGE TYPE / DISK NUMBER / RAID LEVEL: MD3000i, 300GB, 7x15k RPM SAS, RAID 5
Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read19.1131229720%
RealLife-60%Rand-65%Read22.9420271540%
Max Throughput-50%Read19.3031059719%
Random-8k-70%Read21.2821031643%

SERVER TYPE:  Windows Server 2003R2x32 VM running on ESXi 4.1

CPU TYPE / NUMBER: VCPU x 1, NO JUMBO FRAMES, MPIO RR

HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5520

STORAGE TYPE / DISK NUMBER / RAID LEVEL: MD3000i, 300GB, 6x15k RPM SAS, RAID 10
Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read15.20394212318%
RealLife-60%Rand-65%Read19.0225141939%
Max Throughput-50%Read17.09358511215%
Random-8k-70%Read18.9425581938%

SERVER TYPE:  Windows Server 2003R2x32 VM running on ESXi 4.1

CPU TYPE / NUMBER: VCPU x 1, NO JUMBO FRAMES, MPIO RR

HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5520

STORAGE TYPE / DISK NUMBER / RAID LEVEL: MD3000i, 300GB, 7x15k RPM SAS, RAID 5
Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read15.13395712318%
RealLife-60%Rand-65%Read23.9520901632%
Max Throughput-50%Read16.60368211515%
Random-8k-70%Read22.8121071636%

===========================

Interface  Port Group/DVPort             IP Family       IP Address             Netmask               MAC Address           MTU     Type        VMotion
vmk0       Management Network            IPv4           192.168.10.1           255.255.255.0        xx:xx:xx:xx:xx:xx    1500    STATIC      Disabled
vmk1       iSCSI1                                  IPv4          192.168.100.10        255.255.255.0        xx:xx:xx:xx:xx:xx    9000    STATIC      Disabled
vmk2       iSCSI2                                  IPv4          192.168.101.10        255.255.255.0        xx:xx:xx:xx:xx:xx    9000    STATIC      Disabled

===========================

Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks

vSwitch0                 128                    4                    128               1500      vmnic4,vmnic0

   PortGroup Name                VLAN ID   Used Ports      Uplinks

   Management Network            0                 1               vmnic0

Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks

vSwitch1                  128                   6                  128                 1500    vmnic9,vmnic8,vmnic3,vmnic2

   PortGroup Name                VLAN ID   Used Ports      Uplinks

   VM Network                          0                  1               vmnic9,vmnic8,vmnic3,vmnic2

Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks

vSwitch2                  128                    5                  128                9000    vmnic5,vmnic1

   PortGroup Name                VLAN ID   Used Ports      Uplinks

   iSCSI2                                    0                     1               vmnic5

   iSCSI1                                    0                     1               vmnic1

============================

Reply
0 Kudos
s1xth
VMware Employee
VMware Employee

SERVER TYPE: W2K8 32bit on ESXi 4.1 Build 320137 1vCPU 2GB RAM
CPU TYPE / NUMBER: Intel X5670 @ 2.93Ghz
HOST TYPE: Dell PE R610 w/ Broadcom 5709 Dual Port w/ EQL MPIO PSP Enabled
NETWORK: Dell PC 6248 Stack w/ Jumbo Frames 9216
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS4000X 16 Disk Raid 50
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read8.12741023129%
RealLife-60%Rand-65%Read10.6533472659%
Max Throughput-50%Read7.19786124534%
Random-8k-70%Read11.3733872655%
http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
fbonez
Expert
Expert

Sono fuori ufficio.

Rientrerò il 10/01/2011.

Per urgenze contattare l'assistenza tecnica allo 045 8738738.

Francesco Bonetti

RTC SpA

-- If you find this information useful, please award points for "correct" or "helpful". | @fbonez | www.thevirtualway.it
Reply
0 Kudos
pinkerton
Enthusiast
Enthusiast

Bis zum 03.01. bin ich nicht im Büro. Bitte wenden Sie sich in dieser Zeit an support@mdm.de.

I'm out of office until January 3rd. Please contact support@mdm.de instead.

Mit freundlichem Gruß / Best Regards

Michael Groß

MDM IT Abteilung / MDM IT department

Reply
0 Kudos
JonWeatherhead
Contributor
Contributor

My results have been crammed into a spreadsheet that I sent around to some of the guys I work with, I'm not going to take the time to format them the same way everyone else has been doing... sorry for that.

Background:

SERVER TYPE: Windows 2003 Standard R2 (aligned to 1024K)
CPU TYPE / NUMBER / Memory: VCPU / 2 / 1024M
HOST TYPE: Dell PowerEdge R510, 32GB RAM, 2 x Intel L5520, 2.266GHz, QuadCore, Running ESXi 4.1
STORAGE TYPE / DISK NUMBER / RAID LEVEL: DAS PERC H700 / (6) 300GB SAS 15k / RAID 10

STRIPE UNIT SIZE: Default 64k, 256k, 512k, 1024k

VMFS ALIGNMENT: Default 64k, 256k, 512k, 1024k

VMFS BLOCK SIZE: 8M for all except the first one that was run looking at all the default settings and a misaligned guest

I've attached my results as a spreadsheet.

I was interested in seeing that the stride used in my RAID array had a way bigger effect on performance than what guest alignment did.  Although I suspect that alignment of the guests would be a bigger deal as I start adding more VMs.  Also, back in ESX3 they had a best practice of setting your Windows guests up to use a FAU (File Allocation Unit) of 32k.  This just doesn't seem to have a big effect at all on disk I/O.

So for our setup where we are going to use DAS instead of a SAN, I believe we will:

1. Run our RAID strides to 1024k

2. Align any new VMs but disregard alignment on existing ones (maybe... jury is still out)

3. Not bother with setting regular (non DB) VMs up to use an FAU of 32k

4. Not bother with re-aligning the VMFS volumes to match our strides (it helps but not that much)

5. Continue to use 8M blocks on datastores since that is what we have already been doing for a while.

Has anyone else noticed any real benefit to setting the guest OS to use FAU of 32k?

Reply
0 Kudos
cbseydler
Contributor
Contributor

Hi everybody,

here are my test results. We are using a NetApp FAS 3140 and the datastores sit in a volume that's made available via NFS. The ESXi 4 host has two 1Gbps connections to the filer. The maximum throughput is somewhere near the limit of a 1 Gbps connection - as one would expect. The random throughput is near 20 MB/s.

These numbers look normal, or what do you think?

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


SERVER TYPE: ESXi 4, VM Windows 2003 Standard

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP ProLiant BL460c G6, 48GB RAM; 2x Intel Xeon E5540, 2.53 GHz, QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: NetApp FAS3140 / 13 Disks (2 Parity) / RAID DP

SAN TYPE: Storage Link 2xEthernet 1Gb / NFS / NO JUMBOFRAMES / NO FLOWCONTROL


##################################################################################

TEST NAME--


--------------------------------------------------------------------------------
Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################


Max Throughput-100%Read........__17.5395__.........._3431.58_........._107.24__


RealLife-60%Rand-65%Read......__19.9659__.........._2588.61_........._20.22__


Max Throughput-50%Read..........__12.2198__.........._4921.79_........._153.81__


Random-8k-70%Read.................__19.6347__.........._2489.96_........._19.45__


EXCEPTIONS: CPU Util.-%;


##################################################################################

Thanks in advance for your hints.

Alex

Reply
0 Kudos
ryanmaclean
Contributor
Contributor

A quick test of a recent Nexenta build.

Dedupe and compression both turned off.

Using direct-connect iSCSI with software initiator on Win7 64bit client prior to testing same box with vSphere.

SERVER TYPE: Desktop Test
CPU TYPE / NUMBER: Core i7 920, 4 cores
HOST TYPE: Physical whitebox, 12GB RAM, 2x gbit NICs
NETWORK: Direct-attached iSCSI with 1500 MTU, 32k block size on storage pool and NTFS formatted LUN.
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Nexenta Enterprise, 3x 1.5TB Seagate 7200.11 SATA in ZPool, 1x 55GB OCZ Revodrive ZIL, 1x 55GB Revodrive Cache
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read16.7435711110%
RealLife-60%Rand-65%Read7.136187480%
Max Throughput-50%Read12.3444821400%
Random-8k-70%Read6.506198480%
Reply
0 Kudos
Gabriel_Chapman
Enthusiast
Enthusiast

SERVER  TYPE:  W2K3  R2 SP2 x32 1vCPU 2GB RAM

CPU TYPE  2 X Nehalem 7560 8Core  @2.27GHz

HOST TYPE:   IBM X3960 X5 128GB RAM (2)8GB FC - (2)10GB  Eth

STORAGE TYPE / DISK NUMBER /  RAID LEVEL:   XIV/180/NORAID

Test TypeLatencyAvg IOPSAvg MBpsCPU Load
Max Throughput: 100% Read12.03532016616%
Real Life 60% Rand-Read3.961617712643%
Max Throughput 50% Read3.651754513747%
Random 8k-70% Read11.05579318119%
Ex Gladio Equitas
Reply
0 Kudos
Adrian_Buchmann
Contributor
Contributor

SERVER TYPE: ESXi 4.10 / Windows Server 2008 R2 x64, 2 vCPU, 4GB RAM
CPU TYPE / NUMBER: Intel Xeon X5670 @ 2.93GHz
HOST TYPE: HP ProLiant BL460c G7
STORAGE TYPE / DISK NUMBER / RAID LEVEL: NetApp FAS6280 Metrocluster, FlashCache / 80 Disks / RAID DP
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read4.071156236163%
RealLife-60%Rand-65%Read1.67229011781%
Max Throughput-50%Read3.931168436561%
Random-8k-70%Read1.45255091991%
Reply
0 Kudos
meistermn
Expert
Expert

Maybe following is interessting for you

http://thebackroomtech.com/2008/09/10/perfomance-test-windows-server-2003-r2-partition-alignment-on-...

or the different block sizes for different ms applications (sql ...)

http://communities.vmware.com/docs/DOC-3961

Reply
0 Kudos
pinkerton
Enthusiast
Enthusiast

After having some trouble with storage performance as described in http://communities.vmware.com/message/1680855#1680855 and moving the Heavy Hitter to a different set of disks, we additionally decided to buy four more drives and move the VMFS LUNs on our HP EVA 4400 from RAID5 to RAID1.

Here are my results:

SERVER TYPE: VM Windows 2008 R2, 4 GB RAM, LSI Logic SAS

CPU TYPE / NUMBER: 2 VCPU

HOST TYPE: ESXi 4.1 320137, HP DL380 G6, 64GB RAM, 2x E5520, 2,27 GHz QC

STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP EVA 4400 / 32/36x FC 10k 300GB / RAID5 and RAID10

SAN TYPE / HBAs : FC, HP FC1142SR QLogic HBA, HP StorageWorks 8/8 SAN Switches

#################################################################################################

Vraid5 / 32 HDDs

Test NameResponse TimeIOPSMB/s

Max Throughput-100%Read

2,6710.809337,78

RealLife-60%Rand-65%Read

8,293.05523,87

Max Throughput-50%Read

34,031.53047,81

Random-8k-70%Read

8,642.86622,39

Vraid1 / 36 HDDs

Test NameResponse TimeIOPSMB/s

Max Throughput-100%Read

5,0911.686365,20

RealLife-60%Rand-65%Read

10,844.28433,47

Max Throughput-50%Read

6,635.113159,78

Random-8k-70%Read

10,414.78337,37

As you can see, especially test 3 and 4 run MUCH faster with Vraid1 instead of Vraid5! It actually seems that the RAID Level is more important for performance than the amount of disks!

Reply
0 Kudos
1538moss
Contributor
Contributor

SERVER TYPE: VM Windows 2008 R2, 2 GB RAM, LSI Logic SAS
CPU TYPE / NUMBER: 2 VCPU
HOST TYPE: ESXi 4.0 U1, HP DL380 G7, 96GB RAM
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IO Accelerator 160GB ("SSD DISK")

Test NameResponse TimeIOPSMB/s

Max Throughput-100%Read

2,67
22.743
710

RealLife-60%Rand-65%Read

1.41
39.721
310

Max Throughput-50%Read

3.64
16.031
500

Random-8k-70%Read

1.55
36.861
287
Reply
0 Kudos
JonT
Enthusiast
Enthusiast

1538moss, these numbers are impressive but I wonder why the IOPs were so low for the IO Accelerator device. Your throughput numbers indicate great bandwidth to the disk but the overal IOs were much lower that I had experienced with a 360GB Duo version of the IO Accelerator. Here is what I observed when using this SLC memory type of disk. Keep in mind that this is an unrealistic storage type for our mainstream production VM's, but it does present a great solution to the most disk intensive applications such as Active Directory or SQL.

I only ran the "RealLife-60%Rand-65%Read" and the 100% Read test, but the numbers were pretty nice:

SERVER TYPE: VM Windows 2008 SP2 x64, 4 GB RAM, LSI Logic SAS

CPU TYPE / NUMBER: 2 VCPU

HOST TYPE: ESXi 4.1, HP DL380 G7, 32GB RAM

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IO Accelerator 360GB Duo SLC

Test Name

Response Time

IOPS

MB/s

Max Throughput-100%Read

0.500

319,321.40

155.9

RealLife-60%Rand-65%Read

6.11

100,218.33

48.9

Max Throughput-50%Read

N/A

N/A

N/A

Random-8k-70%Read

N/A

N/A

N/A

Reply
0 Kudos
AGrueninger
Contributor
Contributor

@JonT

You have very high IOPS and a low values for MB/s. Your configuration seems to use a blocksize of 512 bytes and not 32k for "Max Throughput..".

The calculation is: 155.9*1024*1024/319,321.40 is about 512 and it should be 32k.

The same ist true for the "real life" where the result shoud be 8k and we got again 511 with your results.

You may check with one of my results (see below).

Andreas

SERVER TYPE: VM Windows 2003 SP2 x86, 2 vCPU, 1 GB RAM, paravirtual

HOST TYPE: ESXi 4.0 (332073), Fuijtsu RX300S6, 48 GB RAM

STORAGE TYPE / DISK NUMBER / RAID LEVEL: OpenSolaris B134, 64 GB RAM / 12 SAS 146 GB / 150 GB, 8k recordsize, RAIDZ1 with 4 vDevs, 32 GB SLC SSD as writezilla, 160 MLC SSD as readzilla

STORAGE NETWORK / PROTOCOL: Infiniband DDR, 20 Gb/s / SRP

Test Name

Response Time

IOPS

MB/s

CPU%

Max Throughput-100%Read

3.5716,856526.7

26

RealLife-60%Rand-65%Read

2.127,800210.8

34

Max Throughput-50%Read

2.98

19,900624.7

30

Random-8k-70%Read

2.1024,582192.0

39

Reply
0 Kudos
larstr
Champion
Champion

SERVER TYPE: Windows 2003 VM ON ESXi 4.1

HOST TYPE: HP DL380G6

STORAGE  TYPE / DISK NUMBER / RAID LEVEL: Hp LeftHand 4 Nodes across 2 locations. Local RAID5, iSCSI, Network RAID 1+0. 12 SAS disks in each node.

Test nameLatencyAvg iopsAvg MBpscpu load
Max Throughput-100%Read17.29342910730%
RealLife-60%Rand-65%Read13.5431492461%
Max Throughput-50%Read13.08381611952%
Random-8k-70%Read8.6742873370%
Reply
0 Kudos
ngerasim
Contributor
Contributor

SERVER TYPE: Windows Server 2008 R2 CPU TYPE / NUMBER: 4 vCPU (16xE7440 CPUs) HOST TYPE: BL680c G5 STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC CX3-40F (RAID 5 MetaLUN 32 spindles)
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read5.571051732814%
RealLife-60%Rand-65%Read118.15494314%
Max Throughput-50%Read57.209743017%
Random-8k-70%Read114.58502320%
Reply
0 Kudos