VMware Cloud Community
christianZ
Champion
Champion

New !! Open unofficial storage performance thread

Hello everybody,

the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.

Oliver will make a few links between the old and the new one and then he will close the old thread.

Thanks for joining in.

Reg

Christian

574 Replies
zenomorph
Contributor
Contributor

Hi there,

I've just setup our VMware environment with vSphere 4.0 U1 withEMC CX3-80 and 3 HK DL580G5 x7460 servers and I'm trying to get some disk i/o stats on the performance for the VMF.

The thing is I'm not sure how I need to configure the disk IOmeter software such that the configuration and simulations are correct, what I've done so far is just use the "All in One" test on the C:\ drive and here are my results.

For the EMC CX3-80 configuration we've used 45*600GB FC disk on 3 DAE's with RAID5 7+1, then we've setup MetaLUN's for the VMFS across the 3DAE's ceating 700GB VMFS.

SERVER TYPE: VM Win2k8 R2

CPU TYPE / NUMBER: VCPU / 2

HOST TYPE: DL580 G5, 64GB RAM; 4 x Intel x7460

STORAGE TYPE: EMC CX3-80 / DISK NUMBER 45(3*DAE) 600GB FC / RAID LEVEL: (RAID5 7+1) MetaLun

##################################################################################

TEST NAME----


##################################################################################

'Target Type Target Name Access Specification Name # Managers # Workers # Disks IOps Read IOps Write IOps MBps Read MBps Write MBps ALL All All in one 1 1 1 1185 587 598 15 7 8 MANAGER WIN-CE3BQQ4D4HH All in one 1 1 1185 587 598 15 7 8 PROCESSOR CPU 0

WORKER Worker 1 All in one 1 1185 587 598 15 7 8

DISK C:System 1185 587 598 15 7 8

I've run the basic all-in-one test on a Win2k8 R2 (64bit) VM for 5mins and the results are as above, can anyone advise how I can setup the IOMeter to produce the results as per standard on this thread or give some advice on the stats produced from the simulation.

Many thanks........

Reply
0 Kudos
pinkerton
Enthusiast
Enthusiast

I'm currently on holiday and therefore not available until May 3rd. In urgent cases please contact support@mdm.de. Thank you.

Michael Groß, MDM IT dept.

Reply
0 Kudos
larroyd
Contributor
Contributor

SERVER TYPE: VM WIN Server 2008 x64 SP2 on ESX4 U1

CPU TYPE / NUMBER / RAM: vCPU / 2 / 4GB

HOST TYPE: IBM 3650 M2, 16GB RAM, 2x Intel X5506 2.133Ghz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: IBM DS3300 / 8x 300GB 15k / RAID 1+0

SAN TYPE / HBAs: iSCSI / QLogic 4062C

SWITCH TYPE: HP PROCURVE 2510G

Description

Avg. Response Time

Avg. I/O per Second

Avg. MB per Second

Max Throughput - 100% Read

17.22

3514

109.8

Real Life - 60% Rand / 65% Read

22.10

2558

19.9

Max Throughput - 50% Read

13.45

4521

141.2

Random 8K - 70% Read

20.94

2685

20.9

I executed this test 4 times and this is the average output

OTHER:

Disk.SchedNumReqOutstandin: 64

Round Robin (Path Selected) -- Active / Active

Enable Flow Control

Disable Jumbo Frames

CAT6 Cables

IP_ARP_Redirect = On

ExeThrottle = 128

Switches dedicated for iSCSI only

All with lastest firmware and updates.

Jumbo frames end-to-end killed my perfomance about 20%, so I disabled it. Also I ProCurve 2510G doesn't support Jumbo frams and Flow control in the same port.

Using jumbo frames the switch highest value was 77% and with no JF the highest value was 99%

Reply
0 Kudos
itpunkt
Contributor
Contributor

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE of RESULTS -Virtual Machine

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM, VMDK DISK

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: DELL R610, 16GB RAM; 2 x Intel E5540, QuadCore

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EqualLogic PS5000XV / 14+2 DISK (15k SAS) / R50)

NOTES: 3 NIC, modified ESX PSP RR IOPS parameter, jumbo on, flowcontrol on

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........__6,48__..........__9178,56__........._286,83__

RealLife-60%Rand-65%Read.......__13,08__..........__3301,94__.........__25,8__

Max Throughput-50%Read.........__9,06__..........__6160,2__.........__192,51__

Random-8k-70%Read..............__13,59__..........__3215,69__.........__25,12__

EXCEPTIONS:

Reply
0 Kudos
s_buerger
Contributor
Contributor

Ich bin vom 29.05. bis 13.06.2010 nicht im Büro. E-Mails werden nicht weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an meine Kollegen unter 0351 / 49701-150 bzw. per E-Mail saxonia.hotline@saxsys.de.

Reply
0 Kudos
s_buerger
Contributor
Contributor

SERVER TYPE: Win 2003 R2 SP2 (2GB RAM, 200 GB vmdk) on ESX 4.0u1

CPU TYPE / NUMBER: VCPU / 2

HOST TYPE: HP DL380G5 - 32GB - 2x Xeon5345 2.33GHz Quadcore

STORAGE TYPE / DISK NUMBER / RAID LEVEL: Equallogic PS4000 / 8x 1TB 7200rpm 3,5" SATA (2 Spare) / Raid 50

SAN TYPE / HBAs: iSCSI; 2x vmk <-> 2x GB pNIC (2 different Intel ET 1000 Dual Port), no Jumbo Frames, vmware Round Robin

SWITCH TYPE: Dell Powerconnect 5448 and 5424

Test Name

Avg. Response Time

Avg. I/O per Second

Avg. MB per Second

CPU Utilization

Max Throughput - 100% Read

21.75

2,709.28

84.67

11

Real Life - 60% Rand / 65% Read

42.81

1,074.93

8.39

22

Max Throughput - 50% Read

10.36

5709.23

178

15

Random 8K - 70% Read

48.92

941.38

7.35

21

Same values shown in SAN Headquartes.

Reply
0 Kudos
JonT
Enthusiast
Enthusiast

First off let me say its good to be part of a "testing" group again. Last time I worked with Oliver we were testing the VM NIC's on various platforms.

JRink,

I think everyone on this thread (and the original) has tested in various formats on various platforms. My suggestion would be that if you have a host configuration that is identical to your Production implementation, in a lab type or pre-production environment then use that. If not then use whatever you have available. The testing can be pointed to any virtual drive on your guest but generally i think the C: drive is most common.

I have run some of my own tests of a running VM, on a production host with all of our standards in place. This host is part of an 8 node cluster that is about 50% full of guests to our roughly 160 guest/cluster limitation. This 160 limitation is self-imposed. Here are my results from testing one virtual machine with a VERY average configuration. I plan to run a more thourough test using what I would consider an "optimized" guest machine, running on some very unused host capacity to truly benchmark our hosts prior to all of our production load.

++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP BL680 G5, 64GB RAM; 4x XEON E7340, 2.4 GHz, Quad-Core

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC Symmetrix / ? Disks / Raid ?

##################################################################################

TEST NAME--


Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........_13.958___.........._4189.733_........._130.929__

RealLife-60%Rand-65%Read......_47.901___.........._1071.772_........._8.373____

Max Throughput-50%Read.........._8.557____.........._6200.164_........._193.755__

Random-8k-70%Read................._47.567___.........._1121.890_........._8.764____

EXCEPTIONS: CPU Util.- 33.16% to 70.52% (varied between all 4 tests)

##################################################################################

Hope this helps JRink

Reply
0 Kudos
cdu
Enthusiast
Enthusiast

Got some crazy numbers in my home lab. I think it's the IO compression that makes the result look strange.

SERVER TYPE: Windows 2003 R2 32bit VM ON ESX 4.1, 1 vCPU, 2G ram, 20G test disk, PVSCSI.

ESX HOST TYPE: Custom - i7930, 12G ram. Qlogic 4G FC HBA 2462.

STORAGE TYPE / DISK NUMBER / RAID LEVEL: custom - Opensolaris b134, Intel XEON W3520, 12G ECC ram, 8300G 15K SAS in RAID-10. Intel X25-E SSD as slog. 2Corsair V SSD as L2ARC. Qlogic 4G FC HBA 2462

SAN TYPE / HBAs : FC Qlogic 4Gb 2462

Dedup is off on LUN, IO compression is on.

Test Name

Avg. Resp. Time ms

Avg. IOps

Avg. MBps

Max Throughput-100%Read

4.898204

12254.97145

382.967858

RealLife-60%Rand-65%Read

0.966508

54871.89216

428.686657

Max Throughput-50%Read

2.877991

20503.42329

640.731978

Random-8k-70%Read

0.995281

54142.57245

422.988847

Reply
0 Kudos
chaddy
Contributor
Contributor

Chad Anderson has concluded his employment with The Christian and Missionary Alliance. We greatly appreciate his contributions and commitment to the Alliance.

Please remove his email address from your address book. For assistance please contact David DeCoste at decosted@cmalliance.org or 719-265-2091.

Thank you.

Reply
0 Kudos
smartsysnv
Contributor
Contributor

SERVER TYPE: VM Windows 2008R2 / 4GB

CPU TYPE / NUMBER: 1 VCPU

HOST TYPE: HP DL 380 G5

STORAGE TYPE / DISK NUMBER / RAID LEVEL:HP MSA2312FC / 4x300GB 15K RAID10 / 6x300GB 15K RAID10 / 2x300GB 15K Hotspare

##################################################################################

RAID10- 6HDDs -


Av. Resp. Time ms----Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read_______5,02_______________11758_______367

RealLife-60%Rand-65%Read_____45,3______________1132_________8,8

Max Throughput-50%Read________3,95______________14800________462

Random-8k-70%Read___________43,83______________1103________6,05

#############################################################################

Rather high response time 😕

Reply
0 Kudos
chaddy
Contributor
Contributor

Thank you for your message! I will be out of the office until Sept 13, 2010 and will respond to email when I return. Have a great day!

David DeCoste

Reply
0 Kudos
Valley911
Contributor
Contributor

Smiley Happy

Reply
0 Kudos
s_buerger
Contributor
Contributor

Ich bin vom 13.09. bis 15.09.2010 nicht im Büro. E-Mails werden nicht weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an meine Kollegen unter 0351 / 49701-150 bzw. per E-Mail saxonia.hotline@saxsys.de.

Reply
0 Kudos
EllettIT
Enthusiast
Enthusiast

I may have an opportunity to run some of these test (mid November)on some Equallogic Boxes if my project proposal goes through. Is there a "dummies" walk through on how to setup the test? Thanks!

Reply
0 Kudos
ancUltimate
Contributor
Contributor

Hi folks!

Here are my tests but results are some kind of disappointing. I saw some PS5000E (with SATA Drives) results here which looked much better. Can somebody give me a hint, please?

Server Type: VM Windows Server 2008 R2 x64 Std. on VMware ESXi 4.1

CPU Type / Number: vCPU / 1

VM Hardware Version 7

Two vmxnet3 NICs (10 GBit) used for iSCSI Connection (10 GB LUN directly connected to VM, no VMFS/RDM)

MS iSCSI Initiator (integrated in 2008 R2)

SAN Type: EQL PS6000XV (14+2 SAS HDDs, 15K, RAID 50)

Switches: Dell PowerConnect 6224

ESX Host is equipped with four 1GBit NICs (only for iSCSI connection)

Jumbo Frames and Flow Control enabled.

##################################################################################

Test--


Av. Resp. Time ms--Total IOs/sek---Total MB/sek----

##################################################################################

Max Throughput-100%Read........___10.1929_____.......___4967.06_____.....____155.22______

RealLife-60%Rand-65%Read......_____12.6970____.....____3933.39____.....____30.73______

Max Throughput-50%Read.........____9.5941____.....____5115.05____.....____159.85______

Random-8k-70%Read..............____12.9845_____.....____4030.60______.....____31.49______

The values with VMFS or RDM volumes are even lower.

Reply
0 Kudos
JonT
Enthusiast
Enthusiast

ancUltimate,

You may want to try running your tests again with Jumbo Frames disabled but leave Flow Control enabled. I have seen a trend of this being an issue, especially when using iSCSI LUN devices. Also you could consider mounting it as an RDM and let the host do the iSCSI work instead of the guest O/S. You may have a slight bit of overhead added by using the guest Windows for iSCSI but not sure if it will make a difference. Lastly make sure that your guest is not being starved for CPU resources. If this is the only guest on this host then don't worry, but if there are other guests running the IO Meter tests are very CPU intensive and I have seen guests perform differently depending on how much CPU resource time they get on the host.

Hope this helps!

Reply
0 Kudos
ancUltimate
Contributor
Contributor

Hi JonT,

Jumbo Frames are currently enabled on virtual NICs in guest OS, the vSwitch and vmk Ports in ESXi and the physical SAN Switches. Do you mean to disable Jumbo Frames on all devices?

Flow Control is enabled on physical switches but I see no option for it in vmxnet3 configuration (Advanced Tab).

If I use a RDM in virtual guest, the performance is even worse.

CPU should be not a problem because there are only very few machines on this ESXi Host, currently used for testing purposes only.

Addition:

I have a Windows 2003 R2 Server VM with equal configuration like the 2008 R2 VM above mentioned. MS iSCSI initiator 2.08 is and Dell MPIO installed. This VM gets only half of the performance values (IOPS, MB/s) of the 2008 R2 VM in Max Throughput 100% Read test. I mean exactly half of the values. Weird!

There must be some misconfiguration or what else but I can't find anything in this direction. Has anybody seen such behavior?

Following values result from testing the 2003 VM:

Max Throughput 100% Read - 50% IOPS and MB/s of 2008 R2 VM

Real Life - 5% less IOPS and MB/s than 2008 R2 VM

Max Throughput 50% Read - 30% less IOPS and MB/s than 2008 R2 VM

Random - 5% less IOPS and MB/s than 2008 R2 VM

In above mentioned cases I was testing only one single VM on SAN.

Reply
0 Kudos
Valley911
Contributor
Contributor

Smiley Happy

Reply
0 Kudos
EllettIT
Enthusiast
Enthusiast

ESX should auto detect flow control for the NIC's dedicated to your iSCSI SAN. There's a command you can run from the SC to show if it's active or not, I can't remember it off the top of my head but I'll post back later once I find it. I did run across a "guide" of sorts for setting up that switch:

http://www.delltechcenter.com/thread/3521967/PowerConnect6224stackconfigurationfor+EqualLogic

http://sites.google.com/site/mellerbeck/Home/Dell_EqualLogic_Configuration_Guide.pdf

Reply
0 Kudos
JonT
Enthusiast
Enthusiast

(i hate seeing all the OOO notices posted to these threads....)

ancUltimate,

First my suggestion for disabling Jumbo Frames would only be on the Guest O/S. Leave it enabled on the switch and the vSwitch on the host. As for performance between 2008 and 2003, my guess would be the NIC driver in the vmtools, without knowing specifically. All of the tests I ran IOMeter on were using an x86 2003 vm, with 1 vCPU and 1GB of RAM. The guest I used had 2 basic disks on 1 vmdk file, stored on the SAN in a vmfs datastore. My results were similar to what you posted for your 2k8 R2 machine so I am not sure that you are seeing any really terrible IOPs results here. Keep in mind that also your response times in ms are pretty good considering the overhead from vmkernel and iSCSI, using the 4 1xGB NIC's.

Lastly, how are those 4x 1GB NIC's configured? How many vSwitches and how many vmkernel Interfaces? I would hope that you have all 4 in some sort of "port channel" with the upstream switch so the bandwidth is aggregated, otherwise you may not be seeing the full utilization of your network ports. Honestly 2x 1GB NIC's is what I have connected to my hosts for the Service Console/vmkernel interfaces, and 2x 1GB NIC's for a separate vSwitch that operates only the guest port groups (VLAN tagged networks).

(moral of this long winded story) When using iSCSI you may need to make sure you have adequate separation of the vmkernel and guest traffic on the ports. If that means splitting your vSwitch configuration in half, you may actually see a perf gain.

Hope this helps!

JonT

By the way, sorry to hear that your results with an RDM were actually worse. That maybe highlights an issue within your guest OS and its storage drivers that are installed. We use RDM's connected as "physical" devices and the guest OS uses its own multi-path/storage management software for that LUN. Our performance of IOPs on RDM's is typically about 20% better than using a vmdk file for an additional disk, typically used for SQL servers.

Reply
0 Kudos