New !! Open unofficial storage performance thread

[christianZ]

Hello everybody,

the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.

Oliver will make a few links between the old and the new one and then he will close the old thread.

Thanks for joining in.

Reg

Christian

[larstr]

Something must be wrong here. You get only 500 IOPS (Random & RealLife) with 32 FC drives. These numbers are not very good, so you should probably check if there's anything wrong with your SAN setup.

Lars

[_VR_]

eagleh,

The performance numbers for the 32 disk SAN are very poor. You should be seeing much better numbers for RealLife/Random. The Max Throughput Read test looks good so I don't think the FC network is at fault. Were there any heavy writes from other systems while the test was performed? A failed disk in a RAID 5 can also have the same affect (fast reads/slow writes).

[eagleh]

Thanks VR!!! I am engaging HP engineer to look into this issue now.

[eagleh]

Thanks larstr. I am engaging HP to look into this issue now.

[Valley911]

Good thread, not sure if I am reading the output correctly in the CSV file. But here is what I get from my EMC CX4-120:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM ON ESX 3.5 U3

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: HP DL380 G5, 24GB RAM; 4x XEON 5410(Quad), 2,33 GHz,

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC CX4-120 / 41 / R5 / 141total disks

SAN TYPE / HBAs : 4GB FC HP StorageWorks FC1142SR (Qlogic)

MetaLUNS are configured with 200GB LUNs striped accross all 14 disks for total datastore size of 600GB

##################################################################################

TEST NAME--

---

Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read...............__6______..........___9320___.........___291____

RealLife-60%Rand-65%Read......___24_____..........__1638___.........____13____

Max Throughput-50%Read................____5____..........___11057___.........___345____

Random-8k-70%Read.................____23____..........___1800___.........____14____

EXCEPTIONS:

#################################################################################

-Jason

[tomatdt]

ran iometer with "OpenPerformanceTest.icf" found in this thread, however can't sort out how to produce the summary "TABLE OF RESULTS" users are posting here. what am i missing?

(great information here, thanks)

[JohnADCO]

don't take them from the produced output. The numbers are right on the summary screen after the test is done right? Not sure beyond that?

[tomatdt]

everyone running each of the 4 test scenarios one-by-one then cutting and pasting results into a template?

[misteriks]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM windows 2008 enterprise.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 32GB RAM; 2x XEON 5450, 3,00 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 1 / 14+2 Disks / R50

MTU: 1500

#####################################################################

TEST NAME--

---

Av. Resp. Time msAv. IOs/sekAv. MB/sek---AV. CPU Utl.

Max Throughput-100%Read.............45,0..................1257,5................38,0..................37.........

RealLife-60%Rand-65%Read.........21,7...................2237,8................17,5..................52.........

Max Throughput-50%Read...............23,5...................1672,9.................50,4.................96.........

Random-8k-70%Read......................23,7...................2087,9.................16,3.................47.........

SERVER TYPE: VM windows 2008 enterprise.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 32GB RAM; 2x XEON 5450, 3,00 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 1 / 14+2 Disks / R50

MTU: 9000

####################################################################

TEST NAME--

---

Av. Resp. Time msAv. IOs/sekAv. MB/sek---AV. CPU Utl.

Max Throughput-100%Read.............16,3.................3638,3..............113,7.................35.........

RealLife-60%Rand-65%Read.........21,7..................2237,8................17,5.................43.........

Max Throughput-50%Read..............17,7...................2200,6................67,8.................80.........

Random-8k-70%Read......................23,6..................2098,4................16,3.................41.........

It suprise's me that the difference in Av. IOs/sek and Av. MB/sek with the different MTU settings is only there for the Max Throughput-100%Read test. Is this expected behavior? The CPU usage is much lower with an MTU of 9000.

How are these figures in general for this setup? Are they normal or can they be higher?

[misteriks]

sorry, don't know what happend with my post above. Let me try again.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: VM windows 2008 enterprise.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 32GB RAM; 2x XEON 5450, 3,00 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 1 / 14+2 Disks / R10

MTU: 1500

#####################################################################

TEST NAME--

---

Av. Resp. Time msAv. IOs/sekAv. MB/sek---AV. CPU Utl.

Max Throughput-100%Read.............45,0..................1257,5................38,0..................37.........

RealLife-60%Rand-65%Read.........21,7...................2237,8................17,5..................52.........

Max Throughput-50%Read...............23,5...................1672,9.................50,4.................96.........

Random-8k-70%Read......................23,7...................2087,9.................16,3.................47.........

#####################################################################

SERVER TYPE: VM windows 2008 enterprise.

CPU TYPE / NUMBER: VCPU / 1

HOST TYPE: Dell PE2950, 32GB RAM; 2x XEON 5450, 3,00 GHz

STORAGE TYPE / DISK NUMBER / RAID LEVEL: EQL PS5000E x 1 / 14+2 Disks / R10

MTU: 9000

####################################################################

TEST NAME--

---

Av. Resp. Time msAv. IOs/sekAv. MB/sek---AV. CPU Utl.

Max Throughput-100%Read.............16,3.................3638,3..............113,7.................35.........

RealLife-60%Rand-65%Read.........21,7..................2237,8................17,5.................43.........

Max Throughput-50%Read..............17,7...................2200,6................67,8.................80.........

Random-8k-70%Read......................23,6..................2098,4................16,3.................41.........

####################################################################

It suprise's me that the difference in Av. IOs/sek and Av. MB/sek with the different MTU settings is only there for the Max Throughput-100%Read test.

Is this expected behavior? The CPU usage is much lower with an MTU of 9000, so that is good

How are these figures in general for this setup? Are they normal or can they be better?

[dennes]

Those results seem very poor. Even at MTU 1500 (no jumbo frames) you should have much better IOPs and throughput.

Check out the previous posts/thread for comparable setups.

[JohnADCO]

If he is using esxi and the software initiator, The numbers ain't so bad.

[dennes]

Really? I didn't know there was a performance penalty in using esxi as opposed to esx?

[JohnADCO]

No nic teaming / load balancing. Every thing goes out one nic.

I think I am right on that one. The software initiator produces low numbers on IO meter compared to an HBA card.

[misteriks]

Thx for the responses. I'm trying to setup the boxes (Dell PE 2950) with the best performance before I go into production.

I tested on a virtual windows 2008 Enterprise running @ ESX 3.5 (no other guests running). I ran the tests with a single vNic and also with 2 vNics with MPIO on the MS initiator 2.0.8. results are almost the same.

The vSwitch of the iSCSI segment has 2 intel pNics connected. 1 on a intel Quad nic and the second port on the 2nd Intel quad nic. The cables are connected to a 1 of the 2 Dell PowerEdge 6224 switches that are stacked togehter. Jumbo frames and Flow control are enabled everywhere. The only thing I'm not sure of is the vNic. I choose the enhanced vmxnet adapter but i'm only able to configure the MTU and 2 other values but eg not the flow control setting.

If I look to the other EQL results in this thread something tells me I can get better results. Maybe I should do a test with a host direct connected to the EQL to see if the network may be a issue.

http://communities.vmware.com/message/1047880#1047880

http://communities.vmware.com/message/689555#689555

[christianZ]

The numbers I reached with Win2003 were better - the question here is what is with win2008?

Have you enabled "flow control" on your switches?

Direct attaching to PS series is not recommended (because of automatic load balancing on Eql box). Have you a chance to create a Win2003 vm - I would check whether that makes any differences.

[radimf]

Hallo,

I run into exactly same problem with EQL 5000 series from Windows 2008.

I was testing 4!!! EQL 5000 boxes joined into one SAN in a local DELL

showroom, and results were also abyssmal for maximum throughput numbers.

(20-50 Mb/S) I expected over 300 mb/s!!!!!!!!

We tested from brand new Nehalem Xeon box with Windows 2008 64.bit -

16GB RAM.

6xNIC with TOE joined in MPIO.

Look at your random numbers - they are good, very good!

Look at your throughput - terrible. Exactly the same as my results.

We were not able to get past 1gbit performace in throughput tests.

If I had to bet - there is some glitch in round robin MPIO with W2k8

default config.

We had no time to reinstall server to W2k3.

There has to be some patch we did not know about during our testing, or

there is some missconfig in default configuration after install.

In our case all of 6 NICs were used, but it maxed out even below 1Gbit

performance numbers.

If anyone knows the solution - please post it - I was almost buyer of

new PS6000 series EQL box, but this issue forces me to postpone my

purchase.

Regards

[Mnemonic]

I do not understand you results.. They should not be able to perform like that. What disks are you using?

Is you OS caching somehow?

[_VR_]

radimf,

Take a look at my results for the PS6000XV. The unit can easily saturate each of the NICs. My tests were performed with 2 nics and software iscsi, but the results were still impressive. The network switch I used was a Cisco 6500.

[_VR_]

Mnemonic,

I thought it was OS caching at first, but when I ran tests to the SAN from the same machine there was no improvement over what I've seen on tests from other servers. If it was OS caching it would cache reads from the SAN as well. I've repeated the tests many times yet the subsequent tests showed even better results (39,000+ iops for both Max Throughput tests). Perfmon also confirmed disk read's at 1200MB/s as well as 39000 IOPS. The disks are HP OEM 2.5" 10k disks.

I agree that these disks can not perform this fast. I'm not sure what else can be done to get more accurate numbers.