Hello everybody,
the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.
Oliver will make a few links between the old and the new one and then he will close the old thread.
Thanks for joining in.
Reg
Christian
Long time FC shop decided to deploy iSCSI for remote office. iSCSI performing surprisingly well.
Physical assets: 2x HP P4300G2 (Lefthand) ver. 8.5, 2x HP 2910AL, 2x HP DL360G6
Network: All 1GbE, 2 Ports each
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: VM Windows 2008 R2 Standard (Version 7), 1x vCPU, 4 GB RAM, 2x 40GB VMDKs
HOST TYPE: HP Proliant DL360 G6, 36GB RAM, 2x X5650
VMWARE: ESXi 4.1 on USB Drive, Software iSCSI Intiator on 2x VMkernel
SAN Type: HP P4300G2 (LeftHand) / Disks: 450GB 10k SAS / RAID LEVEL: Raid5 / 8 Disks
##################################################################################
Test Name | Average Response Time (ms) | Average I/O per Sec. | Average MB per Sec. |
Max Throughput - 100% Read | 1.75 | 25,916.19 | 809.88 |
Real Life - 60% Random 65% Read | 35.93 | 1,426.11 | 11.14 |
Max Throughput - 50% Read | 17.48 | 3,333.33 | 104.17 |
Random - 8K 70% Read | 15.59 | 1,627.12 | 18.15 |
Christian,
For automatic interpretation of iometer results.csv I've setup a web page here:
I also put the .iso file from the older thread in there.
Lars
Bis zum 13.12. bin ich nicht im Büro. Bitte wenden Sie sich in dieser Zeit an support@mdm.de
Best Regards
Michael Groß
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 1.97 | 28525 | 891 | 71% |
RealLife-60%Rand-65%Read | 24.32 | 1394 | 10 | 59% |
Max Throughput-50%Read | 2.33 | 24523 | 766 | 55% |
Random-8k-70%Read | 25.47 | 1392 | 10 | 56% |
Here are my results. I can't determine why our performance is degraded when using jumbo frames on our MD3000i. We're using (2) PowerConnect 5424's for our dedicated iSCSI network and jumbo frames are enabled from end-to-end.
SERVER TYPE: Windows Server 2003R2x32 VM running on ESXi 4.1
CPU TYPE / NUMBER: VCPU x 1, JUMBO FRAMES, MPIO RR
HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5620
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 19.61 | 3036 | 94 | 20% |
RealLife-60%Rand-65%Read | 19.24 | 2558 | 19 | 37% |
Max Throughput-50%Read | 19.62 | 3061 | 95 | 19% |
Random-8k-70%Read | 18.34 | 2728 | 21 | 37% |
SERVER TYPE: Windows Server 2003R2x32 VM running on ESXi 4.1
CPU TYPE / NUMBER: VCPU x 1, JUMBO FRAMES, MPIO RR
HOST TYPE: PowerEdge R710, 24GB RAM, 2xE6520
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 19.11 | 3122 | 97 | 20% |
RealLife-60%Rand-65%Read | 22.94 | 2027 | 15 | 40% |
Max Throughput-50%Read | 19.30 | 3105 | 97 | 19% |
Random-8k-70%Read | 21.28 | 2103 | 16 | 43% |
SERVER TYPE: Windows Server 2003R2x32 VM running on ESXi 4.1
CPU TYPE / NUMBER: VCPU x 1, NO JUMBO FRAMES, MPIO RR
HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5520
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 15.20 | 3942 | 123 | 18% |
RealLife-60%Rand-65%Read | 19.02 | 2514 | 19 | 39% |
Max Throughput-50%Read | 17.09 | 3585 | 112 | 15% |
Random-8k-70%Read | 18.94 | 2558 | 19 | 38% |
SERVER TYPE: Windows Server 2003R2x32 VM running on ESXi 4.1
CPU TYPE / NUMBER: VCPU x 1, NO JUMBO FRAMES, MPIO RR
HOST TYPE: PowerEdge R710, 24GB RAM, 2xE5520
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 15.13 | 3957 | 123 | 18% |
RealLife-60%Rand-65%Read | 23.95 | 2090 | 16 | 32% |
Max Throughput-50%Read | 16.60 | 3682 | 115 | 15% |
Random-8k-70%Read | 22.81 | 2107 | 16 | 36% |
===========================
===========================
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 4 128 1500 vmnic4,vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 6 128 1500 vmnic9,vmnic8,vmnic3,vmnic2
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 1 vmnic9,vmnic8,vmnic3,vmnic2
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 5 128 9000 vmnic5,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI2 0 1 vmnic5
iSCSI1 0 1 vmnic1
============================
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 8.12 | 7410 | 231 | 29% |
RealLife-60%Rand-65%Read | 10.65 | 3347 | 26 | 59% |
Max Throughput-50%Read | 7.19 | 7861 | 245 | 34% |
Random-8k-70%Read | 11.37 | 3387 | 26 | 55% |
Sono fuori ufficio.
Rientrerò il 10/01/2011.
Per urgenze contattare l'assistenza tecnica allo 045 8738738.
Francesco Bonetti
RTC SpA
Bis zum 03.01. bin ich nicht im Büro. Bitte wenden Sie sich in dieser Zeit an support@mdm.de.
I'm out of office until January 3rd. Please contact support@mdm.de instead.
Mit freundlichem Gruß / Best Regards
Michael Groß
MDM IT Abteilung / MDM IT department
My results have been crammed into a spreadsheet that I sent around to some of the guys I work with, I'm not going to take the time to format them the same way everyone else has been doing... sorry for that.
Background:
SERVER TYPE: Windows 2003 Standard R2 (aligned to 1024K)
CPU TYPE / NUMBER / Memory: VCPU / 2 / 1024M
HOST TYPE: Dell PowerEdge R510, 32GB RAM, 2 x Intel L5520, 2.266GHz, QuadCore, Running ESXi 4.1
STORAGE TYPE / DISK NUMBER / RAID LEVEL: DAS PERC H700 / (6) 300GB SAS 15k / RAID 10
STRIPE UNIT SIZE: Default 64k, 256k, 512k, 1024k
VMFS ALIGNMENT: Default 64k, 256k, 512k, 1024k
VMFS BLOCK SIZE: 8M for all except the first one that was run looking at all the default settings and a misaligned guest
I've attached my results as a spreadsheet.
I was interested in seeing that the stride used in my RAID array had a way bigger effect on performance than what guest alignment did. Although I suspect that alignment of the guests would be a bigger deal as I start adding more VMs. Also, back in ESX3 they had a best practice of setting your Windows guests up to use a FAU (File Allocation Unit) of 32k. This just doesn't seem to have a big effect at all on disk I/O.
So for our setup where we are going to use DAS instead of a SAN, I believe we will:
1. Run our RAID strides to 1024k
2. Align any new VMs but disregard alignment on existing ones (maybe... jury is still out)
3. Not bother with setting regular (non DB) VMs up to use an FAU of 32k
4. Not bother with re-aligning the VMFS volumes to match our strides (it helps but not that much)
5. Continue to use 8M blocks on datastores since that is what we have already been doing for a while.
Has anyone else noticed any real benefit to setting the guest OS to use FAU of 32k?
Hi everybody,
here are my test results. We are using a NetApp FAS 3140 and the datastores sit in a volume that's made available via NFS. The ESXi 4 host has two 1Gbps connections to the filer. The maximum throughput is somewhere near the limit of a 1 Gbps connection - as one would expect. The random throughput is near 20 MB/s.
These numbers look normal, or what do you think?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TABLE oF RESULTS
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SERVER TYPE: ESXi 4, VM Windows 2003 Standard
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP ProLiant BL460c G6, 48GB RAM; 2x Intel Xeon E5540, 2.53 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: NetApp FAS3140 / 13 Disks (2 Parity) / RAID DP
SAN TYPE: Storage Link 2xEthernet 1Gb / NFS / NO JUMBOFRAMES / NO FLOWCONTROL
##################################################################################
TEST NAME--
--------------------------------------------------------------------------------
Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----
##################################################################################
Max Throughput-100%Read........__17.5395__.........._3431.58_........._107.24__
RealLife-60%Rand-65%Read......__19.9659__.........._2588.61_........._20.22__
Max Throughput-50%Read..........__12.2198__.........._4921.79_........._153.81__
Random-8k-70%Read.................__19.6347__.........._2489.96_........._19.45__
EXCEPTIONS: CPU Util.-%;
##################################################################################
Thanks in advance for your hints.
Alex
A quick test of a recent Nexenta build.
Dedupe and compression both turned off.
Using direct-connect iSCSI with software initiator on Win7 64bit client prior to testing same box with vSphere.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 16.74 | 3571 | 111 | 0% |
RealLife-60%Rand-65%Read | 7.13 | 6187 | 48 | 0% |
Max Throughput-50%Read | 12.34 | 4482 | 140 | 0% |
Random-8k-70%Read | 6.50 | 6198 | 48 | 0% |
SERVER TYPE: W2K3 R2 SP2 x32 1vCPU 2GB RAM
CPU TYPE 2 X Nehalem 7560 8Core @2.27GHz
HOST TYPE: IBM X3960 X5 128GB RAM (2)8GB FC - (2)10GB Eth
STORAGE TYPE / DISK NUMBER / RAID LEVEL: XIV/180/NORAID
Test Type | Latency | Avg IOPS | Avg MBps | CPU Load |
---|---|---|---|---|
Max Throughput: 100% Read | 12.03 | 5320 | 166 | 16% |
Real Life 60% Rand-Read | 3.96 | 16177 | 126 | 43% |
Max Throughput 50% Read | 3.65 | 17545 | 137 | 47% |
Random 8k-70% Read | 11.05 | 5793 | 181 | 19% |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 4.07 | 11562 | 361 | 63% |
RealLife-60%Rand-65%Read | 1.67 | 22901 | 178 | 1% |
Max Throughput-50%Read | 3.93 | 11684 | 365 | 61% |
Random-8k-70%Read | 1.45 | 25509 | 199 | 1% |
Maybe following is interessting for you
or the different block sizes for different ms applications (sql ...)
After having some trouble with storage performance as described in http://communities.vmware.com/message/1680855#1680855 and moving the Heavy Hitter to a different set of disks, we additionally decided to buy four more drives and move the VMFS LUNs on our HP EVA 4400 from RAID5 to RAID1.
Here are my results:
SERVER TYPE: VM Windows 2008 R2, 4 GB RAM, LSI Logic SAS
CPU TYPE / NUMBER: 2 VCPU
HOST TYPE: ESXi 4.1 320137, HP DL380 G6, 64GB RAM, 2x E5520, 2,27 GHz QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP EVA 4400 / 32/36x FC 10k 300GB / RAID5 and RAID10
SAN TYPE / HBAs : FC, HP FC1142SR QLogic HBA, HP StorageWorks 8/8 SAN Switches
#################################################################################################
Vraid5 / 32 HDDs
Test Name | Response Time | IOPS | MB/s |
---|---|---|---|
Max Throughput-100%Read | 2,67 | 10.809 | 337,78 |
RealLife-60%Rand-65%Read | 8,29 | 3.055 | 23,87 |
Max Throughput-50%Read | 34,03 | 1.530 | 47,81 |
Random-8k-70%Read | 8,64 | 2.866 | 22,39 |
Vraid1 / 36 HDDs
Test Name | Response Time | IOPS | MB/s |
---|---|---|---|
Max Throughput-100%Read | 5,09 | 11.686 | 365,20 |
RealLife-60%Rand-65%Read | 10,84 | 4.284 | 33,47 |
Max Throughput-50%Read | 6,63 | 5.113 | 159,78 |
Random-8k-70%Read | 10,41 | 4.783 | 37,37 |
As you can see, especially test 3 and 4 run MUCH faster with Vraid1 instead of Vraid5! It actually seems that the RAID Level is more important for performance than the amount of disks!
SERVER TYPE: VM Windows 2008 R2, 2 GB RAM, LSI Logic SAS CPU TYPE / NUMBER: 2 VCPU HOST TYPE: ESXi 4.0 U1, HP DL380 G7, 96GB RAM STORAGE TYPE / DISK NUMBER / RAID LEVEL: IO Accelerator 160GB ("SSD DISK")
Test Name | Response Time | IOPS | MB/s |
---|---|---|---|
Max Throughput-100%Read | 2,67 | 22.743 | 710 |
RealLife-60%Rand-65%Read | 1.41 | 39.721 | 310 |
Max Throughput-50%Read | 3.64 | 16.031 | 500 |
Random-8k-70%Read | 1.55 | 36.861 | 287 |
1538moss, these numbers are impressive but I wonder why the IOPs were so low for the IO Accelerator device. Your throughput numbers indicate great bandwidth to the disk but the overal IOs were much lower that I had experienced with a 360GB Duo version of the IO Accelerator. Here is what I observed when using this SLC memory type of disk. Keep in mind that this is an unrealistic storage type for our mainstream production VM's, but it does present a great solution to the most disk intensive applications such as Active Directory or SQL.
I only ran the "RealLife-60%Rand-65%Read" and the 100% Read test, but the numbers were pretty nice:
SERVER TYPE: VM Windows 2008 SP2 x64, 4 GB RAM, LSI Logic SAS
CPU TYPE / NUMBER: 2 VCPU
HOST TYPE: ESXi 4.1, HP DL380 G7, 32GB RAM
STORAGE TYPE / DISK NUMBER / RAID LEVEL: IO Accelerator 360GB Duo SLC
Test Name | Response Time | IOPS | MB/s |
Max Throughput-100%Read | 0.500 | 319,321.40 | 155.9 |
RealLife-60%Rand-65%Read | 6.11 | 100,218.33 | 48.9 |
Max Throughput-50%Read | N/A | N/A | N/A |
Random-8k-70%Read | N/A | N/A | N/A |
@JonT
You have very high IOPS and a low values for MB/s. Your configuration seems to use a blocksize of 512 bytes and not 32k for "Max Throughput..".
The calculation is: 155.9*1024*1024/319,321.40 is about 512 and it should be 32k.
The same ist true for the "real life" where the result shoud be 8k and we got again 511 with your results.
You may check with one of my results (see below).
Andreas
SERVER TYPE: VM Windows 2003 SP2 x86, 2 vCPU, 1 GB RAM, paravirtual
HOST TYPE: ESXi 4.0 (332073), Fuijtsu RX300S6, 48 GB RAM
STORAGE TYPE / DISK NUMBER / RAID LEVEL: OpenSolaris B134, 64 GB RAM / 12 SAS 146 GB / 150 GB, 8k recordsize, RAIDZ1 with 4 vDevs, 32 GB SLC SSD as writezilla, 160 MLC SSD as readzilla
STORAGE NETWORK / PROTOCOL: Infiniband DDR, 20 Gb/s / SRP
Test Name | Response Time | IOPS | MB/s | CPU% |
Max Throughput-100%Read | 3.57 | 16,856 | 526.7 | 26 |
RealLife-60%Rand-65%Read | 2.1 | 27,800 | 210.8 | 34 |
Max Throughput-50%Read | 2.98 | 19,900 | 624.7 | 30 |
Random-8k-70%Read | 2.10 | 24,582 | 192.0 | 39 |
SERVER TYPE: Windows 2003 VM ON ESXi 4.1
HOST TYPE: HP DL380G6
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Hp LeftHand 4 Nodes across 2 locations. Local RAID5, iSCSI, Network RAID 1+0. 12 SAS disks in each node.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 17.29 | 3429 | 107 | 30% |
RealLife-60%Rand-65%Read | 13.54 | 3149 | 24 | 61% |
Max Throughput-50%Read | 13.08 | 3816 | 119 | 52% |
Random-8k-70%Read | 8.67 | 4287 | 33 | 70% |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 5.57 | 10517 | 328 | 14% |
RealLife-60%Rand-65%Read | 118.15 | 494 | 3 | 14% |
Max Throughput-50%Read | 57.20 | 974 | 30 | 17% |
Random-8k-70%Read | 114.58 | 502 | 3 | 20% |