Table over results: Physical parameters Virtual Parameters Max Throughput-100% Read RealLife-60%-Rand-65% Read Storage Raid Phys Host vscsi typ...
See more...
Table over results: Physical parameters Virtual Parameters Max Throughput-100% Read RealLife-60%-Rand-65% Read Storage Raid Phys Host vscsi type vmdk Latency Avg iops Avg MBps cpu load Latency Avg iops Avg MBps cpu load Local disks Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 Dell PowerEdge R710 LSI Logic SAS 40 gb thick 2,66 22429 700 88 16,18 2957 23 80 Local disks Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 Dell PowerEdge R710 LSI Parallel 40 gb thick 3 19621 613 0 16,04 3018 23 27 Local disks Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 Dell PowerEdge R710 Vmware Paravirtual 40 gb thick 2,15 27769 867 32 15,9 3011 23 7 iSCSI SAN Md3000i 4 disk dg/vd - Raid 5 Dell PowerEdge R710 LSI Logic SAS 40 gb thick 15,38 3907 122 16 36,71 1108 8 45 iSCSI SAN Md3000i 4 disk dg/vd - Raid 5 Dell PowerEdge R710 LSI Parallel 40 gb thick 15,32 3904 122 0 35,71 1131 8 25 iSCSI SAN Md3000i 4 disk dg/vd - Raid 5 Dell PowerEdge R710 Vmware Paravirtual 40 gb thick 15,14 3967 123 1 35,07 1119 8 18 iSCSI SAN Md3000i 2 disk dg/vd - Raid 1 Dell PowerEdge R710 LSI Logic SAS 40 gb thick 15,17 3958 123 17 52,25 902 7 34 iSCSI SAN Md3000i 14 disk dg/vd - Raid 10 Dell PowerEdge R710 LSI Logic SAS 40 gb thick 17,14 3520 110 16 15,45 3696 28 18 iSCSI SAN Md3000i 14 disk dg/vd - Raid 5 Dell PowerEdge R710 LSI Logic SAS 40 gb thick 17,06 3535 110 16 19,49 2542 19 29 Local SSD no raid - ESB2 intel - Crucial RealSSD C300 2,5" 128gb Dell Precision T5400 n/a n/a 7,15 8243 257 11 6,68 8629 67 9 Local SSD no raid - ICH9 intel - Intel 80gb G2 M Dell Latitude E6400 n/a n/a 9,31 6402 200 35 16,26 3305 25 56 Local disks Perc 5/i - 4 disks 300gb sas 15k raid 5 Dell PowerEdge 2950 n/a n/a 3,64 17175 536 5 37,42 1197 9 3 Hosts used in test: Host Model Cpu Memory i/o controller Local disk(s) OS NIC vSphere server Dell PowerEdge R710 2 x Intel Xeon X5680 3.33 Ghz 6 core, 12M cache, 6.40 GT/s QPI 130W TDP, Turbo, HT 96 GB memory for 2 cpu (12 x 8 GB Dual Rank RDIMMs) 1333 MHz Perc H700 Intergrated, 1 GB NV Cache, x6 Backplane 1 x SDcard 6 x 600 GB SAS 6 Gbps 15K 3.5” , raid 10 VMware ESXi 4.1.0 build 348381 on SDcard Embedded broadcom Gbe LOM with TOE and iSCSI offload (4 port) & Intel Gigabit ET Quad port server adapter PCIe x4 & Intel X520 DA 10GbE Dual Port PCIe x8 Workstation Dell Precision T5400 1 x Xeon E5440 2,83 Ghz quad core 16 GB Memory fully buffered dimm Intel 5400 chipset (intel ESB 2 Sata raid controller) 1 x Crucial RealSSD C300 2.5” 128GB Sata 6 gb/s Windows 7 Enterprise x64 Broadcom 57xx & Intel Pro 1000 PT dual SA Laptop Dell Latitude E6400 1 x 2.53 Ghz Intel Core 2 Duo 4 GB Memory Intel ICH9 1 x Intel 80gb SSD gen2 M Windows 7 Enterprise x64 Intel 82567 Physical server Dell PowerEdge 2950 2 x Intel Xeon 5150 - 2.66 ghz dual core 4mb L2 cache 16 gb memory 533 mhz Perc 5/i 4 x 300 gb 15k sas, raid 5 Windows 2008 R2 Enterprise Broadcom BCM5708C NetExtreme II & Intel Pro/1000 PT Dual Port SA iSCSI SAN used in test: Dell PowerVault MD3000i – 15 x 600gb sas 15k (one global hotspare) 2 x PC5424 Dell PowerConnect switches (2 isolated iscsi subnets as recommended for MD3000i) LAN switches: Cisco 2960 series and Nexus 5010 Virtual Machines used: VM OS vcpu scsi vmdk Memory NIC VM HW vers Iometer01 Windows 2008 R2 SP1 (x64) 1 (default) LSI Logic SAS (default) 40 gb thick 4 gb Vmxnet 3 7 Iometer02 Windows 2003 R2 Sp2 x64 2 LSI Logic Parallel 40 gb thick 8 gb Vmxnet 3 7 Iometer03 Windows 2008 R2 SP1 (x64) 2 Paravirtual 40 gb thick 8 gb E1000 (default) 7 Comparison between virtual scsi types and guest OS iometer performance Comparison of VM Iometer01,02,03 running on local server disks to see if there is a noticeable difference on different guest OS and virtual scsi adapter type: SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 2.66 22429 700 88% RealLife-60%Rand-65%Read 16.18 2957 23 80% Max Throughput-50%Read 1.38 42340 1323 63% Random-8k-70%Read 17.52 2745 21 38% SERVER TYPE: VM iometer02 - W2K3 R2 SP2 x64 - LSI Parallel CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 3.00 19621 613 0% RealLife-60%Rand-65%Read 16.04 3018 23 27% Max Throughput-50%Read 1.34 39659 1239 0% Random-8k-70%Read 17.56 2751 21 26% SERVER TYPE: VM iometer03 - W2K8 R2 SP1 x64 - VMware Paravirtual CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 2.15 27769 867 32% RealLife-60%Rand-65%Read 15.90 3011 23 7% Max Throughput-50%Read 1.22 48797 1524 48% Random-8k-70%Read 17.50 2738 21 7% Logs: iometer01-local-01, iometer02-local-01, iometer03-local-01 Comparison of same VMs on MD3000i SAN: SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 15.38 3907 122 16% RealLife-60%Rand-65%Read 36.71 1108 8 45% Max Throughput-50%Read 12.40 4816 150 17% Random-8k-70%Read 40.56 1103 8 41% SERVER TYPE: VM iometer02 - W2K3 R2 SP2 x64 - LSI Parallel CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 15.32 3904 122 0% RealLife-60%Rand-65%Read 35.71 1131 8 25% Max Throughput-50%Read 16.82 3644 113 0% Random-8k-70%Read 40.50 1107 8 27% SERVER TYPE: VM iometer03 - W2K8 R2 SP1 x64 - VMware Paravirtual CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 15.14 3967 123 1% RealLife-60%Rand-65%Read 35.07 1119 8 18% Max Throughput-50%Read 12.44 4791 149 1% Random-8k-70%Read 41.34 1105 8 12% Logs: iometer01-san-01, iometer02-san-01, iometer03-san-01 Comment: Different windows server OS and virtual scsi adapter types does not change the performance in a dramatic way (adding disks or changing raid systems has a much larger impact) However it looks like LSI Logic SAS uses a lot more cpu than the other types of virtual scsi adapters, I only did each test once so maybe more tests are needed to confirm that. Local server disks are much faster than the cheap MD3000i iscsi SAN for single VM performance Note: SAN diskgroup and virtual disk was only using 4 disks on the SAN box so the results cannot be directly compared to server disks Dell PowerVault Md3000i iSCSI SAN iometer performance with different configurations Comparison to see the effect with various raid and diskgroups: First tested with a small diskgroup & virtual disk/lun of 4 x drives using raid 5. Then I also tested on a 2 drive raid 1 lun on the iscsi SAN. After that I tested with a 14 drive disk group (the virtual disk does not fill all space then because of the VMware 2 tb limit). The 14 drive disk group I tested with raid 5 and raid 10. 1mb block on 4 spindles r5 iscsi san: SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 15.18 3954 123 17% RealLife-60%Rand-65%Read 34.52 1141 8 47% Max Throughput-50%Read 12.45 4798 149 17% Random-8k-70%Read 41.05 1114 8 38% Raid 1 SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 2 disks using Raid 1 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 15.17 3958 123 17% RealLife-60%Rand-65%Read 52.25 902 7 34% Max Throughput-50%Read 12.40 4803 150 17% Random-8k-70%Read 59.63 919 7 23% Raid 10 – 14 disks SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 14 disks using Raid 10 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 17.14 3520 110 16% RealLife-60%Rand-65%Read 15.45 3696 28 18% Max Throughput-50%Read 14.29 4144 129 17% Random-8k-70%Read 13.66 3936 30 24% Raid 5 – 14 disks SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 14 disks using Raid 5 (database i/o type 128k segment) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 17.06 3535 110 16% RealLife-60%Rand-65%Read 19.49 2542 19 29% Max Throughput-50%Read 14.38 4112 128 17% Random-8k-70%Read 16.16 2754 21 38% Comment: As expected raid 10 gives better performance at the cost of less space. Random i/o sees a big improvement with more disks added to the raid. Note: I also tested with different block sizes and with and without storage i/o control, but these settings did not seem to have much of an impact on performance. Storage I/O control should only kick when doing multiple VM I/O loads so that is as expected I think. I will use a default block size of 8MB on all my datastores. Note: All tests are done without using jumbo frames on the iscsi traffic Physical test results for comparison SERVER TYPE: Physical Dell Prec T5400 - Win 7 Enterprise x64 CPU TYPE / NUMBER: 1 x Intel Xeon E5440 HOST TYPE: Phyiscal Dell Prec T5400 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Crucial RealSSD C300 2.5" 128gb sata Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 7.15 8243 257 11% RealLife-60%Rand-65%Read 6.68 8629 67 9% Max Throughput-50%Read 11.13 5067 158 11% Random-8k-70%Read 5.45 10427 81 30% SERVER TYPE: Physical Dell Latitude E6400 - Windows 7 Enterprise x64 CPU TYPE / NUMBER: 1 x 2.53 Ghz intel core duo 2 HOST TYPE: Physical Dell Latitude E6400 - Windows 7 Enterprise x64 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Intel 80gb SSD gen2 M Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 9.31 6402 200 35% RealLife-60%Rand-65%Read 16.26 3305 25 56% Max Throughput-50%Read 62.80 903 28 21% Random-8k-70%Read 10.58 4996 39 37% SERVER TYPE: Physical Dell PowerEdge 2950 - W2K8 R2 x64 CPU TYPE / NUMBER: 1 x 2.53 Ghz intel core duo 2 HOST TYPE: Physical Dell PowerEdge 2950 - W2K8 R2 x64 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc 5/i - 4 x 300 gb sas 15k - raid 5 Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 3.64 17175 536 5% RealLife-60%Rand-65%Read 37.42 1197 9 3% Max Throughput-50%Read 4.91 12721 397 3% Random-8k-70%Read 40.15 1161 9 1% Comment: As expected a server with raid of many disks is faster than single SSD on sequential throughput but slower on random I/O. Note: These are physical tests and since virtualization has some overhead they are usually faster than the virtualized servers (iometer01-03) in similar configurations. Also note that I have tested some mainstream SSD disks here which is not usually used in servers (server ssd costs a lot more), however it is still an interesting comparison when for example a developer has to choose between running a virtual machine in vmware workstation on an SSD laptop/workstation or use a shared VMware LabManager server with SAN storage. The PE2950 server tested is a generation 9 Dell server and is much older than the generation 11 R710 servers, but that is one of the advantages of virtualization that you can buy new servers each year and move virtual servers to new hosts to upgrade the speed (then over time virtualization might actually be faster than the old model of buying a dedicated server for a solution and running it for 4 years).