Help
armaniseal
Contributor
Contributor
‎04-07-2011 07:43 PM

VMware iSCSI TOE using hardware HBA's Broadcom NICs with MD3200i - Performance Problem with 64KB Reads

When i first heard that with vSphere 4 the Broadcom TOE NICs were available for hardware iSCSI configuration i thought great. After initial trial i first learned that you can not use Jumbo frames and HBAs together. If your used to using Jumbo Frames with your VMware software iSCSI initiator that will not work with Boradcom HBAs as the MTU on them is 1500 as can be seen in the service console using esxcli and can not be changed (not that i know of anyways).  This however is not a problem as performance was still far batter with HBAs as per VMware iSCSI SAN Configuration Guide.  This I found interesting having results that i have seen.

After setting up a cluster i run SQLIO ( and then IOMeter to confirm results) to find out something so strange that we thought had to be a problem with the SAN! All tests showed great trhoughput except one! 64KB Sequential Reads showed 1.3MB/sec with 1895ms latency!!  Mind you this is a test suppose to show BEST results being not only Reads and large block size of 64KB but also Sequential!  Other same build show 128MB/sec with 3ms latency on same test! Whats even more crazy is that 64KB writes Sequential were doing normal (100MB/sec) and also 64KB RANDOM reads were doing normal.

Out of all the tests ONLY 64KB Sequential Reads showed this result. I tried everything including swaping hosts, VMs, multiple tests, and the results remained. because the same size and frame tests (64K write) worked fine we assumed it had to be the SAN and nothing to do with transport as it would affect the writes! Even esxtop showed the latency to be in the DEV queue (also in kernel but nowhere as much as DEV which is normal once DEV is very high).

WRONG! only after we replaced the Controllers and even the backplane we realized its the TOE! As soon as i switched back to the iSCSI software everything was fine.  I realized this after i had the 64 Read test running and ran another from different VM which was reporting 1000s IOPs with 16K writes while supposedly the latency was 1893ms on the other VM.  I realized that if SAN was overloaded that test wouldn't be showing off the charts performance! To prove it more i vMotioned the VM running the 64KB Read test over to the host running the secondary test to and as soon as i did the secondary test started dropping. Here i realized its in the transport!!! Why would it affect it from same host but not another if on same LUN!?  here is the full test with problem test cut and pasted at very top:

I will be opening up a ticket with VMware tomorrow since its clearly on this end. Shame we wasted 2 days replacing the SAN parts thinking its that although i have to say all evidence pointed to it!

6 threads reading for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:    25.90

MBs/sec:     1.61

latency metrics:

Min_Latency(ms): 7

Avg_Latency(ms): 1848

Max_Latency(ms): 2434

--------------- FULL TEST BELOW----------------------

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB random IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  3616.12

MBs/sec:    28.25

latency metrics:

Min_Latency(ms): 1

Avg_Latency(ms): 12

Max_Latency(ms): 557

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  1  2  1  0  0  3 11 22 24 18  9  4  1  1  0  0  0  0  0  0

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1768.47

MBs/sec:   110.52

latency metrics:

Min_Latency(ms): 7

Avg_Latency(ms): 26

Max_Latency(ms): 101

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB random IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  4297.82

MBs/sec:    33.57

latency metrics:

Min_Latency(ms): 5

Avg_Latency(ms): 44

Max_Latency(ms): 156

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  1  2 96

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1771.39

MBs/sec:   110.71

latency metrics:

Min_Latency(ms): 12

Avg_Latency(ms): 107

Max_Latency(ms): 262

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB random IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  4292.32

MBs/sec:    33.53

latency metrics:

Min_Latency(ms): 3

Avg_Latency(ms): 55

Max_Latency(ms): 117

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1728.11

MBs/sec:   108.00

latency metrics:

Min_Latency(ms): 9

Avg_Latency(ms): 138

Max_Latency(ms): 2416

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  7094.55

MBs/sec:    55.42

latency metrics:

Min_Latency(ms): 1

Avg_Latency(ms): 6

Max_Latency(ms): 23

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  2 18 42 31  7  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1772.75

MBs/sec:   110.79

latency metrics:

Min_Latency(ms): 5

Avg_Latency(ms): 26

Max_Latency(ms): 46

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  7042.03

MBs/sec:    55.01

latency metrics:

Min_Latency(ms): 5

Avg_Latency(ms): 26

Max_Latency(ms): 42

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  3 95

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1773.47

MBs/sec:   110.84

latency metrics:

Min_Latency(ms): 15

Avg_Latency(ms): 107

Max_Latency(ms): 179

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  7077.37

MBs/sec:    55.29

latency metrics:

Min_Latency(ms): 4

Avg_Latency(ms): 33

Max_Latency(ms): 50

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  1772.35

MBs/sec:   110.77

latency metrics:

Min_Latency(ms): 10

Avg_Latency(ms): 134

Max_Latency(ms): 201

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB random IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  2497.36

MBs/sec:    19.51

latency metrics:

Min_Latency(ms): 3

Avg_Latency(ms): 18

Max_Latency(ms): 90

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  1  1  3  5  6  8  9  9  9  8  7  5  5  4  3 17

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   999.95

MBs/sec:    62.49

latency metrics:

Min_Latency(ms): 10

Avg_Latency(ms): 47

Max_Latency(ms): 1114

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   492.86

MBs/sec:    30.80

latency metrics:

Min_Latency(ms): 12

Avg_Latency(ms): 388

Max_Latency(ms): 3517

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB random IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  2534.43

MBs/sec:    19.80

latency metrics:

Min_Latency(ms): 4

Avg_Latency(ms): 93

Max_Latency(ms): 161

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB random IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   769.28

MBs/sec:    48.08

latency metrics:

Min_Latency(ms): 19

Avg_Latency(ms): 311

Max_Latency(ms): 1964

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  8633.18

MBs/sec:    67.44

latency metrics:

Min_Latency(ms): 1

Avg_Latency(ms): 5

Max_Latency(ms): 26

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  1 20 52 26  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 8 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:    25.90

MBs/sec:     1.61

latency metrics:

Min_Latency(ms): 7

Avg_Latency(ms): 1848

Max_Latency(ms): 2434

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  8671.03

MBs/sec:    67.74

latency metrics:

Min_Latency(ms): 4

Avg_Latency(ms): 21

Max_Latency(ms): 31

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  4  9 26 39 16  4

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 32 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:    26.22

MBs/sec:     1.63

latency metrics:

Min_Latency(ms): 9

Avg_Latency(ms): 7235

Max_Latency(ms): 8570

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0 99

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 8KB sequential IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:  8641.11

MBs/sec:    67.50

latency metrics:

Min_Latency(ms): 2

Avg_Latency(ms): 27

Max_Latency(ms): 43

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1 98

sqlio v1.5.SG

using system counter for latency timings, 3579545 counts per second

parameter file used: param.txt

file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)

file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)

file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)

6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat

using 64KB sequential IOs

enabling multiple I/Os per thread with 40 outstanding

using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat

using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:    26.29

MBs/sec:     1.64

latency metrics:

Min_Latency(ms): 13

Avg_Latency(ms): 8949

Max_Latency(ms): 13804

histogram:

ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

0 Kudos
Reply
2 Replies
msatya
Enthusiast
Enthusiast
‎11-23-2011 06:25 AM

Hi please letus know still you are facing same problem .

-------------------------
please award points if you see fit.Thankshttp://imagicon.info/cat/5-59/vbsmile.png

0 Kudos
Reply
armaniseal
Contributor
Contributor
‎11-23-2011 10:39 AM

since the post is from 2009 i hope not! Smiley Happy

We had to move to SW iSCSI which resolved the issue. It seems to be only affecting when using hardware iSCSI NICs. I know Dell has actaully a better firmware for MD for better performance algorithm which is just sad if you ask me that you buy a unit an intensionaly they constrain your performnace so you spend more to buy better firmware! However i don't suspect this is cause of that given the magnitude.

Thanks

0 Kudos
Reply