anyone familiar with IOmeter?

[zgz87]

Hi all!

I am currently doing performance measurements in the storage through iSCSI and FCoE with IOmeter. I had the feeling that the measurements obtained were not as logical as I was expected so I used esxtop to compare. What I saw is that when the LUN size is below the cache, the results are the same in both tools however, when the size of the LUN is above the cache, I get very different results and I have no idea which ones are correct. Does anyone have any idea of what can be happening?

Apart from that, any help regarding how to properly use IOmeter, advantages, disadvantages, interesting links or even alternatives is really welcomed!

[DavidKlee]

I use IOMeter a LOT. When you do the testing in the way you described, the results are expected. If you're not piercing the SAN cache, most of it ends up in the cache and the performance is going to be much greater than if the test exceeds the cache and the SAN has to go to disk much more frequently.

IOMeter is a great tool. Just make sure to test ALL scenarios - basically, various combinations of read/write, sequential or random, number of worker threads, etc., so that you simulate your average workloads on the SAN. Check out the VMware document that describes more of these scenarios here: http://communities.vmware.com/docs/DOC-3961.

One of the tools that I also like to use is SQLIO. It is from Microsoft, and says that it simulates OLTP database traffic, but I use it in general to determine the performance curve of a SAN based on varying degrees of load. I wrote a free tool to analyze the output, and a test script that walks everyone through how I generate the results test file.

http://tools.davidklee.net/sqlio.aspx

Give it a try and let me know what you think. Between the two tools, I can simulate any sort of load and can quickly get some output that helps me tell the story of what the SAN really does!

[zgz87]

Hi David,

Thank you for your answer. I will try to find time to use the tool you suggest.

I understand that the results are worse when not all can be cached but I still cannot understand why the results are different between esxtop and IOmeter in that case. I also tried a EMC tool and the results say reported are completely different (specially between the EMC tool and IOmeter or between esxtop and IOmeter).

Apart from that, I read that IOmeter in Linux is not reliable, is that true?

Thanks

[DavidKlee]

How different were the SAN reports from the IOMeter reports from esxtop? Have you tried vscsistats in addition to esxtop? Was the traffic limited to just one HBA or NIC? Can you list some of the numbers here? I'm curious to seee how widely different the results are.

Also, I have not personally experienced any problems with IOMeter on Linux, but I think it does not report vCPUs correctly. I do know that it has not been updated in quite some time. Have you tried using VMware Labs' IOBlazer as an alternative to IOMeter on Linux?

[zgz87]

I tried an Analyzer tool from EMC as well which runs in the storage. The traffic right now is limited to one NIC (iSCSI, 1GbE) and for example for the traffic:  RealLife-8K--65%Read-60%random  the results we get are:

        EMC   esxtop     IOmeter                 

IOps  570          330             160       

MB/s  38            36              1.3                         

For this traffic SQLserv-64K-66%Read-100%random

        EMC   esxtop   IOmeter

IOps 525      280        120      

MB/s 39        38          7

These two same traffic types when the size of the disk is below the cache give the following:

traffic1:

IOps 4200      4200      4200

MB/s 33          32           32   

traffic 2:

IOps 780          780       740

MB/s 48             46         46

which is the same in all.

Any idea what could be my problem? Could be something related to ramp time? iSCSI buffer size in ESXi?

Thank you again!

[DavidKlee]

Thanks for posting the numbers. I'll be honest. I don't have a lot of faith in the IOMeter Real Life test you referenced. Check the number of worker threads, or concurrent operations. It's probably 32 or greater. The other tests (EMC) are probably using a single threaded test. Those IOMeter tests aways seem to report low.

Have you given a short run with SQLIO? I trust that level of results more than IOMeter for varying degrees of load.

esxtop will report on what's going on in the background, and it could report lower IOPs in the first test because of partition mis-alignment. Partition mis-alignment means that for every one block request that the VM or ESXi make, the SAN might have to read two or more blocks to fulfill that request. An offset (I usually do 1MB) is placed in front of the logical partition to fix this. Windows Server 2008 supposedly fixed it by default, but occasionally SAN vendors screw it up. Windows Server 2003 almost always required the offset. What OS are you on? Is the partition properly aligned? This could explain the differences between the IOPs numbers being different but the MB/s numbers being close.

I normally use IOMeter for my 100% read or 100% write (max throughput) tests. Watch your number of concurrent threads and operations. Introducing randomness into the mix seems to bury the storage and make it report numbers that are very low. They could be low for the test because of the number of concurrent threads.

Let me know what you think! I hope all of this helps!

[zgz87]

Hi David,

Thanks for your help again. First thing I wanted to clarify is that IOmeter is the only traffic generator we are using, esxtop and EMC tool just check the results. The EMC tool monitors the traffic in the port of our storage solution and reports that (only one port), the esxtop in the host and IOmeter in the vmware. Maybe you are right and there is some disalignment but we are using vCenter and it is claimed that when a database is created using vCenter no disalignmet is created. Is it any way to check this?

Our disks do not have file system, just the vmfs vmware file system. I read that this is a better way to test this, right?

We installed IOmeter in Windows and our test are using 16 outstanding IOs and two workers. Therefore 32 concurrent operations, but again, the EMC is only monitor this same traffic. Can you explain a bit more why the concurrent operations impact the results? Does this not make everything more realistic?

Today I wil try SQLIO! Thanks for the recommendation.

Thank you!

[DavidKlee]

The number of concurrent operations does more closely resemble real life, but keep in mind that in real life, if you have 32 worker threads, each and every one of them is probably not pounding the storage as hard as possible 100% of the time. IOMeter does push things very hard. This is part of the reason why I use IOMeter, but look at SQLIO for additional metrics. I use each one of these tools with different numbers of threads and operations per thread, and varying degrees of stress to get a good understanding of how storage performs under different degrees of load.