Consistent Results for Application Performance Testing using VM’s

[Mark_Jones]

Guys,

One of the more frequent arguments I have with Developers is how they get inconsistent test results when using Virtual Machines for doing application performance testing.

Not being a developer myself I thought I would ask the community for their feedback and experience in this area.

I have read many performance articles that relate to the overall performance of VM's and overheads of virtualisation, but not much about the practical application of application testing.

For completeness lets discuss having 3 VM's on the same ESX host.

My view is that using Resource Pools, Resource Shares and CPU Affinity would guarantee that I have a consistent configuration that can be used testing.

What (in your opinion) do I also need to consider?

I am interested in the guaranteed, consistent performance of each VM, not the contention or overheads of virtualisation.

I would be happy to put the results in a whitepaper if it gets to that.

[davidbarclay]

What (in your opinion) do I also need to consider?

Things resource pools can't control. In my opinion, the biggest influence on performance you can't control with settings in VC is disk I/O. Typically test/dev (at least on projects I am involved with) ends up with higher consolidation ratios, therefore more chance of contention for resources therefore your issue of unable to provide the exact performance each time.

Just a thought...I'm no testing expert...

Dave

[Gabrie1]

I to find this very difficult to guarantee. Problem is that your regular tools to measure performance, don't always give correct results. The only reliable performance measurement is by timing transactions. How long will it take after the user press GO, to complete the query. That sort of things.

When VMware was working on VMMark, I expected this would be a tool I could use to measure performance of single VMs. But unfortunately you have to buy serverall other tools to be able to test and also you still have nothing to compare for your specific app.

So we use only one indicator, does user have happy face or not.

Gabrie

[oreeh]

My view is that using Resource Pools, Resource Shares and CPU Affinity would guarantee that I have a consistent configuration that can be used testing.

This is wrong.

Resource pools and shares don't help in guaranteeing resources.

Shares for example ONLY come to play when there's contention.

The only way to ensure that a VM always has a defined set of resources (CPU, memory) available is to use reservations.

Check this thread and Ken Cline's VMworld presentation #IO20 (not yet available to the public).

[Mark_Jones]

Thanks Dave, yes the common issues that are raised with me are always around contention or resources.

Q. Is the resource contention issue unique for Virtual Machines?

Q. If no, then why is this overlooked or not factored into tests?

I see very similar issues in both environments.

Disk I\O is always an interesting issue to address, but this is not just a Virtual Machine issue, unless you are running VM's on DAS (Direct Attached Storage).

What can be done is a SAN environment? (Lots)

Is the contention in a SAN environment the same for both Physical and Virtual?

Is using dedicated resources going to influence better results? (For instance A virtual machine with a dedicated NIC, Dedicated HBA, Dedicated CPU? )

Would the results be better? Or is this just another level of complexity to support\offer?

Garbie...

Absolutely agree... The only consistent measuring that I have found is transaction based (transactions\sec) or similar.

We employ a long list of test tools, but often our testers look to blame virtual environments, perhaps because they don't understand them?

Oreeh...

Thanks, I think.

Reservations would address giving a machine a defined set of resources, but aren't there other components that would influence a VM's performance (NIC, HBA, Network etc).

Are these influences worth considering?

Are they unique to VM's?

I am really trying to get feedback due to so many questions that are raised by Developers\Testers.

I don't just want to approach this from a numbers point of view, but how do you address their concerns and educate them about any limitations\considerations that they will need to understand when testing in a virtual environment.

I am trying to avoid our current issue, whereby the almost demand a physical environment due to their lack of experience with testing using VM's.

Perhaps this will be an education for me along the way.

[oreeh]

but aren't there other components that would influence a VM's performance (NIC, HBA, Network etc).

Of course they do.

But to be comparable we just have to assume that physical and virtual systems run on the same hardware.

Are these influences worth considering?

Are they unique to VM's?

Yes. A bad (from a performance point of view) disk subsystem has an impact on overall performance.

But it isn't unique to VMs.

With VMs of course this "sums up" since multiple VMs need to use the same physical resources.

Another interesting thread regarding bebchmarking and ESX

[Mark_Jones]

Oreeh,

Perhaps there are several different performance questions that are being asked.

Can Virtual Machines be configured to a consistent baseline to ensure that repetitive testing would yield useful results that would assist in application load testing.

If I have a physical application server and a virtual application server how do they perform and how can they be measured. (I am less interested in this answer)

The outcome that I am looking for is to be able to pass on knowledge to our testers\developers that would help them construct a metric that will assist them in their load testing.

If there results are that they would need to dedicate certain resources to each test VM, then so be it.

If I build 2 VM's, purchase 2x Single NIC, 2x Single HBA's and use reservations as you suggest. Does this provide a valid baseline for testing? (Ignoring SAN and Network congestion\contention as this would be the same in P or V)

[oreeh]

Can Virtual Machines be configured to a consistent baseline to ensure that repetitive testing would yield useful results that would assist in application load testing

IMHO yes.

But this requires you to only run the tested VM on ESX (maybe with different limits / reservations).

As soon as other VMs are running on the same host your results are not easy comparable since you measure the ESX "overhead" (scheduling of VMs, additional CPU task switching,...) as well.

If I build 2 VM's, purchase 2x Single NIC, 2x Single HBA's and use reservations as you suggest. Does this provide a valid baseline for testing?

IMHO no (see above).

However this could be a valid test if these VMs are dependent or you want to include the mentioned overheads in your results.

In this case you have to use limits and reservations to get comparable results

.

In any case I suggest to run the same test a few times and then calculate the mean values.

[oreeh]

FYI: this thread has been moved to the Performance Forum.

Oliver Reeh[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]