VMware Communities: Virtualization Performance testing for ...

VMware

Up to Discussions in Virtualization Technology & Industry Archives

This Question is Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)

10 Replies Last post: Mar 28, 2006 6:37 AM by kix1979

Virtualization Performance testing for VMware ESX/GSX/Server & MS VS posted: Feb 23, 2006 7:36 AM

kix1979

3,769 posts since
Oct 14, 2004

There has been a lot of talk over the past few months/years about the performance differences between ESX & VS, but most of this testing is done by 3rd parties. My thoughts are as follows...

I would like to create a VM or series of VMs that can be used in a standardized way that is 100% repeatable to test performance of each server AS WELL AS test the hardware. We have all had lots of talks about how a DL585 is 10x better then a DL580 or an IBM x366. Forgetting those arguments I would like to make these VMs so they can be used anywhere pretty much for testing. While the Windows/SQL licensing prevents this I could create repeatable instructions to build the VMs in a specific way to ensure that the tests can be verified.

Would there be interest in this from others out there to test your own hardware/OS as well as potentially test new hardware for considerations in your environment? If so what tools/tests would you recommend for testing? I am thinking of doing a variety of MS & Linux boxes that would run network, memory, cpu and disk testing that could effectively max the box out. If 5 is not enough then create 2x or 4x or whatever number of all the servers to max the box. Then record how long it takes for each of the tasks.

VM1:
Windows 2000

VM2:
Windows 2003
SQL Server

VM3:
Windows 2003 SP1

VM4:
Linux flavor

VM5:
Linux flavor
mySQL

Thoughts?

Kix

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

1. Feb 23, 2006 10:51 AM

in response to: kix1979

dpomeroy

3,909 posts since
Apr 12, 2004

I think I put this in the feature request forum, but yes, I would like to see some standardized Test/Troubleshooting/Performance VMs in the VMTN community VMs download area. I would like to see a web test and database test VM for starters along with basic CPU/Network/Disk/RAM load testing.
I think it could be possible with Windows is the VM was syspreped with the switch that forces you to enter your own license key. SQL could be installed after or mySQL could be installed.

I would certainly use these if someone created them.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

2. Feb 23, 2006 7:29 PM

in response to: kix1979

Ken.Cline

5,150 posts since
Jul 7, 2004

You'll have to be careful...go read your VMware EULA carefully. There is a clause in there about not publishing performance testing results without express written permission from VMware.

It would be fine to develop and share a testing methodology for people to use internally within their organization, but - until VMware relaxes the EULA, it would be a violation to share the results (or at least that's how I read the EULA).

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

3. Feb 23, 2006 7:38 PM

in response to: Ken.Cline

sbeaver

7,723 posts since
Nov 1, 2004

But it would be cool if everyone could test with the same base. I think that is a very good idea.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

4. Feb 24, 2006 5:21 AM

in response to: Ken.Cline

kix1979

3,769 posts since
Oct 14, 2004

That makes sense from a publishing standpoint, but you are right I am more interested in developing a methodology and a specific toolset that anyone can use to test various hardware/software platforms with VMWare vs. MS.

With that said, I would think that VMWare would be very interested to see the results of these tests in a variety of environments, if anything it could lead to some good PR considering the number of us that could reproduce the tests around the globe. My main issue is coming up with the specific tools to use. Getting hardware is not a big issue for me, it's developing exactly what people want to see from a testing standpoint that is complete.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

5. Feb 28, 2006 4:57 AM

in response to: kix1979

AMcCreath

1,071 posts since
Jan 7, 2005

I would be more than interested, but would be keen to see firm baselines set up so they are repeatable in all environments.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

6. Mar 21, 2006 4:03 PM

in response to: kix1979

kix1979

3,769 posts since
Oct 14, 2004

Looks like my official testing will begin in a week or so. First up a x366 vs. DL580. I plan to create a SQL VM and do a mass table create/destroy, aka 1million rows etc..., do some massive file copies. Any other tests you think would be good?

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

7. Mar 22, 2006 1:36 AM

in response to: kix1979

king@it.ibm.c…

2,946 posts since
Jan 16, 2004

I see a couple of issues with this:

1) most of the quick&dirty tests (such as file copies etc etc) that are easy to run are not usually very representative of a real life setup. Most of the complex setups that are more representative of real life scenarios are usually very expensive.

2) Repeatability is a concern. There might be subdle things in every single setup (i.e. network speed not set properly, SAN workload that is heavier than usual at the time of tests etc etc etc). This might lead to misleading results that have nothing to do with how good/bad a server is.

I agree we need a standard benchmark environment for virtual environment but doing this in an "anarchic" manner might be dangerous.

Also, as other have posted, there are legal implications in sharing those results if not properly audited by the parties.

Massimo.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

8. Mar 27, 2006 1:43 PM

in response to: king@it.ibm.c…

rMh

20 posts since
Dec 4, 2005

A good way to do this would be to create and distribute a virtual machine using a linux flavour that had all the tools you could use to do the performance testing. Just getting the tools out there to use would be next to useless, because everyone would be running everything slightly differently. E.g. one person could be running with Windows performance settings tweaked while another has it all stock etc.. Obviously distributing a VM with Windows installed would be a "no no".

I am very interested in doing some vm performance testing though. even if it's just internal to show the difference between our SATA and FC Luns and difference in performance of having more smaller Luns compared to larger ones.

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

9. Mar 27, 2006 11:17 PM

in response to: kix1979

AMcCreath

1,071 posts since
Jan 7, 2005

Personally I'd outsource the task to a third party audit company ....

Re: Virtualization Performance testing for VMware ESX/GSX/Server & MS VS

10. Mar 28, 2006 6:37 AM

in response to: king@it.ibm.c…

kix1979

3,769 posts since
Oct 14, 2004

Also, as other have posted, there are legal
implications in sharing those results if not properly
audited by the parties.

I agree if I go spouting off benchmark scores I could get in some hot water, but I see no issues with posting that I ran a bunch of tests and found that X server is better then Y. There is no difference in that to me then the fact that people say the 585's are better then the 580 or a x345. It's one thing to publish a paper on something as fact, but entirely different if it is something for discussion.

As for the giving this to a third party to do, I would love to but that costs money and I still have no way to verify on my own. Believe me I love external consultants and their skills, like myself, but I want to do some testing to get better at it myself. Additionally the fact that there is no standard set of tools or testing methodology for VM environments is frustrating. VMWare has gone from lab to enterprise data center in a short time, but we need tools or standards we can refer to in making informed decisions about hardware.

Kix

Go to original post

Search VMware Communities Advanced Search

Actions

View as PDF

Up to Discussions in Virtualization Technology & Industry Archives