Setting the Record Straight on the Hyper-V Video

bherndon — Mon, 08 Jun 2009 22:19:50 GMT

Let me first introduce myself. I am Bruce Herndon and I manage one of the benchmarking teams at VMware. Our responsibilities include both VMmark and SPECvirt. Let me also state for the record that I am not exactly pleased to be writing on this particular subject in a public venue, but I really have no choice at this point since my team was responsible for the raw footage used in the now infamous "Hyper-V crashes" video posted on YouTube by my colleague in marketing, Scott Drummonds.

I had hoped that this whole kerfuffle would quickly die down, but it shows little sign of abating. I have been convinced to post some details of this work for a couple of reasons. First, many people have rightly objected to the lack of details given in the video. I am the best person to address those concerns. The other reason is to stick up for the fine engineers in my team. I simply can not abide the fact that the legitimate questioning of the underlying configuration has evolved into a series of rather strident attacks on the engineering acumen of my team. As you will soon see, they are top-notch.

Before I present data, let me provide a bit of background. As a matter of course, my team does extensive testing and analysis of competing products to help understand VMware's position relative to other solutions. I think most would agree that this is simply a good business tactic. Although the results help us sleep well at night, we typically don’t publicize them for a variety of reasons. For example, no matter how well the work is done, its credibility will always be suspect to many readers due to the fact it originated with VMware. I certainly understand that. (My preference is to highlight projects that showcase VMware’s strengths, such as the excellent DPM video that my team and Scott produced last year.) We were recently asked if we had any interesting material for an internal demo. Since we had been encountering Hyper-V crashes, we produced raw footage of Hyper-V crashing as a response. Someone along the line passed the footage to Scott, who mistakenly believed it was for external release and produced the finished version you see on the web.

There were a number of requests for more details on the system shown in the video. Below is a basic summary of the hardware used:

Dell PowerEdge R905 with 4 x 2.6GHz Quad Core AMD Opteron 8382
Firmware version 3.0.2 (latest available).
128GB DDR2 Memory.
2 x Intel E1000 dual-port NIC
2 x Qlogic 2462 dual-port 4Gb
2 x EMC CX3-80 Storage Arrays.

I want to point out that this particular testbed has performed flawlessly under much heavier loads and VM consolidation ratios during an array of testing running similar workloads on ESX. I am confident that the hardware is fully functional.

The hypervisor was the fully-patched RTM version of Hyper-V.

The benchmark was based upon VMmark. VMmark’s methodology is the most well-known and battle-tested benchmarking methodology in the virtualization industry. Many of its ideas have subsequently been adopted by SPEC’s virtualization subcommittee in the development of SPECvirt. Due to the lack of SMP support for Linux VMs in Hyper-V, we had to simplify the test by making all Linux VMs single-vCPU. We also used Windows Server 2008 instead of Server 2003 where possible based upon various Microsoft claims that Server 2008 was the best choice for running Windows workloads on Hyper-V. As a result, we increased the memory in the Javaserver from 1GB to 1.4 GB to insure sufficient memory space for the JVM while accounting for the added resource requirements of a default Server 2008 installation. Finally, we created a second virtual SCSI drive for data in addition to the IDE boot drive for added performance based upon Microsoft’s recommendation that it might be efficacious – more on that later. All appropriate integration components were installed on all VMs. The table below provides a summary of each VM's configuration.

Workload	#vCPUs	Memory	Disk	OS
Mailserver	2	1GB	24GB	Windows 2003 32bit
Javaserver	2	1.4GB	12GB	Windows 2008 64bit
Standby Server	1	256MB	12GB	Windows 2008 32bit
Webserver	1	512MB	8GB	SLES 10 SP2 64bit
Database	1	2GB	10GB	SLES 10 SP2 64bit
Fileserver	1	256MB	8GB	SLES 10 SP2 32bit

For IDE/SCSI tests, a second SCSI drive of similar size was created for data storage.

Note (6/8/09): One of my colleagues pointed out an error in the disk usage for some VMs. I have corrected it.

The crashes shown in the video occurred when running a total of 11 tiles (66 VMs). We saw the full spectrum of crashes: random subsets of VMs would crash, or more disturbingly the parent partition would BSOD, or most disturbingly the hypervisor itself would BSOD. In this blog posting, Jeff Woolsey of Microsoft dredges up the familiar complaints about VMware’s ESX Benchmark EULA. I have to point out the obvious here - there were no benchmark results in the video. It is quite difficult to gather data when the system reliably crashes during the test. However, I am happy to provide more data to help improve the signal-to-noise ratio in this discussion. I also believe that more data will put the crashes shown on the video into the proper context.

The figure below shows throughput scaling relative to the throughput of a single tile. (Under ideal circumstances 2 tiles would achieve a scaling of 2, 3 tiles a scaling of 3, etc.) The overall host CPU utilization is shown by the yellow line on the graph. I won’t comment too much other than to say that in my experience with this type of workload, the scaling exhibited by Hyper-V is lower than expected. The drop in throughput for the tenth tile was also surprising.

The system crashes shown in Scott’s video occurred when we attempted to run an eleventh tile on the system. System stability under these conditions is the hallmark of true enterprise-class software. While things like poor response time are to be expected in this regime, catastrophic failure is never acceptable, including in the event of a sudden unexpected spike in usage.

I also want to stress that the results above are as good as we could achieve. Remember that this was intended as an internal-only study. For these types of exercises, we try to create worst-case scenarios for VMware products to find where improvements are needed. This means we have to give the competing products every possible advantage. We do significant research and test several different configurations to accomplish this. Ultimately, we want to insure that VMware’s out-of-the-box performance beats everyone else’s well-tuned configurations. This is why I was surprised to see the following statement in Mr. Woolsey’s blog:

"If you want to run a virtual machine with a single virtual disk just do it. My best guess is this: Hyper-V only boots from virtual IDE, not virtual SCSI, and whoever ran this test thinks that the test must be run from virtual SCSI for the best performance."

I think most folks would be interested to know that the in the course of our extensive research we tested just such a scenario. In this case the system was a Dell 2950 with dual Intel Xeon X5460 (3.16 GHz) and 32GB of RAM. As you can see below, IDE-only exhibited higher utilization (except at saturation as expected) and lower throughput at every data point. Based on these and other results, we standardized on the IDE/SCSI dual-disk solution to give Hyper-V every possible advantage.

As I mentioned earlier, we tend not to publicize these competitive results for reasons such as a perceived credibility gap. Another reason is that we see no point in helping our competitors by pointing out their flaws. We’d much rather they do the work themselves.

Those are the facts folks, so let me briefly recap. I agree with many of you that Scott’s original post was poorly done and we all would have been happier had it remained internal-only. Likewise, I agree that once the cat was out of the bag, more details needed to be provided. I hope this post has cleared things up. I also think the nastier attacks on my team were completely out of line. My team is experienced, methodical, and scrupulously maintains their systems. I stand behind their work 100%. In the mean time, we intend to focus on helping to build amazing rock-solid products that our competitors can’t yet imagine.

Bruce's Blog

Setting the Record Straight on the Hyper-V Video