Perf Difference between ESXi & WS (was Re: Virtual Desktop Benchmarking)

[docsmooth]

I do rough testing of my company's software on ESXi and Workstation. Since we're an authentication software, this mostly involves scripted SSH logins. In testing to the exact same Win2k3/Win2k8 (Virtual, 1GB RAM each) domain controllers, my 32-bit laptop consistently outperforms my 64-bit VM by 10 times the transactions / minute. I'm curious to dig deeper into into this odd bit of data, to see what specifically affects the problem (guests running on the host, memory allocated to guest, cores for guest, etc.). However, as someone not trained in this field I'm not sure where to begin. Anything I can provide to this discussion, especially in terms of true data, I'd love to give.

Message was edited by: Ken.Cline - branched into its own thread since it's really a different question

[jasonboche]

I do rough testing of my company's software on ESXi and Workstation. Since we're an authentication software, this mostly involves scripted SSH logins. In testing to the exact same Win2k3/Win2k8 (Virtual, 1GB RAM each) domain controllers, my 32-bit laptop consistently outperforms my 64-bit VM by 10 times the transactions / minute. I'm curious to dig deeper into into this odd bit of data, to see what specifically affects the problem (guests running on the host, memory allocated to guest, cores for guest, etc.). However, as someone not trained in this field I'm not sure where to begin. Anything I can provide to this discussion, especially in terms of true data, I'd love to give.

I'm not sure if I'm reading you correctly but if your testing environment is ESXi on Workstation as in a hypervisor on top of a hypervisor, then the performance degradation you speak of does not surprise me. Nested hypervisors are for demonstration purposes only.

If the above is not true and you are using either ESXi or Workstation in a mutually exclusive scenario, then my answer would be 9 times out of 10 VMware is not about increasing workload performance through virtualization, but rather maximizing infrastructure utilization while maintaining acceptable environment/application performance. That's a fancy way of saying there is some overhead to virtualization and you'll usually see a slight performance decrease of a VM compared to bare metal, but much of that outcome is significantly dependent on the configuration of the VM and the infrastructure. Your 1/10th performance isn't within normal tolerances and is not acceptable. I'd begin narrowing down the characteristics of the application and looking for bottlenecks in CPU, disk, network, and memory. Off the cuff I'd also investigate context switches/second using conventional performance monitoring tools (ie. perfmon for Windows). I'd hazard to guess around 5,000 context switches/second is going to show noticable delays and in the 15,000+ realm it's really going to be noticable.

Jas

[i]Jason Boche[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[drummonds]

One of our mantras in the performance space is that if we ever seen a performance difference between native and virtual of 10x, usually a mistake was made in the methodology. We say this because we've now done analysis on so many different applications under so many different circumstances, we have an excellent understanding of the worst case scenario. And I can tell you that the worst case scenario for 99% of our customers is when a workload runs at about 65% of native. But the vast majority of apps far above this.

Given the 10x difference that you've seen and the small amount of description you've provided on the workoad, I'd guess that the virtual network driver (vmxnet) isn't properly running in the guest. This would mean that you either didn't install VMware Tools or something went wrong. The paravirtualized network driver can provide an order-of-magnitude performance increase which could explain your problem.

You also mentioned that you do testing on both ESXi and Workstation, but didn't say which one under-performs your laptop. Perhaps you meant that the Workstation-based VM on your notebook outperforms the ESXi VM on a 64-bit server? Given that both of these products use the exact same monitor but one (ESXi) has a much better scheduler, it's possible that the server hardware is slower (ignoring any other simple errors in configuration like a lack of VMware Tools.) An interesting test to level set performance expectations would be to run a simple CPU-bound benchmark like Java SciMark to evaluate each VM's performance. That benchmark runs in user space which means the code is passed through the hypervisor to the hardware. It should run about 95% of native.

Let us know what you find out.

Scott

[Ken_Cline]

docsmooth -

First, as others have said - a 10 fold performance difference between the two platforms is unacceptable and likely due to a configuration issue. To help us help you, please provide more details about your configuration. Let us know what your hardware platforms look like (for both WS & ESXi) - include CPU (# cores, speed type), RAM (including RAM speed), Disk (number, size, speed, availability) & Network (type, speed, etc.), the type of network connectivity between your laptop (testing source) and the VM (testing target).

One thing to consider is that, since you are doing highly redundant, small-scale testing, it is quite likely that you are never touching the disk on the WS system. Workstation runs as an application on the host OS (either Windows or Linux) and benefits from the disk caching performed by the host. ESXi does no caching, so you will see performance closer to native. In other words, for some specific workloads, a VM running inside a hosted virtualization platform can actually out perform a native system. I suspect this is what you are seeing. Monitor the disk I/O metrics at three points to verify:

- 1. On your Workstation host system

- 2. On the Workstation VM

- 3. On the ESXi VM

I suspect that you'll see the most IOPS on the Workstation VM (it's getting the most work done), followed by the ESXi VM, and finally, the Workstation host. If my suspicions are correct, there will likely be orders of magnitude differences between the three.

Ken Cline

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator