VMware Cloud Community
cscarff
Contributor
Contributor

ESXi 7.0 out-performs physically installed RHEL 7.8 on HP Z840 Workstation. Why?

I realized there's lots of possible variables to account for my results, but was wondering in a short port.

I have been testing a very CPU-intensive corporate application.  This app can run on a single server (node) or in a cluster.  In the past, I have processed imagery using more than 800 physical cores using several nodes.  Recently, I've been gathering performance data comparing a single large VM in Hyper-V, ESXi 7, and running the app in a physically installed RHEL 7.8 configuration.

The HP Z840 (approx. year 2017) has dual Intel CPUs (2x14 cores) and Hyperthreading is enabled in BIOS.  System has 256GB of RAM and all solid state disks.  

I'm wondering why the system is processing approximately 4% to 5% faster in ESXi compared to processing in RHEL 7.8, which again is installed physically on the box, when booted into that environment.  I'm using multiple boot configurations to boot into different OSes.

What could account for the improved performance in ESXi 7.0 compared to RHEL 7.8 (minimal install)? This is unexpected.  All CPU threads are close to 80% utilization when processing.

Chris
0 Kudos
4 Replies
vbondzio
VMware Employee
VMware Employee

There could be a bunch of reasons, did you 100% narrow it down to CPU i.e. there is no IO in the critical path?

Then possibly:

  • EVC enabled or an older vHW that could mask instruction sets the application is using but for some reason less optimally (or some we masked for security reasons)
  • The power management policy of RHEL might not be optimal for your workload
  • Configured a non-accurate cache topology which for some reason is helping the workload
0 Kudos
cscarff
Contributor
Contributor

Thanks for the response.  Both test environments were as similar as I could make them.  The data always came from the same NAS on a clean gigabit network; nothing else running on home network at time of tests.

I will rerun both tests and ensure the drives are the exact same and connected in the same manner.   Though, I did run ESXi on an external USB 3.0 SATA connect adapter to an SSD.  RHEL was on an internal SATA to the Mobo.  I would have thought the ESXi test still would have been slower simply due to that fact.

RHEL was a simple Minimal install with all out-of-the-box settings.  Nothing done to kernel in way of optimizations.  Both physical aand VM-based RHELs were identical in regards to installation/setup.

 

 

Chris
Tags (1)
0 Kudos
scott28tt
VMware Employee
VMware Employee

Sure I’ve heard of other instances of “faster than physical” with ESXi in the past, at least partly due to the efficiency of the CPU scheduler compared to the native OS.


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
0 Kudos
vbondzio
VMware Employee
VMware Employee

95% of the "faster than bare metal" is down to better NUMA locality of mulitple VMs, i.e. it's somewhat easier to have 100% local memory access when the whole VM fits into a NUMA node instead of the (guest OS) having to balance / bin package across nodes. ESXi also showed some benefit in over committed scenarios thought, i.e. the balancing was partially more efficient too.

That shouldn't have mattered much for a wide VM if the presentation was the same (sans SMT of course). Besides the points I mentioned, if there was IO then it could be different driver defaults or even a difference in expression (latency) due to lack of usage.

0 Kudos