VMware Cloud Community
JLecht
Contributor
Contributor

Meltdown & Spectre Performance Impact on VSphere 6.5 (VCenter appliance & ESXi)

Hello,

Perhaps there are other threads on this already, but I've been browsing threads on this forum and find the topics covering "what needs to be done", or "present versions of the patch", but not much regarding the performance impact.

What are people seeing after applying the patches?

Thank you.

2 Replies
Psychomike70
Contributor
Contributor

We applied the VMSA 2018-0002 patches to all our 6.5 update 1 ESXi hosts and didn't see any noticeable performance impact.

That was until we updated our Win10 1607 virtual machines with the latest Microsoft Jan security patches (now running Win10 1607 build 14393.2007). After we recomposed our environment, to roll out these new virtual machines, our EMC SAN performance has been terrible. We have seen a marked increase in overall IOPS on the SAN. Read and Write response times anywhere from 12ms to 30ms per LUN/Storage group during main customer logon window (prior to the M$ updates, we seldom showed more than 7ms). However, on the Win10 virtual machines themselves, as well as on the ESXi hosts, we do not see an increase in CPU/Memory utilization. Steady state performance is close to "normal", but during logon or refresh windows, the SAN performance suffers now.

We have not applied any of the Intel microcode updates, nor as I mentioned above, any of the VMSA 2018-0004 patches.

Reply
0 Kudos
bluefirestorm
Champion
Champion

The performance impact of Meltdown will be felt if the CPU does not support PCID feature and INVPCID instruction. The PCID is the Process Context Identifier in the Translation Lookaside Buffer (TLB). Without a PCID field in the TLB, the TLB needs to be flushed constantly for every process context switch leading to the hit in performance. Process context switch would happen more often for I/O intensive tasks than CPU/memory bound tasks. The INVPCID is the INValidate PCID which is used by the OS/hypervisor for to clear the TLB entries of a particular PCID (thereby avoiding having to flush the entire TLB).

The PCID is available from Westmere CPU generation and onwards. The INVPCID is available on Haswell and later CPUs. It would appear that Windows 10 cannot support PCID if the INVPCID instruction is not available as the Powershell Get-SpeculationControlSettings shows the PCID mitigation as FALSE if the INVPCID is masked out. Windows 2008 R2 cannot support PCID at all (so most likely Windows 7 too).

https://support.microsoft.com/en-us/help/4074629/understanding-the-output-of-get-speculationcontrols...

The hardware compatibility of the VM needs to be at least version 11 so that both the PCID and INVPCID is exposed to the VM.

I don't know about Linux and its many variants what its impact with Meltdown and its use of the PCID/INVPCID feature is like. But I also suspect it will be the similar if not same as with Windows.

So if your ESXi hosts have Ivy Bridge or earlier CPU or have EVC Mask for Ivy Bridge or earlier, there will likely be a hit to performance to VMs with disk I/O, network I/O intensive workloads with the Meltdown patch.

For Spectre, the performance impact is not so clear; as even the patches to it have been either pulled out or have an option to be disabled while Intel works on a less buggy microcode patch to Spectre.