Specre/Meltdown EVC Cluster

[Balteck71]

hello, last month HP (with a firmware update from G6 to G10 servers), VMWare (with the last ESXi patch) and Intel (with a OS patch) released an update CPU microcode for several Xeon Processors.

According to the last microcode-update-guidance (https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode-update-guidance.pdf ), at time of writing Intel patched also Nehalem and Westmere processor putting them in production status from beta status.

My customer has a cluster of 3 old HP Proliants (DL380G6 - E5520 / DL380G7 - E5640 / DL380G8 - E5-2450L ) with 4 VMs in a EVC cluster that is not protect from spectre/meltdown.

Now that a new BIOS is out and the 30 march version of ESXi is out, I wish to suggest him to update the cluster.

But reading the various KB, it seems that the actual microcode available is not for Westmere/Nehalem (VMware Knowledge Base ), and testing the VMs for Guest OS mitigation patch, I've discovered that EVC tells to the VM that it is running on CPUID 106A4, so that inspectre tool gives me error with: NO MICROCODE AVAILABLE.

On the microcode update guidance, CPUID 106A4 is referred to Core i7-920 that will be never patched by Intel.

But Nehalem is identified with CPUID 106A5, that it is already patched with microcode 0x1c (final) from 0x1b (beta)

My question is:
How can I identify which microcode version is applied on the VM?

or better: how can I identify which microcode is present on HP BIOS o ESXi patches? Intel document is dated 2 April, while esxi is 30 March and HP is 22 February

If the wrong CPUID is passed on VM, does the guest OS patch work as expected?

[bluefirestorm]

The microcode part of the VMware ESXi updates ESXnnn-2018030402-BG covers only up to Sandy Bridge based on the table in KB52085. So you would need to obtain the microcode update from a firmware/BIOS update from HP to get the updates for Nehalem EP/EX and Westmere EP/EX CPUs.

You can see the CPUID or Family/Model/Stepping from the vmware.log of any VM in CPUID leaf 1 EAX register.

| vmx| I125: hostCPUID level 00000001, 0: 0x00040661 0x02100800 0x7ffafbbf 0xbfebfbff

To check the microcode version, you can also check the vmware.log of any VM.

| vmx| I125: Minimum ucode level: 0x00000019

If you have a Linux VM, you can also check the microcode version from Terminal

cat /proc/cpuinfo

If the wrong CPUID is masked to the VM, I don't think the OS patch will work as it will not contain the STIPB, IBPB, IBRS microcode updates that are used for Spectre mitigation. You can mask CPUID leaf 1 EAX register to fool the VM guest OS about the family/model/stepping but you cannot make up a feature that does not exist.

Meltdown mitigation requires guest OS update; it does not need an update to microcode or update to ESXi.

[Balteck71]

Thank you very much.

Meltdown patch is at Guest OS level, so the VMs are already patched, becuse I did every automatic updates.

Regarding Spectre, ESXi microcode patch seems useless, while HP firmware will do the trick.

But VMs uses the right updated microcode in new firmware?

In vmware.log I found:

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID guest vendor: GenuineIntel

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID guest family: 0x6 model: 0x1a stepping: 0x4

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID *host family: 0x6 model: 0x2d stepping: 0x7

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID guest codename: Nehalem (Core i7)

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID *host codename: Sandy Bridge EP

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID guest level 00000001,  0: 0x000106a4 0x00010800 0x81b82201 0x0fabfbff

2018-04-24T08:04:00.228Z| vmx| I125: guest vs. host CPUID *host level 00000001,  0: 0x000206d7 0x00200800 0x17bee3ff 0xbfebfbff

Hoping that HP firmware contains the last microcode, as I found here: CPUMicrocodes/Intel at master · platomav/CPUMicrocodes · GitHub (there is a method to know which version of CPU microcode is present in a BIOS file?), You can see that 106a4 is not patched, while 106a5 is it, because according to Intel, Nehalem Xeons are stepping 0x5 (106a5).

Why a cluster with EVC enable masks CPU for Nehalem compatibility with this CPUID?

I looked on EVC Cluster settings and I cannot find a way to modify it.

[Balteck71]

a little update: On a vmware.log of a VM running on Nehalem Server:

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID guest vendor: GenuineIntel

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID guest family: 0x6 model: 0x1a stepping: 0x4

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID *host family: 0x6 model: 0x1a stepping: 0x5

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID guest codename: Nehalem (Core i7)

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID guest name: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID *host name: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID guest level 00000001,  0: 0x000106a4 0x00010800 0x81b82201 0x0fabfbff

2018-04-22T10:15:05.184Z| vmx| I125: guest vs. host CPUID *host level 00000001,  0: 0x000106a5 0x00100800 0x009ce3bd 0xbfebfbff

[bluefirestorm]

You should not worry about what EVC assigns.

What matters is whether the Spectre microcode is really present. Look at EDX register CPUID leaf 7 and the Capability Found

| vmx| I125: hostCPUID level 00000007, 0: 0x00000000 0x000027ab 0x00000000 0x0c000000

| vmx| I125: Capability Found: cpuid.STIBP = 0x1

| vmx| I125: Capability Found: cpuid.IBPB = 0x1

| vmx| I125: Capability Found: cpuid.IBRS = 0x1

You should obtain the firmware from HPE and not from some GitHub site that is an official source of the system vendor.

http://h22208.www2.hpe.com/eginfolib/securityalerts/SCFM/Side_Channel_Downloads.html

The minimum VM hardware compatibility should be set to 9.

If you see 0x00000000 for the EDX register CPUID leaf 7, that means the Spectre microcode is not present in the firmware. Similarly, if you don't see the STIBP, IBPB, and IBRS that could also mean the ESXi update is not exposing the microcode to the VM.

[Balteck71]

InSpectre utility misled me about microcode availability on VMs looking for the wrong CPUID, but instead is esxi that uses the right microcode (from its patch or BIOS) for VMs whatever is the EVC masked CPU. Is it correct?

I also read that Meltdown and Spectre have a performance hit on older CPU that not have PCID and INVPCID capabilities. (who tell 2%, who tell 30%, who tell 80%)

Because in Nehalem is not present PCID, How can I test the performance on VM before and after the patch?

Is Meltdown mitigation or is Spectre mitigation that reduces performance?

Is it in some way reversible?

My fear is to update HP BIOS and ESXi on my customer and to have a customer angry with me because its system is worst than before

[bluefirestorm · Accepted solution]

InSpectre utility misled me about microcode availability on VMs looking for the wrong CPUID, but instead is esxi that uses the right microcode (from its patch or BIOS) for VMs whatever is the EVC masked CPU. Is it correct

I don't know about the InSpectre utility. You should check the vmware.log as I mentioned earlier (CPUID leaf 7 EDX register and the STIBP, IBPB, IBRS capabilities).

I also read that Meltdown and Spectre have a performance hit on older CPU that not have PCID and INVPCID capabilities. (who tell 2%, who tell 30%, who tell 80%)

PCID attribute is available in Westmere and later CPUs and INVPCID instruction is available in Haswell and later CPUs. It is only useful for mitigation against performance hit due to Meltdown patch and only certain versions of Windows can support INVPCID instruction. For example, it is explicitly stated that Windows 2008 does not support PCID. The PCID and INVPCID has nothing to do with Spectre. Linux VMs will behave in similar manner to the Meltdown patch with regards to PCID/INVPCID. The hit will vary according to the workload of the VM.

https://support.microsoft.com/en-us/help/4074629/understanding-the-output-of-get-speculationcontrols...

Is it in some way reversible?

For Spectre microcode, you can always mask out the CPUID leaf 7 EDX register bits 26, 27 which will mask the STIPB, IBPB, IBRS microcode updates. But masking these out will raise issues with EVC cluster.

https://kb.vmware.com/s/article/52345

For Windows OS, there are the registry settings that will mask out both Spectre and Meltdown patches.
https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-spec...

I don't know what recourse you have for Linux Meltdown patch but I suppose if you want to able to reverse, you can try a restore to an earlier state from a backup of the VM. Linux VMs also makes use of LFENCE instruction and RETPOLINE as mitigation against Spectre but this requires the application(s) and OS kernels to be rebuilt with a RETPOLINE capable compiler.

I would guess the VM(s) with older guest OS and running on old CPUs will hardly be any noticeable hit in performance as these VMs/OS could not take advantage of the new CPU features that would have made things run faster/more efficiently. In short, if they were not fast before the Spectre/Meltdown patch, would you still notice that they slowed down? Would you notice if a snail is crawling more slowly if it is on a sticky surface versus a slippery surface?

[Balteck71]

So performance hit is with Meltdown patch enabled. Because is a guest OS software patch, I can always disable it if the performance is too worse (my customer has a SQL server 2008, Exchange server 2007 and Windows 2012r2 DC plus a Linux Centos 5 or 6 - I dont remember exactly).

Patching BIOS and ESXi for Spectre doesn't create a performance it. Is it right?

Surely this cluster is not very fast, but I don't know a synthetic test to do against this VMs giving me where the performance goes down.

Is there available a benchmark test for VM workloads?

For example the administrative program that uses SQL will begin to execute any SQL commands 2 or more times longer.

Or searching a mail in Exchange: Now it almost instantaneous (he has 25 employees), and then?

[bluefirestorm]

I don't know of any particular synthetic benchmark for SQL Server 2008 and Exchange 2007. But even if you have one, it is still synthetic and it may not represent the actual workload that your customer has.

I think there is still a performance hit for Spectre patch as the predictive execution within the CPU is toned down but the significant hit is with the Meltdown patch.

From what I read and understand, the Meltdown patch hits performance on I/O intensive workloads (disk and/or network). So the SQL Server and Exchange Server could be potentially be hit. The key is whether the end-users will notice it or not. The more likely will be the SQL Server if there are large query result sets that needs to be read from disk and transmitted over the network. The best case scenario is that end-users don't notice; although as ESXi administrator you may notice that the CPU percentage utilisation go up.

Since 3-node cluster does not even have a Haswell CPU, you don't have any ESXi host that has the INVPCID instruction. Even if you do, being in an EVC cluster with Nehalem and Westmere CPUs, the EVC would have masked it to the lowest common denominator of Nehalem. So you don't have much choice. On the other hand, even if there is a noticeable performance hit, if you want disable the either Spectre patch or Meltdown patch or both; you still have to ask your customer whether they accept the risk their system won't be protected against Spectre/Meltdown attack.

[Balteck71]

My suggestion is to block any possible risk, but the money to buy new servers is on my customer pocket...

Now CPU utilization is under 50%, so I don't think that a bigger utilization is a problem

Anyway do you think that replace nehalem CPU with Westmere CPU (that it is very cheap now) may help? is it worth it?

[bluefirestorm]

I think it is a complex situation to do the hardware and software upgrade. Your customer has very old software (SQL Server 2008 and Exchange 2007) which may not be able to take advantage of new features of newer CPUs/hardware. So cost/money aside, it will be tricky to talk about upgrades.

Westmere Xeon has features such as AES, PCLMULQDQ instructions, PAUSE loop exiting that Nehalem does not have. AES and PCLMULQDQ instruction can make encryption/decryption a lot faster but both SQL Server 2008 and Exchange 2007 were available before Westmere (2010); so unlikely these software can take advantage of it. PAUSE-loop exiting is virtualisation feature that ESXi can take advantage of.

Anyway, I think it is better to handle the Spectre/Meltdown situation first rather than jumping to discuss about changing/upgrading hardware/software with your customer. Because the outcome of this will likely influence the need/desire/push to change or not. If performance is not an issue, going with the status quo with the hardware/software will be strong. The tricky bit is really the SQL Server. If the customer environment is such that end-users can create their own queries, the chances of having very inefficient SQL queries being generated is very high. It might be some user will run a monthly/quarterly/annual report and uses a very inefficient query you won't be able to catch it immediately until that periodical inefficient query is run. There is also the risk of "poisoning the well" if you begin to discuss about performance/slowdowns, especially with end-users, as they might just end-up looking for slowdowns or start becoming defensive if you ask them how they use SQL Server or operate their business with their IT infrastructure.

Well it does not stop you and I from throwing around some ideas.

Instead of upgrading the Nehalem to Westmere, what if you decommission the Nehalem from the EVC cluster? If the Westmere G7 and Sandy Bridge G8 can handle the load of all VMs, removing the Nehalem G6 allows you to set the EVC mode to Westmere. At most you lose 4 cores (assuming dual socket E5502). If you want to do an upgrade perhaps add another E5640 to the G7 if there is still one socket available and then decommission the Nehalem G6.

https://ark.intel.com/compare/64610,47923,37092

[Balteck71]

Maybe there is another way.

If the slowdown is made with OS Meltdown patches, I can disable it and block access to the VMs leaving only SQL or Exchange service available to the users.

Meltdown can not be used remotely and the risk of an attack is very very very low...

Also Spectre variant 2 can be disabled on VM. But I didn't find a clear KB that explain how Windows mitigate Spectre variant 1 and if it can be disabled too.

What I don't know if the new HP firmware (that apply the new CPU microcode) and the last ESXi patch of 30 March for Spectre variant 1 and 2 create a slowdown with all guest OS mitigation disabled.

[bluefirestorm]

I think the ESXi patches is only to expose the vCPU capability (stibp, ibpb, ibrs) to the VM for CVE-2017-5715 (branch target injection); so unlikely that introducing the ESXi patch and CPU microcode will cause any slowdown.

For CVE-2017-5753 (bounds check bypass), in theory, can be exploited by any application that can allow access to memory beyond its allowed range (example: accessing an array in memory beyond its allocated size). That's why you see the statement from the Google security blog

https://security.googleblog.com/2018/01/more-details-about-mitigations-for-cpu_4.html

This vulnerability affects specific sequences within compiled applications, which must be addressed on a per-binary basis.

So if there is a(re) patch(es) for CVE-2017-5753, it will depend on the operating systems and applications if that type of vulnerability exists and discovered to be exploitable for Spectre. If I am not mistaken, Apple released a patch for the Safari browser in this regard.