VMware Cloud Community
TRBoeh
Contributor
Contributor
Jump to solution

ESXi 6.5U1 HPE Custom Image / Protection against Spectre

Dear all,

I have two HPE ML350 Gen9 Server running with ESXi 6.5U1 Build 7388607 HPE Custom Image.

Latest BIOS-Update P92 2.56_01-22-2018​ 23.02.2018 is installed.

All running Windows 2012 R2 VMs got the latest Microsoft updates (including KB4056098). Antivirus software is also running and up-to-date.

The registry-keys were set, as mentioned in: https://support.microsoft.com/en-za/help/4072698/windows-server-guidance-to-protect-against-the-spec...

If I run the the Microsoft "SpeculationControl"-Powershell-script, I get as result, that the CPU still have to be updated by microcode (please see attachment).

Does anybody know which additional vmWare-Update I need to install?

Thanks in advance for your assistance!

Regards

Marcus

1 Solution

Accepted Solutions
bluefirestorm
Champion
Champion
Jump to solution

That is bizzarre!

Running on the same ESXi host (so that means the microcode is present and active in the host hardware), OK for 2 VMs but not the DC VM. Running on same ESXi host also precludes the /etc/vmware/config mask from the "Intel Sightings" KB as the culprit.

How about the DC VM hardware compatibility setting? Is it also version 13 (supported by ESXi 6.5) or at least same as the TS/Exchange VMs? I think the recommended minimum is version 9.

Other than HW compability version, I can't think of any other possible reason why. But the fact the stibp, ipbp, and ibrs capabilities show up in the vmware.log that means the ESXi hypervisor found the microcode update and should be exposed to the VM.

Alternatively, you could remove those registry settings; because those registry settings were really introduced to mitigate the crashing in January (just like the VMware Intel Sightings KB).

View solution in original post

Reply
0 Kudos
21 Replies
bluefirestorm
Champion
Champion
Jump to solution

Intel released microcode updates for Spectre for some Broadwell/Haswell CPUs last week. Given what happened (servers rebooting after the microcode updates) back in January, I wonder how aggressive system vendors like HP and VMware this time around in pushing out these updates.

It is interesting that Intel is planning updates for CPUs as far back as the Penryn generation.

https://newsroom.intel.com/wp-content/uploads/sites/11/2018/03/microcode-update-guidance.pdf

TRBoeh
Contributor
Contributor
Jump to solution

Dear MBreidenbach0,

thanks for the links, some of them I still have read before, but I suspect, that I do not really understand what exactly I have to do now...

What does it mean in VMware Knowledge Base ?

Editing the config file will bypass or solve the "problem"?

I still have installed HPE latest and (as they worte) tested BIOS-Update...

Sorry, but the confusion won't go away...

Thanks in advance for your "enlightenment"!

Regards,

Marcus

Reply
0 Kudos
MBreidenbach0
Hot Shot
Hot Shot
Jump to solution

My understanding of the situation is:

The CPU microcode update adds new CPU functionality. To support this a ESXi patch, a vCenter update and some configuration is required (EVC, VM hardware version etc).

The CPU microcode update and the ESXi patch have been pulled since they caused problems (I updated my HP Z230 workstation BIOS on january 11 and never saw so many blue screens in such a short time - until I disabled it via registry key). Intel has released new microcode for some CPUs but not for all; VMware hasn't re-released the ESXi patches yet (AFAIK).

So: wait for CPU microcode update and ESXi patch re-release

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

What does it mean in VMware Knowledge Base ?

The Intel Sightings KB was in response to reports of servers rebooting with the initial microcode update provided by Intel in January. VMware was one of the first to break ranks, if you will, advising customers to undo the patches and recall the VMware patches.

Editing the config file will bypass or solve the "problem"?

The microcode updates supply these CPU updates. They are called

IBPB - indirect branch predictor barrier

IBRS - indirect branch restricted speculation

STIBP - single thread indirect branch predictors

If you refer to "the problem" as the unexpected reboot, the edit of the /etc/vmware/config is to mask out the faulty microcode updates from January. Any new microcode update will require that line that masks out bits 26, 27 of CPUID leaf 7 EDX register is to be removed in order to be effective. That is why you see these two key bullet points in the same KB about the /etc/vmware/config edit.

  • This will hide the speculative-execution control mechanism for virtual machines which are power-cycled afterwards on the ESXi host.
  • This line will need to be removed after applying a future fixed microcode from Intel in order to enable the full guest OS mitigations for CVE-2017-5715.

To fix the Spectre problem in an ESXi context requires two parts: the microcode update (from Intel presumably delivered to you by the server hardware vendor) and the software patch from VMware so that ESXi hypervisor recognises the microcode updates and also expose IBPB, IBRS, and STIBP "features" to the VMs.

You can read through this KB https://kb.vmware.com/s/article/52085 , there is a section about how to verify the microcode update is working and exposed to the VMs.

TRBoeh
Contributor
Contributor
Jump to solution

Dear MBreidenbach0 and bluefirestorm,

thanks for your replies and your suggestions.

I suspected that the only way ("best practice") at the moment is: sit and wait for updates.

What I could do is done (BIOS,Update, ESX-Update, Win2012R2-Updates) - the rest is outside my "force"...

Regards,

Marcus

edsanchez13
Contributor
Contributor
Jump to solution

Hi Marcus;

Did you power cyle the VM (shut down, then power on) after the BIOS update was performed? That needs to be done in order for the VM to pick up the new CPU features.

Suggested steps to check:

* BIOS update applied - you have already done this

* Guest OS patched -  you have already done this

* Guest OS registry keys set to enable protection - you have already done this

* VM hardware level at v9 or newer? - since you have 2012 VMs on ESXi 6.5 I would suspect you are good here, but double check

* Power cycle the VM (Cold boot), rebooting it wont help.

Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Hi Edsanchez13,

thanks for your reply!

Everything you suggested I still have done... And yes, all VMs are > V9, they are V11.

But thanks again for your suggestions!

Sit and wait is the best thing I could do at the moment?!

Regards

Marcus

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

Looks like you just have to wait for the announcement from VMware as to when the ESXi patches will be re-issued.

You can also confirm whether the EFI update is effective by checking the vmware.log and look at the output of EDX register CPUID leaf 7.

I pasted here an output from a Workstation 12.5.9 Windows VM with a Skylake CPU with the Spectre microcode patch. The value 0x0c000000 is the EDX register output and the hexadecimal "c" indicates that bits 26-27 are 1 and therefore the Spectre microcode updates are present in the host CPU.

vmx| I125: hostCPUID level 00000007, 0: 0x00000000 0x029c6fbf 0x00000000 0x0c000000

Reply
0 Kudos
pkohn
Contributor
Contributor
Jump to solution

Hi Marcus,

same problem here with ESXi 5.5.

ESXi 5.5 Cluster fully patched / Windows VMs still vulnerable (Get-SpeculationControlSettings & ...

Regards Philipp

“If you expect great things of yourself and demand little of others, you’ll keep resentment far away.” -Confucius
Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Dear bluefirestorm,

I think this will be the best opinion... Do I have any other option?

Regards

Marcus

Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Hi Philipp,

as mentioned to bluefirestom - I think the only way is to sit and wait...

Same as you - I still do not like it anyway - but what shall we do?!

We are adidcted.........

If you like we could write in German by PM?

Regards

Marcus

Reply
0 Kudos
edsanchez13
Contributor
Contributor
Jump to solution

take a look at this, if you can run that script and share the output would be great. https://virtualcornerstone.com/2018/03/19/verify-new-spectre-mitigation-patches-using-powercli-and-v... 

Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Hi Edgar,

thanks for your reply!

I promise you, I will do this - hoping, I will manage it this week..................

Cheers,

Marcus

Reply
0 Kudos
edsanchez13
Contributor
Contributor
Jump to solution

Hi Marcus;

VMware today released new updates that include the new microcode updates as well. If you want to try installing the VMware updates then the script should be able to validate both scenarios.

https://kb.vmware.com/s/article/52085VMware Knowledge Base

VMSA-2018-0004.3 | United States

Edgar

edsanchez13
Contributor
Contributor
Jump to solution

Hi Phillip;

In your case being ESXi 5.5 and Build ID: 7618464, this update removed both the microcode updates and the framework to allow guest OSes to utilize the new speculative-execution control mechanisms. When the first microcode updates were released, ESXi 5.5 had 1 patch (framework and microcode), and ESXi 6.0 and 6.5 it was 2. VMware Knowledge Base

go ahead and update your 5.5 today. note that in this release the control mechanism is a separate patch not like the first release that was pulled.

Edgar

Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Hi Edgar,

finally I managed it, and done my homework... Smiley Wink

I've applied the BIOS update for my ML350G6 and microcode update 6.5.0.

Two VMs are telling me, everything is fine, but the Windows 2012R2 DC doesn't "see" the spectre protection...

All VMs have the same patch-level and also the registry-keys set.

Please find attached the summary - also with the output form your script!

Do you have any suggestions, what I could do? (I've still uninstalled and re-installed the Microsoft patches...)

Thanks in advance!

Regards

Marcus

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

Is the domain controller VM running on the same ESXi host as the Terminal Server and Exchange Servers?

Do you see the following lines in the vmware.log of the DC VM?

| vmx| I125: Capability Found: cpuid.STIBP = 0x1

| vmx| I125: Capability Found: cpuid.IBPB = 0x1

| vmx| I125: Capability Found: cpuid.IBRS = 0x1

If you don't see those lines in the vmware.log and it is running on the same host as the Terminal/Exchange server VMs, somehow the Spectre microcode is masked/disabled for the DC VM.

The ESXi650-201803402-BG (microcode update) does not cover Westmere CPUs based on the table in KB52085 https://kb.vmware.com/s/article/52085

So you would need to have the BIOS update from HP to have the Spectre microcode. Intel did release the microcode update for Westmere EP series.

https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode-update-guidance.pdf#page=15...

The FeatureMaskOverride registry settings for Windows requires a reboot in order to take effect (whether to enable/disable).

Perhaps you can attach the vmware.log of the DC VM if you are still stuck with the same problem with the DC VM.

Reply
0 Kudos
TRBoeh
Contributor
Contributor
Jump to solution

Hi bluefirestorm,

thanks for your reply.

The DC is running on the same host as the terminal and exchange server.

The three lines are in the vmware.log:

pastedImage_0.png

I've installed latest BIOS from HP and also the vmware-Update 'ESXi-6.5.0-20180304001-standard' (https://esxi-patches.v-front.de/ESXi-6.5.0.html#2018-03-20 )

(the host isn't running a custom HP Image 6.5.0). Done a cold-boot.

And I always rebooted the DC-vm after changing the registry...

Reply
0 Kudos