Intel CPU bug - VMware fix on the way?

[andreaspa]

I've read up on forums, mailing lists and on The Register that there seems to be a severe hardware bug with Intel CPUs:

'Kernel memory leaking' Intel processor design flaw forces Linux, Windows redesign • The Register

There are Linux patches in the works, and Microsoft will release patches during January's patch tuesday. Is ESXi vurnerable, and if so, when can we expect a patch for this? Since it's a critical issue, it will require lots of patching and planning - any heads up would be appreciated!

[ITaaP]

I am still not clear on patching of the guest OS. Is this required in order in order to mitigate the vulnerabilities?

[kenthhjerpe]

As far as I can understand you also need to patch the Guest OS :

VMSA-2018-0004

VMware Requirements

• Deploy the updated version of vCenter Server listed in the table (if vCenter Server is used).
• Deploy the ESXi patches and/or the new versions for Workstation or Fusion listed in the table.
• Ensure that your VMs are using Hardware Version 9 or higher. For best performance, Hardware Version 11 or higher is recommended. VMware Knowledge Base article 1010675 discusses Hardware Versions.

Third party Requirements

• Deploy the Guest OS patches for CVE-2017-5715. These patches are to be obtained from your OS vendor.
• Update the CPU microcode. Additional microcode is needed for your CPU to be able to expose the new MSRs that are used by the patched Guest OS. This microcode should be available from your hardware platform vendor.
  VMware is providing several versions of the required microcode from INTEL and AMD through ESXi patches listed in the table. See VMware Knowledge Base 52085 for more details.

[BenediktFrenzel]

"Hypervisor-Specific Mitigation

Mitigates leakage from the hypervisor or guest VMs into a malicious guest VM. VMware’s hypervisor products are affected by the known examples of variant 1 and variant 2 vulnerabilities and do require the associated mitigations. Known examples of variant 3 do not affect VMware hypervisor products.

VMware hypervisors do not require the new speculative-execution control mechanism to achieve this class of mitigation and therefore these types of updates can be installed on any currently supported processor. No significant performance degradation is expected for VMware’s hypervisor-specific mitigations."

- VMware Knowledge Base - VMware Virtual Appliances and CVE-2017-5753, CVE-2017-5715 (Spectre), CVE-2017-5754 (Meltdown) (52264)

but as wila said, you may want to read the full articles.

- Benedikt

[Masch73]

And what about VM version of appliances such VCSA, PSC etc? My VCSA 6.0 U3d is VM version 8. It needs upgrade too? VMware requires - Ensure that your VMs are using Hardware Version 9 or higher. For best performance, Hardware Version 11 or higher is recommended....

[wila]

Hi,

Yes that sounds very likely, but it is a good question.

I've asked lamw​ on twitter what his thoughts on this are.

--

Wil

[andreaspa]

I'd argue that in order to solve the issue that one VM can read data from another VM, the VMware patches should be sufficient.

In order to solve memory leaks within a virtual machine, you would need to patch the Guest OS (if there is any patches available, if not, you're screwed).

[wila]

Hi,

William answered as follows:

"Investigation of VAs are still underway, once complete analysis has been provided & along w/resolution, my understanding is vHW guidance will be provided. I’d hold off unless KB explicitly instructs you to update vHW"

William Lam on Twitter: "@wilva Investigation of VAs are still underway, once complete analysis has ...

--

Wil

[ITaaP]

Well, I am glad the latest update from VMware addresses the CPU microcode. So far I am not seeing updated BIOS releases from Supermicro.

[nponeccop]

amitpatel001 On HP servers you need to update using the HP ESXi ISO (to update HP-specific drivers as well) and then apply the vmware patch.

If you didn't use custom iso to install ESXi and don't have non-standard drivers to update, you only need to apply the vmware patch.

To apply the patch you need

1) go to https://esxi-patches.v-front.de/ and read the instructions on how to turn on ssh and login

2) choose your ESXi version at the top

3) click the imageprofile link of the latest patch there - a small popup window will open

4) paste the commands from there into the ssh

5) Reboot the server

6) If ESXi crashes use Ctrl-R during boot to rollback

You only need the latest patch, they are cumulative

[ITaaP]

VMware says to use at least Virtual Hardware Version 9. Well, what about VCSA? That is only running version 8. Can we safely upgrade VCSA and PSC?

[wila]

Hi ITaaP,

Please see my answer to Masch73, it is currently not recommended to change the virtual hardware on any of VMware's appliances.

--

Wil

[ITaaP]

I guess with older servers running 5.0 or 5.1 that can't be upgraded, the only option is to replace the hardware? I realize those versions of ESXi are no longer supported.

[wila]

Hi ITaaP,

Most likely yes. Replacing the hardware is the only way forward in that case.

--

Wil

[AtosMatt]

I hoping someone can answer this question, can a Virtual Machine that is not patched (legacy OS) still access data from a Virtual Machine that is fully patched? If so, do we need to start separating vulnerable VM's from protected VM's?

[dalo]

No, not if you already patched your ESXi and VC.

[wila]

Hi AtosMatt,

By what means do you mean "can it still access data" ?

If you mean via normal guest OS network sharing then I do not see why it would not be able to do so.

--

Wil

[Kosch]

Would appear that HP BL460c Gen 7 servers do not have a BIOS update but apparently are not vulnerable according to HP.

Matrix  - HPE | Side Channel Analysis Method

Patched ESX 6 with Windows 2012 R2, Hotfix and registry keys enabled. Seems to hint I need a BIOS update but HP say I dont need one.

[AtosMatt]

I mean can a non-patched Virtual Machine still potentially read the kernel memory of other Virtual Machines that have received the January Meltdown/Spectre patches?

[dalo]

AtosMatt​ as I told you: If you patch your VC and all ESXi you should be save, according to the known info.

Patching the OS is needed for mitigation inside the VM process to process.

[JoeGasper]

We found that you could have everything patched and still not be all green with SpeculationControl.

Per this KB, the order of patching/microcode is important, as we seem to have encountered.

https://kb.vmware.com/s/article/52085

We patched hosts, then VC.  Still had absence of hardware support = true.

Reboot of the hosts (all in the cluster, as stated), and after a VM reboot, SpeculationControl was all green.