Latest 6.0U3 VMware Tools causes Spectre variant 2 vulnerability in CentOS??

[cesprov]

Seeing a strange issue with VMware Tools here that I don't see anyone else talking about in respect to the Spectre variant 2 vulnerability.  I have a vSphere environment that is fully patched through the 6.0U3e vCenter and the ESXi 3/20/18 patches.  Within that environment, I have a bunch of CentOS 6.9 servers.  Now we have slacked off for quite some time on upgrading the VMware Tools within these CentOS VMs so most are still running a 5.1.x version, 9.10.1.47876 (build-2791197).  These VMs are fully patched with the latest kernel so they shouldn't be vulnerable to Meltdown or Spectre.  Prior to upgrading the VMware Tools, the Meltdown/Spectre check script, found here, reports it has not being vulnerable to all three, as shown below:

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'

* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)

* Kernel has array_index_mask_nospec (x86):  NO

* Kernel has the Red Hat/Ubuntu patch:  YES

* Kernel has mask_nospec64 (arm):  NO

> STATUS:  NOT VULNERABLE  (Mitigation: Load fences)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'

* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)

* Mitigation 1

  * Kernel is compiled with IBRS/IBPB support:  YES

  * Currently enabled features

    * IBRS enabled for Kernel space:  NO

    * IBRS enabled for User space:  NO

    * IBPB enabled:  NO

* Mitigation 2

  * Kernel has branch predictor hardening (arm):  NO

  * Kernel compiled with retpoline option:  YES

  * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)

> STATUS:  NOT VULNERABLE  (Mitigation: Full retpoline)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'

* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)

* Kernel supports Page Table Isolation (PTI):  YES  (found 'CONFIG_PAGE_TABLE_ISOLATION=y')

* PTI enabled and active:  YES

* Running as a Xen PV DomU:  NO

> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

Then if I upgrade to the latest version of VMware Tools for 6.0U3, 10.1.10.63510 (build-6082533), and re-run the script, it now shows the VM is vulnerable to Spectre variant 2:

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'

* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)

* Kernel has array_index_mask_nospec (x86):  NO

* Kernel has the Red Hat/Ubuntu patch:  YES

* Kernel has mask_nospec64 (arm):  NO

> STATUS:  NOT VULNERABLE  (Mitigation: Load fences)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'

* Mitigated according to the /sys interface:  NO  (kernel confirms your system is vulnerable)

* Mitigation 1

  * Kernel is compiled with IBRS/IBPB support:  YES

  * Currently enabled features

    * IBRS enabled for Kernel space:  NO

    * IBRS enabled for User space:  NO

    * IBPB enabled:  NO

* Mitigation 2

  * Kernel has branch predictor hardening (arm):  NO

  * Kernel compiled with retpoline option:  YES

  * Kernel compiled with a retpoline-aware compiler:  UNKNOWN

> STATUS:  VULNERABLE  (Vulnerable: Retpoline with unsafe module(s))

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'

* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)

* Kernel supports Page Table Isolation (PTI):  YES  (found 'CONFIG_PAGE_TABLE_ISOLATION=y')

* PTI enabled and active:  YES

* Running as a Xen PV DomU:  NO

> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

Using this to determine what the "unsafe module(s)" are shows:

VULNERABLE - No Retpoline found - vsock

VULNERABLE - No Retpoline found - vmci

Obviously these are VMware Tools components.  These were not reported as a problem with VMware Tools 9.10.1.47876 (build-2791197) but are being reported with 10.1.10.63510 (build-6082533).  What's the deal?  Are these VMs that are reporting those two components really vulnerable?  And if so, when can we expect VMware to fix this?

[bluefirestorm]

From the looks of it, the CPU does not have the Spectre microcode update or it has been masked out from the VM. That's why the report shows that it is relying on retpoline as mitigation instead of relying on IBPB and IBRS for mitigation.

    * IBRS enabled for Kernel space:  NO

    * IBRS enabled for User space:  NO

    * IBPB enabled:  NO

From what I understand the retpoline requires any software to be rebuilt using a C/C++ compiler than can generate the retpoline code to mitigate the risks. So without rebuilding any software with such a retpoline-capable compiler, there can be the possibility of that vulnerability without the CPU Spectre microcode update. It is unlikely the vmci and vsock were built with such compilers as there has been no VMware Tools release after the Spectre/Meltdown vulnerabilities were disclosed in January. I can't say whether what you see is a false positive or not.

Anyway, going back to the microcode update, if you look at this KB, https://kb.vmware.com/s/article/52085

The table shows the microcode update part (ESXinnn-201803402-BG) only covers certain CPUs. Notably absent in the table are Nehalem and Westmere Xeon CPUs and AMD CPUs. Intel has issued Spectre microcode updates for certain Nehalem and Westmere Xeons. Some Nehalem microarchitecture Xeons such as Bloomfield Xeons will not have any update.

So you should check if the server/motherboard vendor has BIOS/EFI firmware updates containing the Spectre mitigation if the CPU does not appear in KB list.

You can compare the EAX register of CPUID leaf 1 (from vmware.log of any VM) against the FMS (Family/Model/Stepping) in the table.

Example, this is an Ivy Bridge Xeon that is in the list:

vmx| I125: hostCPUID level 00000001, 0: 0x000306e4 0x00200800 0x7fbee3ff 0xbfebfbff

You can verify the presence/absence of the microcode update by checking the vmware.log of any VM. For Intel CPUs, you check CPUID leaf 7 EDX register it should show up as 0x0c000000; if it is 0x00000000 that means the Spectre microcode is not there.

vmx| I125: hostCPUID level 00000007, 0: 0x00000000 0x000027ab 0x00000000 0x0c000000

You should also check if the Spectre microcode is found and exposed to the VM.

vmx| I125: Capability Found: cpuid.STIBP = 0x1

vmx| I125: Capability Found: cpuid.IBPB = 0x1

vmx| I125: Capability Found: cpuid.IBRS = 0x1

In case you had masked it out and forgot to remove the "Intel Sightings" KB masks in the /etc/vmware/config file, that could also be the reason why it the IBPB and IBRS are indicated as not enabled.

https://kb.vmware.com/s/article/52345