Virtual Machine missing expected CPU Flags

[sentania]

With meltdown and the associated impacts to performance, I've read a lot that a processor capable of PCID is able to mitigate this impact in some scenarios.  This capability appears to have been made available with Westmere processors.

My hosts in my lab have X5675s in them, which are west mere processors.  I have verified the presecense of this flag by booting the host to a gentoo live CD and executing /proc/cpuinfo (screenshot).

I have my cluster configure to run *without* EVC, and when I run /proc/cpuinfo on both an ubuntu VM and a Centos 7 VM - I *do not* see the PCID flag; however at work when I check on VMs running the same CentOS Kernel version, same ESXi version, but on E5-2680 v4s - I see the pcid flag.

What is ESX doing to hide these flags from the guest on my westmere chips, but then exposing them on a newer chip?

Lab VM:

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz

stepping        : 2

microcode       : 0x15

cpu MHz         : 3066.775

cache size      : 12288 KB

physical id     : 0

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 11

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm tsc_adjust arat

bogomips        : 6133.55

clflush size    : 64

cache_alignment : 64

address sizes   : 42 bits physical, 48 bits virtual

power management:

[mihailim]

You need to upgrade your VM to virtual hardware version 11 (ESXi 6.0) or newer in order for the CPU PCID support to be exposed.

[mihailim]

Oh, forgot: And you then need to powercycle the VMs (full shutdown, boot up again). A simple reboot doesn't destroy the world instance, and the CPUID flags are masked/unmasked only on a cold boot of the VM.

[bluefirestorm]

Westmere generation of CPUs introduced the PCID attribute in the TLB. It needs a minimum of hardware compatibility version 9 for this to be exposed.

But in order for the PCID to be used, my understanding is that just like Windows Meltdown patches, the Linux Meltdown patches also require the INVPCID (INVadliate PCID) instruction to be available to mitigate against possible performance hit. The INVPCID instruction was introduced in the Haswell generation of CPUs. Hardware compatibility version 11 or higher is required for INVPCID instruction (and other Haswell instruction such as movbe, bmi1, bmi2) to be available in a VMware VM.

You can check bit 17 of CPUID leaf 1 ECX register for the presence/absence of PCID in the vmware.log of any VM. INVPCID can be checked on bit 10 of CPUID leaf 7 EBX register.

So the Westmere X5675 will not have the INVPCID instruction. So you might see "pcid" in the flags but you won't see "invpcid" and "invpcid_single" on a Linux VM with the Meltdown patches. The E5-2680 v4 being Broadwell CPUs should have both: the PCID TLB attribute and the INVPCID instruction.

[mihailim]

No; unlike Windows, the Linux Meltdown patches *do not* rely on INVPCID support. E.g., from a Sandy Bridge workstation on 4.15.5 or higher (mainline and stable; RHEL kernels have these patches backported):

user@host:~$ grep ^flags /proc/cpuinfo |uniq

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid xsaveopt ibpb ibrs stibp dtherm ida arat pln pts

user@host:~$ cat /sys/devices/system/cpu/vulnerabilities/meltdown

Mitigation: PTI

[mihailim]

Ah, I parsed your reply incorrectly. Indeed, you're correct! Without INVPCID the performance hit is more noticeable (_dramatically_ so in some scenarios--without PCID I've measured 850% slowdowns--that's right, almost 9-fold perf loss--for a mixed SQL+webserver+appserver workload, with PCID but no INVPCID dropping to 310% loss). Just wanted to underline that lack of INVPCID does *not* mean the KPTI mitigation is ineffective.

[sentania]

HW version 13, and many cold boots later - and I still don't see PCID.

I agree and understand I won't see INVPCID because westmere doesn't have them, but I'm not clear why I'm not seeing PCID.

Unless there is a BIOS option on the hosts I am missing, but I'm seeing simlar behavior on both my Dell C6100s with what I believe to be the optimal settings enabled, and a Supermicro box with, again optimal settings.

I've created a gist of the log of a given VM that displays this issue:

Missing cpu flags · GitHub

Notables:
ESX 6.5

HW v13

Many a cold boot.

What am I missing?

[mihailim]

According to the vmx log, the capability is being exposed to the VM:

2018-04-16T03:26:56.206Z| vmx| I125: Capability Found: cpuid.PCID = 0x1

However, the guest kernel also needs to support this... Is it possible you're runing a kernel old enogh to not support it in the first place? What is the complete kernel version of the guest?

[mihailim]

For reference, X86_FEATURE_PCID was first added in kernel 3.2: https://elixir.bootlin.com/linux/v3.2/ident/X86_FEATURE_PCID  -- so if your kernel is older than 3.2.0 and/or if your distribution hasn't backported that support, it will not be exposed.

[sentania]

[sadmin@www ~]$ uname -a

Linux www.int.sentania.net 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux