VMware Cloud Community
ronikh
Contributor
Contributor
Jump to solution

XXX esx.problem.hyperthreading.unmitigated.formatOnHost not found XXX (Build 9313334)

Hey everyone,

today we updated one of our x3650M4 hosts from Build 9239799 (ESXi-6.0.0-20180704001) to Build 9313334 (ESXi-6.0.0-20180804001) in order to mitigate the latest security vulnerabilities.

BaseImage: Lenovo Customized Image ESXi6.0u1b (based on ESXi 6.0 Vmkernel Release Build 3380124)

CPU:  Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz


The update was successful, but after reboot, we received this message: "XXX esx.problem.hyperthreading.unmitigated.formatOnHost not found XXX"  (Images attached below)

Any recommendations/solutions? Couldn't find anything online.

Thank you.

UPDATE:  after updating the vCenter from VC-6.0.0U3g to VC-6.0.0U3h, then rebooted vCenter and the Host, the issue disappeared, but still shows empty notification and warnings in Tasks&Events. (Images attached)

1 Solution

Accepted Solutions
Vijay2027
Expert
Expert
Jump to solution

VMware published a KB article about this behavior after patching to latest build.

L1TF in the VMware Knowledge Base

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered

View solution in original post

Reply
0 Kudos
11 Replies
JohnSc42
Contributor
Contributor
Jump to solution

I also just ran into this problem.  See the release notes about a new hyperthreading option, here  VMware vCenter Server 6.0 Update 3h Release Notes  and the kb article here VMware Knowledge Base . I enabled the new setting in Advanced Settings.VMkernel.Boot.hyperthreadingMitigation and rebooted the server and the warning went away. Looks like there is a suppress warning option also, if you don't want to implement it.

Reply
0 Kudos
IRIX201110141
Champion
Champion
Jump to solution

If you apply the current patches to 6.x Hosts you will get a warning after the Host comes back and it looks like "This host is potentially vulnerable to issues described in CVE-2018-3646, please refer to https://kb.vmware.com/s/article/55636 for details and VMware recommendations.  KB 55636 ". You than have the option to supress the Warning.

About your Error message please search in the VMware KB for the term "no found XXX" and you'll find a couple of KBs about the Issue.  It might have to do with your language or account settings and its very common and now the client cant find the transalation for that message. I see it more with 3rd. Party plugins rather than original VMware stuff.

Regards,

Joerg

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

VMware published a KB article about this behavior after patching to latest build.

L1TF in the VMware Knowledge Base

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

Let us know if you fix your problem KB55806.

Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly.

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
ronikh
Contributor
Contributor
Jump to solution

Thanks for your help guys.
So what we did is basically suppress the warning in Configuration > Advanced Settings (Software) > UserVars > UserVars.SuppressHyperthreadWarning = 1

Wafi_aljunaedi
Contributor
Contributor
Jump to solution

Hi
i have too tested this way in KB55806. and it succeed :

Scheduler-Enablement Phase: Enable the ESXi Side-Channel-Aware Scheduler

Enabling the ESXi Side-Channel-Aware Scheduler using ESXi Embedded Host Client

  1. Connect to the ESXi host by opening a web browser to https://HOSTNAME.
  2. Click the Manage tab
  3. Click the Advanced settings sub-tab
  4. Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
  5. Select the setting by name and click the Edit pencil icon
  6. Change the configuration option to true (default: false)
  7. Click Save.
  8. Reboot the ESXi host for the configuration change to go into effect.
Reply
0 Kudos
WuGeDe
Enthusiast
Enthusiast
Jump to solution

@all

I can not recommend to enable this ESXi Side-Channel-Aware-Shedule until VMware has fixed ALL the flaws that comes with it.

  1. Planning Phase: Assess Your Environment

The Concurrent-context attack vector is mitigated through enablement of the ESXi Side-Channel-Aware Scheduler which is included in the updates and patches listed in VMSA-2018-0020. This scheduler is not enabled by default. Enablement of this scheduler may impose a non-trivial performance impact on applications running in a vSphere environment. The goal of the Planning Phase is to understand if your current environment has sufficient CPU capacity to enable the scheduler without operational impact.
The following list summarizes potential problem areas after enabling the ESXi Side-Channel-Aware Scheduler:

  • VMs configured with vCPUs greater than the physical cores available on the ESXi host
  • VMs configured with custom affinity or NUMA settings
  • VMs with latency-sensitive configuration
  • ESXi hosts with Average CPU Usage greater than 70%
  • Hosts with custom CPU resource management options enabled
  • HA Clusters where a rolling upgrade will increase Average CPU Usage above 100%

Important: The above list is meant to be a brief overview of potential problem areas related to enablement of the ESXi Side-Channel-Aware Scheduler. The VMware Performance Team has provided an in-depth guide as well as performance data in KB55767. It is strongly suggested to thoroughly review this document prior to enablement of the scheduler.


Note: It may be necessary to acquire additional hardware, or rebalance existing workloads, before enablement of the ESXi Side-Channel-Aware Scheduler. Organizations can choose not to enable the ESXi Side-Channel-Aware Scheduler after performing a risk assessment and accepting the risk posed by the Concurrent-context attack vector. This is NOT RECOMMENDED and VMware cannot make this decision on behalf of an organization.

This is simple ridicolous what VMware here expects of its customers.

Regards

Reply
0 Kudos
cypherx
Hot Shot
Hot Shot
Jump to solution

I'd like to know why am I getting this message even after I installed the latest manufacturers BIOS images (which includes Intel CPU microcode)?  If I have installed the latest BIOS which in Dell's changelog specifically states it mitigates this CVE, then WHY after updating to ESXi 6.0.0 10719132 and vSphere 6.0 u3h is this still alerting?  Isn't it supposed to do a little test on the server CPU and see if its mitigated or not? 

In vSphere C# client it just says configuration issue.

In vSphere web client it has a link to the far right that says __MISSING RESOURCE__ but if you click on __MISSING RESOURCE__ it goes away, as does the triangle icon on the host in both web client and C# client.

missing-resource.PNG

Reply
0 Kudos
sk84
Expert
Expert
Jump to solution

Are you sure that the BIOS update and cpu microcode patches are sufficient for mitigating CVE-2018-3646 (L1TF)?

There are two essential components that need to be applied to mitigate the above mentioned vulnerabilities:

1. System BIOS was previously released for CVE-2018-3639 and CVE-2018-3640 which contains the necessary microcode. Please check those Product Tables for your system.

2. Operating System & Hypervisor updates.

Speculative Execution Side-Channel Vulnerabilities (CVE-2018-3615, CVE-2018-3620, and CVE-2018-3646)...

And I've never seen this __MISSING RESOURCE__ error in the warning before. It should say "Suppress warning". For this reason, this warning disappears when you click on it.

--- Regards, Sebastian VCP6.5-DCV // VCP7-CMA // vSAN 2017 Specialist Please mark this answer as 'helpful' or 'correct' if you think your question has been answered correctly.
Reply
0 Kudos
cypherx
Hot Shot
Hot Shot
Jump to solution

Dell has not released any bios version higher than the one I'm on.  I also used VUM to get ESXi 6 to the latest patch, so that takes care of the Hypervisor.  The Operating systems on individual VM's should not matter, because it appeared once the machine rebooted with the latest vmware patch, therefore no VM's were yet running on them.

I clicked on the __MISSING RESOURCE__ hyperlink in the right of the yellow warning message in vSphere web client and the warning has never reoccurred.

Reply
0 Kudos
sebcrj92
Contributor
Contributor
Jump to solution

Hi,

I did the same, but after the reboot, I saw in the summary tab that hyperthreading was inactive and number of logical processor was divided by 2 obviously.

Does this option desactivate hyperthreading on the ESX side even if it's activated in the BIOS ?

Reply
0 Kudos