EVC and HPE Gen10 Plus

[mtidgewell]

We have a cluster with three HPE DL360 Gen10 hosts, it contains one host with an Intel 5100 series CPU and two with an Intel 5200 series CPU.  We have the EVC level set to Skylake and everything is fine.  We are trying to add a DL360 Gen10 Plus with an Intel Gold 6338 cpu but it fails stating the host's hardware does not support the cluster's current Enhanced vMotion Compatibility mode.  The hosts are running 6.7 U3 with all the patches and the vcenter is is running the latest version of 7.0 U2 code.  Even if we make a new cluster with evc enabled and put the Gen10 Plus host into it the highest level we can set the EVC to is Haswell.  Is there some BiOS setting we missed or is the VMWare compatibility guide incorrect, because it clearly list SkyLake as a supported EVC level for the this server/CPU combination.

[Techstarts]

ESXi version matters in your case. You have nothing wrong configured. After you upgrade ESXi then you get this baseline.

[mtidgewell]

Spent an hour on a support call for this yesterday, they said there is an unresolved issue with any EVC over Sky Lake.  They even forwarded a KB article from 2018.  IceLake was not mentioned in the KB and was only released Q2 2021.  The solution (or lack of solution) was to set the EVC level on the cluster to Broadwell.  Are you suggesting that an upgrade of esxi to 7.0.2 will allow me to move my EVC back to SkyLake?  If so, is there a document from VMWare that states this?  Clearly the compatibility guide  has an error in it because clearly states 6.7 u3 supports this CPU and EVC levels.  It would be nice if thee was a footnote stating vCenter 7.0 U2 and esxi 7/0 u2 are required.

[Techstarts]

Sorry. I was wrong. It should work. I checked and found the all Haswell, Broadwell, Skylane should be possible.

With ESXi version you get more EVC mode but in combination of vCenter version. You are already at highest vCenter version.

I assumed you are referring to Gold 5100/6100 series

[mtidgewell]

The error in the compatibility matrix is exactly what I'm pointing out, it clearly states the 6300 which is IceLake supports SkyLake (6100) EVC level and for my host running a 6338 on an HPe Gen10 plus with esxi 6.7 u3, but it does not.  Still waiting for an answer from VMware to see if this is a corner case for just this CPU or a bug in esxi.

[Leomhann]

I have found this exact issue.  Not specific to HPE but to Ice Lake processors in general.  In 2019 (I think) Intel announced that it is deprecating the MPX feature in it's processors.  I believe it is still present in Cascade Lake, but clearly not in Ice lake.  From the VMware side, Skylake is the only mode that requires MPX and therefore Ice Lake is not compatible with that mode.  The compatibility matrix is clearly wrong.  So, if you are running 6.7, the highest compatibility mode allowed is Skylake.  We have a customer with Cascade Lake based hosts running at Skylake...they will not be able to introduce Ice Lake into those clusters.

After upgrading to 7.0 they will be able to raise the EVC level to Cascade Lake and that will allow Ice Lake hosts to be added to the cluster, but there is still the issue of vMotioning VMs to those new hosts.  They will be running at Skylake (per VM mode) and would have to be powered down and back up in order to elevate to Cascade Lake for true compatibility.  

VMware has stated that MPX will no longer be exposed to VMs starting with ESXi 6.7 P02 and ESXi 7.0 GA as per KB76799.  I have to confirm, but the issue here is with the EVC Mode definition.  MPX is still required and is masked in Skylake mode in both vCenter 6.7.0.48000 & 7.0.2.00200 as per the evcModes.xml file.  I have confirmed this in each.

In KB76155 it states "Memory protection extensions (MPX) were introduced in Skylake and provided hardware support for bound checking. However, Intel began removing the MPX feature beginning with Ice Lake CPUs. As such, while this feature continues to be supported, it will not be exposed by default to virtual machines at power-on and is not included at the EVC baseline level."

I believe this statement in reference to VMs, not Hosts as the MPX requirement for Skylake Mode still exists.  So what is VMware's guidance on adding Ice Lake hosts?  There is clearly an issue here and needs to be addressed formally.

[Leomhann]

I also have a case submitted with Support on this issue.  They informed me that this will be addressed in 6.7 P06.  No ETA on availability however.  I have inquired on a fix for 7.0u3 as well.

[mtidgewell]

Here is the final response I received from support.

Our engineering team has provided an update with the Skylake support on EVC.
They have certain code changes after which we see the support being at 6.7P06 for Skylake EVC .
We will be publishing the KB article regarding the update soon with the support on 6.7P06.

Tentatively we have the patch releasing end of October or the 1st week of November.

The support on 7.0U2 should not have any issues with the EVC.

[Leomhann]

I have verified in a lab that the issue IS present in 7.0u3 and I have informed support of this and am awaiting a reply.  So, I am not sure how they can say there is no issue in 7.0u2.  The evcModes.xml file from vCenter 7.0u3 still has the MPX feature listed for Skylake.

The fact they are stating a patch for ESXi is going to fix this issue concerns me as EVC is a function of vCenter.  I do not think the Support teams understand the issue fully.  I am going to press for a beta release of 6.7 P06 for my own validation if I can.  If not I am requesting this get escalated.

[mtidgewell]

I agree with you, it is a vCenter problem, but I will have now to test this until I sell another Gen 10+ host because the one I opened this case on is set to Haswell and there it will stay.  With the incredibly long lead times on servers it might be 2022 before I can test.

[Leomhann]

I have a few servers that (hopefully) I don't need to ship out right away.  I will be setting up a test with them, partially for testing my own resolution, and for demonstrating to VMware how their product works.  Luckily I also have 1 Skylake and 1 Cascade Lake server in addition to several Ice Lake servers.  I will gladly share results.  If you are interested in what my proposed fix is, please send a private message...I do not want to post an unsupported process publicly.

[mtidgewell]

The simplest test I found was to set a cluster to evc level SkyLake and try to drop a host with a SkyLake into it.  I assume your work around is to edit the EVC file, if that is it, thanks, but if someone else takes over servicing the account (something that happens too much) and upgrades the vCenter, the risk of breaking something too high.  I will just have to wait until VMWare delivers a fix, or leave evc at Hasswell.

[Leomhann]

Yes, and I agree.  But I still want this tested as I'm sure I will have other customers come across this eventually and I need to understand the impact of any such changes whether they be official or not.  I will deploy some test VMs at skylake with default evc definitions and then add in a new ice lake host using modified definitions.  If, as VMware states, MPX is no longer exposed, then a vmotion should succeed without any impact to VMs.

[mark49808]

@mtidgewell @Leomhann did you have any updates for this? or a support case number i can reference to expedite the process with vmware support?

[mtidgewell]

They say the update will be released soon.  VMware vCenter Server 7.0 - SR 21246641408

[Leomhann]

I have received no updates.  SR 21253785509

[Leomhann]

Was informed that a patch for 6.7 would be available soon.  Possibly late November