Can't enable the EVC on cluster where the vCenter VM is part of the cluster

[adam900331]

Hy!

I have a vCenter 7.0u3 which contain 3 ESXi host. The hosts is still 6.7u3, later I will upgrade to latest 7.0u3. The vCenter contain a cluster and 2 ESXi host parts of this cluster. The vCenter VM is running on one of this ESXi host. The third ESXi is only datacenter member, not part cluster, because this host has different generation CPU. This is the exactly CPU version:

Cluster member:

- HostA: Intel Xeon Gold 5317 CPU @ 3.00GHz

- HostB: Intel Xeon Gold 5317 CPU @ 3.00GHz

Standalone, datacenter member only:

- HostC: Intel Xeon CPU E5-2667 v4 @ 3.20GHz

I would like to add the HostC ESXi server to the cluster to available migrate VMs online between the hosts, so firstly I would like to enable the EVC mode on the cluster in yesterday. I powered off all VMs which are running on the 2 cluster hosts (HostA and HostB), except the vCenter VM. (If I powered off the vCneter too, I didn't enable the EVC right 🙂 )

I checked the CPUs compatibility and knowing this, I decided to select the Broadwell generation, because this is the highest supported version on HostC:

- HostA and HostB:

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=cpu&productid=147&deviceCat...

- HostC:

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=cpu&productid=111&deviceCat...

So, I selected the Broadwell generation, and got the following error message:

"The host cannot be admitted to the cluster's current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode: HostB"

Yes, that is true, the vCenter VM is running on HostB. But what can I do? If I power off the vCenter, I cannot do nothing without vCenter, so it seems to be a dead end...

Please help me, and suggest possible solution to enable EVC on cluster. If I enable the EVC on cluster, I will add the HostC to the cluster.

I checked the vCenter VM EVC CPU Mode on the HostB->VMs menu, and show: Intel "Cascade Lake" Generation. After the unsuccessfull EVC configuration I powered on all VMs and checked the VMs EVC CPU Mode. I attached a photo. The EVC CPU Mode in VMs is higher then I want to set in cluster mode (Broadwell). Why show different EVC CPU mode, because the EVC is not enabled...

Thanks for the help.

[hirschinho]

Hi @adam900331 

You can try a workaround, it has worked for me in the past.

Create a new empty cluster - enable EVC in Broadwell, disconnect host and reconnect to new cluster.

[adam900331]

Hy!

Are you thinking about this workaround?

https://kb.vmware.com/s/article/2147821

[RajeevVCP4]

You required disabled or enabled EVC on vCenter VM I mean we can do it on vm level once it done you can try on cluster level

https://virtuallyvtrue.com/2019/12/16/vsphere-6-7-what-is-per-vm-evc-and-how-to-configure-manage/

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-A1C095EF-1B0F-4C1...

[hirschinho]

Hy,

I would do it not exactly the KB way.

In my opinion i would to the following steps:

1. Create new cluster / Disable HA in old cluster

2. Enable EVC Mode / Disable HA

3. Power Off all possible VMs

4. Enter maintenance mode Host C (CPU E5) / Drag and Drop Host to new cluster

5. Enter maintenance mode Host A (Gold CPU) / Drag and Drop Host to new cluster

6. Move all Powered Off VMs to cluster with Migrate Option

7. Power on all VMs if you want minimize downtime.

8. Power down vCenter VM on Host B / directly connect to Host B / remove from inventory

9. Connect to Host A directly / register vCenter VM and power on.

10. Enter maintenance mode Host B (Gold CPU) / Drag and Drop Host to new cluster

Finish. Remove old Cluster.

[adam900331]

Hy RajeevVCP4

There isn't VMware EVC option on vCenter VM, so I can't confgiure it. See atteched photo.

[RajeevVCP4]

This is expected behavior

https://kb.vmware.com/s/article/59801

[adam900331]

On the HostB see the following EVC CPU mode (attached photo). The Per-VM EVC mode and the cluster EVC is disabled. The HostB has the following CPU:
Intel Xeon Gold 5317 CPU @ 3.00GHz

The supported EVC mode is: IceLake

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=cpu&productid=147&deviceCat...

But I can see some VMs which has lower CPU generation, for example: Broadwell and Cascade Lake

The HostC CPU is: Intel Xeon CPU E5-2667 v4 @ 3.20GHz, the highest supported EVC mode is Broadwell.

So I would set the cluster EVC is Broadwell generation. When I enable it, I won't able to migrate the VMs which has Cascade Lake and IceLake generation??? Refer this sentence: "The EVC setting that is shown when a VM is powered on is what EVC mode is the minimum the VM can enter."

Thanks.

[hirschinho]

In my latest post I explained possible way to move the VMs, and yes for sure online migration is not possible, you have to power down vms and move it.

[adam900331]

Ok, but after move the VMs in powered off state, and power on in cluster, and set the EVC mode Broadwell in cluster, all of the VMs EVC moe will be Broadwell generation?

[hirschinho]

yes correct.

[adam900331]

Hy hirschinho

I don't see any other solution, so I will try Your suggestion. If you already tried it, and works well, then It should be work with me. 🙂 I will give feedback.

[adam900331]

Hy hirschinho

I have another question about EVC. As I said above, the EVC on the cluster settings and the per VM EVC isn't enabled. In vCenter console I see that vCenter VM EVC CPU Mode is Cascade Lake generation.

I checked this VMware website and it say, that the EVC CPU mode is determined by the parent host:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-77A0EE88-779E-...

But the vCenter server VM is on HostB which has the following CPU model:

Intel Xeon Gold 5317 CPU @ 3.00GHz

This CPU model generation is Ice Lake. So what determine the vCenter VM EVC CPU mode? Why should be the Cascade Lake CPU mode and not Ice Lake. Ice Lake is highest generation like the Cascade Lake.

Another question can I modify the EVC CPU generation in vCenter VM? The is no VMware EVC menu in VMs configure.

Thanks.

[hirschinho]

Hi @adam900331 

Your Hosts are on 6.7 so update for Host to 7.x and also upgrade VM Hardware Version - than you can set EVC per VM.

[adam900331]

Thanks, but what defines, the VM EVC CPU mode in this situation?

As I said above, the EVC on the cluster settings and the per VM EVC isn't enabled. In vCenter console I see that vCenter VM EVC CPU Mode is Cascade Lake generation.

I checked this VMware website and it say, that the EVC CPU mode is determined by the parent host:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-77A0EE88-779E-...

But the vCenter server VM is on HostB which has the following CPU model:

Intel Xeon Gold 5317 CPU @ 3.00GHz

This CPU model generation is Ice Lake. So what determine the vCenter VM EVC CPU mode? Why should be the Cascade Lake CPU mode and not Ice Lake. Ice Lake is highest generation like the Cascade Lake.

Thanks.

[a_p_]

A VMs EVC mode not only depends on the hardware (the physical CPU), but also on the configured virtual hardware version (compatibility mode). See e.g. https://kb.vmware.com/s/article/1005764 for supported EVC modes. That said, please don't change the vCenter Server's hardware version!!

If you are still looking for a way to move the vCSA into a lower EVC mode cluster, either do this by unregistering the vCSA from the current host, and registering it on a host in the EVC enabled cluster. This of course requires downtime, and shared storage.

If you don't have shared storage, you can create a dummy VM in the EVC enabled cluster, and configure it with a per-VM EVC mode. This will add the required CPU settings to the VM's configuration (.vmx) file. Then shut down the vCSA, and add these additional settings to the vCSA's configuration file. After that, reload the vCSA (see https://kb.vmware.com/s/article/1026043) and power it on. It should now run with in the EVC mode that allows you to migrate it to the cluster.

Please consider to backup the vCSA's configuration (.vmx) file before making the changes!

André

[Kinnison]

Hi,

I apologize, comment off topic and therefore edited in this way, if I could I would have deleted it right away.

Regards.

[adam900331 · Accepted solution]

Hy!

I used this method without create another cluster:

1. Turn off all VMs on the cluster member hosts expect vCenter VM.

2. Log in to the ESXi directly where the vCenter VMs is located (HostB).

3. Turn off the vCenter VM and unregister from inventory.

4. Log in directly to the non-cluster member ESXi (HostC) and register the vCenter VM, and powered on.

5. Log in back to vCenter and now I could turn on the EVC with Broadwell generation successfully, because the cluster didn't contain powered on VMs.

6. Turn off the vCenter VM in HostC unregister from inventory and register it to the one of the cluster member ESXi, for example HostA.

7. Log back to vCenter, turn off all VMs on HostC, enter to amintenance mode and Move to the cluster.

8. Finally turn on all VMs and check the online migration between the HostC and HostA, HostB. (different CPU).