Enable / Disable AES-NI instruction set in VMWare Workstation Client

[netlib]

This is a longshot, but here goes.

I am using VMWare Workstation 14.1.3 on Windows 10 x64 with a Windows 10 x64 Client VM.  The host machine (Dell Precision 7510) has a newer processor that supports the AES-NI instruction set. That seems to propagate to the Client machines which also support the AES-NI instruction set.  Is there any way to disable the AES-NI instruction set on the Client VM (for testing purposes)?  This is usually found in the Bios, but when I went into the Setup for the Client VM, the setting for AES-NI support was nowhere to be found.  Any suggestions?  Thanks.

[bluefirestorm]

You could mask it out in the vmx configuration file.

Shut down the VM and add the following line:

featMask.vm.cpuid.aes = "Val:0"

AES-NI has been around (I think) since Westmere although the speeds of the instrucitons between CPU generations are different.

[netlib]

Thanks for the suggestion.  I tried adding the line for a VMX file to a Windows 2012 R2 VM and got this popup:

Feature 'cpuid.AES' was 0x1, but must be 0.

Module FeatureCompatLate power on failed.

Failed to start the virtual machine.

I tried different variations on it, for example

featMask.vm.cpuid.aes = "0"

featMask.vm.cpuid.aes = 0

featMask.vm.cpu.aes = "Val:0"

I did not complain about any of them, but AES-NI was still enabled.

[bluefirestorm · Accepted solution]

You could try the bit field masking method on CPUID.

cpuid.1.ecx = "----:--0-:----:----:----:----:----:----"

The featMask is the more recent method. The cpuid bit masking is the "old" way and I have to look up the Intel manual to find out the leaf/register/bit where a particular CPU feature is indicated.

On the Linux VM (on Fusion 11), with AES masked out using either method, the "aes" does not appear in the /proc/cpuinfo flags. But once you comment out the masking, "aes" appears again the /proc/cpuinfo. I used a Linux VM as the flags is something more comprehensive compared to something like CPU-Z on Windows.

Since you said it didn't work for you, I also tried both technique separately on a Windows 10 1803 VM on Workstation Pro 15 with a Windows 10 host; and the "AES" disappears from CPU-Z utility "Instruction" section.

Both techniques worked fine even with Workstation 12.x and Fusion 8.x in the past. It should work also even for an ESXi VM. That is essentially what EVC masking is about.

Remember the VM has to be shut down (not suspended). Otherwise I can't think of any reason why it wouldn't work unless Windows 2012 R2 requires "AES" or at least it could not be masked out once you install it with AES active. I don't have any Windows 2012 R2 VM.

[bluefirestorm]

If the CPUID bit masking also does not work, maybe there is already some other mask that is already present and conflicts with it.

If the VM running on the Workstation Pro was downloaded or imported from an ESXi host that has EVC masking (e.g. Westmere EVC compatibility), the vmx configuration might still have remnants of EVC masking that will conflict with the standalone masking.

You can attach the vmx configuration file if you want another set of eyes to look at it.

[netlib]

Yes, the VM was powered off each time.

The CPU masking worked! Thanks.  I added the line you suggested:

cpuid.1.ecx = "----:--0-:----:----:----:----:----:----"

I have attached the entire VMX file so maybe you can tell why the first strategy did not work.

Thanks again!

[bluefirestorm]

It looks like it is a bug of VMware Workstation/Player 14.x. That is if you could call it a bug, since these configuration entries are not in any official VMware documentation.

Nothing obvious in the vmx file that you attached indicated anything wrong. So I then tried it on a system where I still had VMware Player 14.1.3. And I got the same error that you did.

| vmx| I125: [msg.featurecompat.requirement.number.mismatch] Feature 'cpuid.AES' was 0x1, but must be 0.
| vmx| I125: [msg.moduletable.powerOnFailed] Module 'FeatureCompatLate' power on failed.
| vmx| I125: [msg.vmx.poweron.failed] Failed to start the virtual machine.

But the

featMask.vm.cpuid.<cpu feature> = "Val:0"

format is more understandable that the

cpuid.<leaf>.<register> = <bit mask>

In some ways the featMask method is self documenting, assuming one knows the format

featMask.vm.cpuid.aes = "Val:0"

easily shows that AES is masked out

whereas

cpuid.1.ecx = "----:--0-:----:----:----:----:----:----"

one would likely need to go through the Intel manual to know what is masked out.

For that it would be better to include some sort of comment either above or below the cpuid bitmask. Something like

# CPUID leaf 1 ecx register bit 25 mask out AES

It becomes some sort of note to self, some years down the line, or to anyone else.