VMware Cloud Community
asg2ki
Enthusiast
Enthusiast

ESXi 6.7 Unsupported CPU - Ideas for workarounds

Hi All,

With the recent release of vSphere 6.7 many of us who built home lab environments just hit a dead end on the upgrades due to CPU incompatibilities. Taking the fact that some of us have built quite extensive versions of the labs with rather expensive equipment, it makes a situation even more difficult, since if we would like to upgrade all the back-end equipment then this would require tremendous financial support which we usually can't get out of nowhere. So I was thinking about some potential workarounds with hopes that some of you may pop into this discussion and provide additional ideas.

In my lab I have various types of machines some of which are fully supported at this time for running ESXi 6.7 but the question of "until when ?" is always out there. So far I managed to run most of my hardware up till ESXi 6.5 without any issues regardless of the warning message that my CPU's are falling into the scope of unsupported stuff, but oh well we never care about such warnings when it comes to home labs right ? Smiley Happy

Anyway so my stuff is organized as follow. I have one ESXi Cluster based on Xeon E5-2620 CPU's which runs absolutely fine with ESXi 6.7, however I have another ESXi Cluster based on older Intel Core i7-950 CPU's and I'm using it for testing some vCloud Director related VCD features. Unfortunately this second cluster definitely doesn't fall into the scope of upgrading it to ESXi 6.7 and for my worst luck it just refused to boot up when I tried to enforce it to v6.7 via the offline-depot upgrading method. The machines were always getting back to the black screen stating that my CPU type is unsupported.

Personally I'm not very happy with VMware's decision on completely dropping out the older CPU's because this seriously limits the options for home labs and frankly I don't even think we usually test the most complex features requiring the additional hardware support. Of course I agree to some extend with the decision to limit support for extremely old machines and especially when it comes to production environment, but on the contrary I have another very old ESXi cluster based on Intel Core 2 Quad Q9550 CPU's which still totally works with ESXi 6.5 and I'm using it occasionally for testing out vCloud Edge related services. So I found it kinda unnecessary that VMware went too drastic on this decision instead of letting the home-lab users to test out on their own equipment what may or may not work. Anyway...

So in my case upgrading the hardware from scratch on at least two cluster of 3 machines to new generation is totally going to drain my finances and therefore I started to look for some workarounds to this. My main idea here was to try the nested-virtualization method by installing ESXi 6.7 on the top of ESXi 6.5 layer and enforce the VM via CPUID modifications to actually think that it does have a supported CPU class underneath. I took the approach by making a test VM on one of my Intel Core i7-950 CPU based machines (still running ESXi 6.5). When I boot the VM and try to install ESXi 6.7 the obvious result was this one:

pastedImage_2.png

At this point I decided to play around a bit with the EAX registers and implement a mask that would enforce ESXi 6.7 to think the VM has a different CPU model and stepping. Since my knowledge on CPUID implementation is somewhat limited I just decided to take the EAX register from one of my Intel E5-2620 based hosts instead:

pastedImage_4.png

Once I applied the mask, I was able to install ESXi 6.7 without any issues. I didn't even have to rebrand the CPU via the 0x80000002 registers in order to be able to install the new OS, so it seems that ESXi 6.7 is primarily looking at the CPU Family ID, model and stepping for the initial CPU identification procedure.

pastedImage_3.png

While this method is not ideal at all (in fact it's a very ugly workaround) at least it provided the option to install the new OS and therefore to put hands on the basic functionality. Not sure if this method is suitable for nested virtualization, but if I get some time I'll definitely test it out. I suppose the EAX registers could be tweaked a bit better in order to be able to support certain set of features, so if anyone has a good idea on how to craft a better CPUID EAX registers purely for lab uses with as much compatibility on features as possible, feel free to put your magic in this thread.

Any other workarounds on running ESXi 6.7 on unsupported CPU's are also very much welcomed.

19 Replies
bluefirestorm
Champion
Champion

I suppose the EAX registers could be tweaked a bit better in order to be able to support certain set of features

What you have done with CPUID leaf 1 EAX register is just to change the Family/Model/Stepping (in essence the Intel CPU product ID) and that's about it and just enough to get past installation. Specific features are found in other CPUID leafs and registers (EAX, EBX, ECX, EDX) and Model Specific Registers (MSRs). No amount of masking will make up for what is missing from the actual silicon. We can mask Family/Model/Stepping to be a Skylake Xeon Platinum but it won't make any difference to the features that the actual CPU you have can provide.

Most installs rarely check for CPU features and if does, it usually make the judgment to proceed/abort based on the Family/Model/Stepping. Some OS/applications only check during runtime so it can at least give an error message; and those that don't check for feature available likely just crash badly. For example, the i7-950 is a Nehalem Bloomfield CPU; so it does not have the AES (Westmere), AVX (Sandy Bridge), AVX2 (Haswell) instruction set. It also does not have some virtualisation specific features such as PAUSE-loop exiting (Westmere), virtual interrupt delivery (Ivy Bridge), nested paging access/dirty bits (Haswell), etc.

So if either ESXi 6.7 itself or the guest OS VM inside it starts looking for advanced CPU features and does not handle it well, you might just start singing "purple screen, purple screen". :smileylaugh:

Reply
0 Kudos
asg2ki
Enthusiast
Enthusiast

Thanks for the detailed clarification. Of course I had no illusions at all that we can somehow emulate certain features that are missing from the physical hardware but in any case staying on the latest ESXi version may have other useful advantages like for example testing out the latest v6.6 Distributed Switch which is not compatible with previous ESXi versions. There are tons of scenarios why ESXi 6.7 would be desired for older home-lab configurations. Obviously upgrading hardware is becoming inevitable at some point in time like when we had to move from x86 to x64 Smiley Happy

In any case I still find the ESXi 6.7 installation cut-off step a bit too drastic and I hope VMware might reconsider its decision or perhaps they could release some sort of a community version for the home-labs.

Reply
0 Kudos
asg2ki
Enthusiast
Enthusiast

So I made couple of more tests in my lab and tried to run some VM's via nested virtualization over the virtualized ESXi 6.7 with the modified CPUID's. Unfortunately this have miserably failed with the message that the host does not support the "VMX Unrestricted Guest" feature which I guess is something coming out from the tweaked CPU model and stepping parameters but I might be wrong on this one.

I tested the same scenario with un-tweaked virtualized ESXi 6.5 image and the nested virtualization works just fine. I was able to run a VM inside that virtual ESXi 6.5 just as expected. To my biggest surprise when I tweaked the CPUID of the ESXi 6.5 VM with exactly the same E5-2620 based EAX register, the nested VM was also able to start properly,. This means that ESXi v6.7 by default has much granular check on the CPU supported features as compared to ESXi v6.5, but I might be hitting some sort of a bug as well. Not exactly sure what ESXi 6.7 is looking for when I'm starting the nested VM but it definitely behaves in a different way than ESXi 6.5.

Maybe I'll try to run the virtualized ESXi with a different CPUID EAX register to check if the results would be any different.

Reply
0 Kudos
bluefirestorm
Champion
Champion

VMX Unrestricted Guest is a virtualisation feature that was introduced in Westmere CPUs. Since the i7-950 is a Nehalem Bloomfield CPU (one generation before Westmere), it does NOT have the VMX unrestricted guest feature. Workstation 14.x/Fusion 10.x already requires VMX Unrestricted Guest.

As I have said earlier, the CPUID leaf 1 EAX register is just the family/model/stepping and it has nothing to do with what features is available/visible. No amount of tweaking of CPUID leaf 1 EAX can make up for what is not available in the silicon.

VMware software (ESXi, Workstation Pro, Fusion, etc) checks the specific features of CPUs through CPUID instruction call using the different leaf IDs and register output as well as looking at the MSRs. For example, VMX Unrestricted Guest is MSR 0x485 bit 5. I don't think VMware has masking facility for MSRs. Even if it does, fooling the ESXi VM that a CPU feature is present while it is not available in the actual silicon will just lead to problems.

Reply
0 Kudos
asg2ki
Enthusiast
Enthusiast

I think you misunderstood the situation. I'm not trying to emulate the VMX Unrestricted Guest feature on my old CPU. What I'm trying to do is to make a workaround for running ESXi 6.7 on my older CPU which I'm pretty sure it is totally capable of for the basic features like simply running couple of VM's. Furthermore my old CPU is totally capable of running nested virtualization as it does include all basic prerequisites such as VT-x and I've been using that feature many times already.

As I stated in my last post this VMX Unrestricted Guest failure comes up only on ESXi 6.7 for some reasons. The same issue doesn't happen with ESXi 6.5 even when I set the EAX register to something else. By my best guess this means that the VMX Unrestricted Guest feature was made a mandatory requirement with the ESXi 6.7 code for running VM's.

Reply
0 Kudos
bluefirestorm
Champion
Champion

I am not misunderstanding your situation.

VMX Unrestricted Guest is a virtualisation feature in Intel CPUs that are Westmere and later (although there some exceptions with some Atoms and 2014 Pentiums).

From the error that you encountered, it looks like that ESXi 6.7 requires VMX Unrestricted Guest feature to powerup/run VMs (just like Workstation 14.x on Windows/Linux and Fusion 10.x on macOS). Therefore if the underlying CPU does not have that feature, it will fail. At the very least ESXi 6.7 checks for the availability of the actual feature instead of relying on the spoofed Sandy Bridge family/model/stepping that you applied and that is why you are getting that error message.

The CPUID leaf 1 EAX register is just an Intel product ID in the eyes of the VMware hypervisor software (ESXi, Workstation/Player, Fusion). It does not determine actual CPU features it finds/detects.

ESXi 6.5 still supports Nehalem CPUs (which does not have the VMX Unrestricted Guest feature) and that's why you don't get that error on ESXi 6.5.

You can see it in the VMware Compatibility Guide.

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=cpu

Reply
0 Kudos
asg2ki
Enthusiast
Enthusiast

Yup, that totally makes sense and thanks again for the clarification. This means that there is literally no way of getting around this problem due to the hard coded restrictions unless VMware would decide to put back in the compatibility in ESX 6.7 for older CPU's. Still feels unjustified why they dumped the older CPU's and leaving tons of labs without much of a choice but HW upgrade. Of course I would totally agree that nobody should be running production workload on unsupported configurations (I suppose many people do that too) but that's a whole different story anyway.

So let's hope that we will see a revised ESXi 6.7 release with less restrictions. In the meantime I'll definitely would need to look for new motherboards, CPU's and RAM especially since nested virtualization anyway wouldn't have been ideal solution even if it worked.

Oh dear... here we go again on another lab building adventure Smiley Happy

Reply
0 Kudos
compevo
Contributor
Contributor

I completely agree I think VMWare is ultimately going to lose marketshare over this one.  There is a lot of expensive and perfectly good hardware that can't run VMWare anymore yet the same hardware will probably be able to work forever with other opensource virtualization solutions.

Trying to get 6.7 up and running is definitely a test and makes me worry that shortly after that support for whatever is currently supported will be dropped in the future.

Cheers
Reply
0 Kudos
uhlered
Contributor
Contributor

What's the other product?  KVM, Microsoft ?  Hmmm.

Reply
0 Kudos
bkraul
Enthusiast
Enthusiast

There are several now. I actually experienced this very thing this morning. Disappointing. Already disappointed at the workstation level as well. Looking into ProxMox. Seems like it could make the cut.

Reply
0 Kudos
ODB2001
Enthusiast
Enthusiast

Hoping someone may know the eax code I need but I'm trying to nest 6.7 as a VSAN witness on a HP Microserver N54L (which has a AMD Turion N54L CPU) and it's not booting with the error about unsupported CPU's.

I tried the eax code on this thread but didn't make any difference, suspect due to having an AMD CPU and not intel?

This is just a home lab environment, so not worrying about it being supported.

Thanks!

Reply
0 Kudos
bluefirestorm
Champion
Champion

It doesn't work because the CPUID in this thread is for Intel Sandy Bridge.

You could try this to mask it to look like an Bulldozer Opteron 3200.

cpuid.1.eax = "0000:0000:0110:0000:0000:1111:0001:0000"

I do not have any AMD CPU and have no idea if there is an AMD equivalent to the Intel VMX Unrestricted Guest. But the same concept applies for AMD CPUs, changing the CPUID leaf 1 EAX is just changing the family/model/stepping and does NOT change the actual CPU capabilities. It might get you past installation but not necessarily be able to run with any VM within the ESXi 6.7 VM when the actual CPU virtualisation capabilities will be needed/invoked.

ODB2001
Enthusiast
Enthusiast

Hi, you're a lifesaver, thank you Smiley Happy

That eax worked and it still reports the CPU on the host correctly.

Not worried about CPU capabilities etc as not running any VM's on it.

pastedImage_1.png

Reply
0 Kudos
wayne4186
Contributor
Contributor

asg2ki thats a great workaround and I have been chasing my tail all day trying to figure out how to upgrade my system running an AMD CPU that made it on the unsupported CPU list and hence I can not upgrade from 6.0.0u2 to 6.7.  Your workaround is for it to run in a nested environment so I am guessing that your underlying ESXi is still running 6.0.0 and you have installed a VM guest on this running ESXi 6.7 am I right?

Do you know of any way to tell the install to bog off and upgrade the underlying ESXi to 6.7 and mask the CPU during the install that is directly installed on the hardware so I don't have to run a nested environment which will be very ugly for my lab?

Reply
0 Kudos
meoli
Enthusiast
Enthusiast

many thx for this method!

I agree - building yourself homelabs where you actually have some stuff running e.g. mediacenter, mail, nas and stuff, or for learning, designing, evaluating a whole setup with backup-vms, replication, databases,, and so on running. It really hurts the bank account.

I'm actually glad that i can run ESXi 6.7 with on my Ryzen and even use the same desktop with Passtroughed USB-PCIe Controller and my RX480 to my Displays with really good FPS (It is a virtual machine!)

Unfortunately a 500GB Samsung SSD isn't seen anymore Smiley Sad and the VMFS from another 1TB SSD couldn't be recognized..

but well - its unsupported so thats to be expected and therefore i reformatted the 1TB drive - the 500GB SSD doesn't matter now :smileysilly:

I've got 2 small AMD Liano FM1 CPUs here and gonna try to look which ESXi Version is running 🙂 I pretty much won't believe that i even can start the setup of 6.7 which means i need to create an old cluster making learning new things harder.

I'm looking forward for the arm esxi so that i can actually try some really small vsan environments out, because the whole setup is perfect for making replication and other Applications evaluations and trainings.

But for the above Stuff i want power (from where i do hope that Threadripper will run on ESXi 6.7).

It would be really great if there would be a VIB or something else which comes with a huge disclaimer that says: "This Installation is unsupported and comes with no warranty or whatsoever" and no restrictions hardware size.

If you want, include a hardware collection tool which we can trigger manually (be nice) -->  community based Verification List @vmware <-- and that sounds cool! 😉

Also - would it be somehow possible to rewrite the Product ID and other related Codes in the BIOS? Or does the ESXi or media gets it directly from the CPU?

Does the setup verificates it trough a list which could be changed?

Edit:

Allowing Unsupported CPU’s on ESXI 6.7 | Home | Tech Adventures
https://commander614.wixsite.com/website/single-post/2018/07/08/Allowing-Unsupported-CPU%E2%80%99s-o...

On my FM1 CPU ESX 6.5 is running - so far so good, now i need Network :smileysilly:

Reply
0 Kudos
cowswillbecows
Contributor
Contributor

I recently purchased a third used dell r610, expecting to be able to install esxi 6.7.0 (as I had on two other 610s) but encountered the unsupported CPU message.  In this case the message states that the machine is running  Intel family 0x86, model 0x1e stepping  0x5  Intel(R)  Xeon(R) CPU X5570 @ 2.93GHz

When I look at the compatibility matrix  at http://www.vmware.com/resources/compatibility for information on CPU Series, and for "product release version" select ALL, and for CPU Series select Intel Xeon 55xx series, the tool tells me that this CPU is compatible with versions of vmware from 5.5 to 6.5U3.   6.7 does not appear in the list. Indeed, 6.7 won't install.  That's disappointing but makes sense.

So I wondered what the CPUs are in the other two machines here that are running 6.7.

Exploring this, I learned that my number two r610, which reports that it is running esxi 6.7.0 has as its CPU set 2 Xeon 5650 @ 2.6hGHz .  Again consulting the compatibility matrix for Intel Xeon 56xx processors, the tool tells me that this CPU is compatible with versions of vmware from 5.5 to 6.5U3 -- the same outcome as for the 5570 CPU.   So 6.7 is unsupported, Yet, it is a reality that 6.7 is installed and operating on the machine with 5650 CPUs.

So this introduces some uncertainty about what steps I might take to replace the CPU set in the number 3 machine that won't install esxi 6.7.  I'm hoping that I might be able to replace the CPUs in the third r610.

I'm just an academic struggling to keep a not-for-profit's servers online, not a hardware person, and am pretty new to vmware.  I would be immensely grateful for insight and suggestions.

Thanks!

Reply
0 Kudos
VN101
Contributor
Contributor

Another home lab user here that ran into this problem last night. I chased my tail all day until I found this thread. Sure would be nice if VMware would make such significant changes in major version releases not minor (i.e. ESXi 7, not 6.7) like most reasonable companies do. The result certainly is going to disappoint and upset a lot of the user base. But when you have such a large following, who cares about upsetting 5% really? I'm sure they polled the data and determined the number of users who would be disrupted by this and therefore SOL and stuck at 6.5. Very frustrating indeed.

Thank you to the OP for starting this thread and opening a dialog for users to chime in about the incompatibility of their CPU's with ESXi 6.7 - I'm the exact same CPU even with an Intel i7 950, loaded to the max with RAM when I had the $$. Now, they've forced a major decision and that's a shame really.

Reply
0 Kudos
bleri3
Contributor
Contributor

Hi

I know this post is somehow old, but I was able to use another workaround:

I added the allowLegacyCPU=true parameter to the BOOT.CFG file which bypasses the CPU check.

allowLegacyCPU.PNG

When you run the installer you need to manually add using the Shift+O. But after the installation is completed, adding the parameter to boot.cfg works well, at least for me. I was able to upgrade my 2 old servers from 6.5 to 6.7 and even to 7.0.0 but ran into a roadblock on which the network card drivers are needed. This is my next challenge.

Anyway, nicely found workaround!

Oxenrider
Contributor
Contributor

how did you get past the network card driver problem?

Reply
0 Kudos