TimMann Tracker

Re: VM Guest BIOS/UEFI Boot

TimMann — Thu, 20 Oct 2022 22:50:44 GMT

I don't know of any plan to do that. Obviously, part of the value of VMware software is being able to migrate old hardware installs to VMs and run the old operating system in the VM.

Re: Upgrade ESXI 7.0.1 - Fatal CPU mismatch

TimMann — Thu, 11 Mar 2021 22:38:32 GMT

This is actually a bug. Intel specifies only bits 52:50 of MSR 0x17 to be architectural. The other bits are reserved, so the vmkernel shouldn't insist that those bits be the same across all cpus. In the PSOD screen, only some of the lower-order bits differ. Bits 52:50 are zero in both.

Note that MSR 0x17 is not the microcode revision. It's the platform ID; see Intel documentation for details.

You can boot with cpuUniformityHardCheckPanic=FALSE to work around this.

Thanks to Valentin for alerting me to it.

Re: strange thermal issue after patching to ESXi 7.0b

TimMann — Wed, 15 Jul 2020 23:15:02 GMT

The next ESXi release will package 0x02006906. However, the fix from Supermicro might have been elsewhere in the BIOS -- it's possible that using 0x02006906 with an old BIOS might still cause the same problem, or it may not. That's unclear from what little information I have. So updating your BIOS to the latest from Supermicro is the best thing to do.

Re: strange thermal issue after patching to ESXi 7.0b

TimMann — Tue, 14 Jul 2020 00:00:29 GMT

Can you also file an SR with VMware if you're entitled to VMware support?

You can check the microcode version that ESXi loaded with this command from an ESXi console shell:

vsish -e get /hardware/cpu/cpuList/0 | grep Revision

"Original Revision" is what ESXi saw at boot time (usually loaded by the BIOS), and "Current Revision" is what's running now, possibly loaded by ESXi.

I've looked and can't find any microcode issues documented by Intel with symptoms similar to what's described in this thread. For Skylake-D, I believe the latest microcode that's publicly released as of today is 0x02006906. ESXi 7.0p01 includes the slightly older 0x02006901. Version 0x02000069 is still a little older.

You can try having ESXi not load any microcode by running this command and then doing a clean shutdown/reboot:

esxcli system settings kernel set -s microcodeUpdate -v false

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Sat, 16 Nov 2019 01:24:21 GMT

@Himilou: Your machine isn't on our compatibility list (VMware Compatibility Guide - System Search ), so our evaluation program doesn't really apply. However, you can try the free version of vSphere at Free vSphere Hypervisor | VMware . I checked, and if you register for the free version, you do get the option to download older releases like 6.7u2, not only 6.7u3. I hope that helps.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Fri, 15 Nov 2019 22:57:56 GMT

@Himilou: I've sent out some email internally to try to find who could help, perhaps by giving you access to 6.7u2.

In the meantime, you could try updating to the latest BIOS. From HP Z420 Workstation Software and Driver Downloads | HP® Customer Support , I see the latest is 03.95 Rev A, dated Jul 1, 2019. That shouldn't hurt, but of course may not help.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Fri, 15 Nov 2019 19:16:11 GMT

Just to be sure: did you try 6.7u3 on the HP machine and see the failure discussed in this thread? I haven't heard of the issue affecting an HP machine before, so just for our information, can you say what model it is, and the BIOS version?

As a development engineer, I'm not familiar with what we do and don't make available to evaluation customers or what support channels are available to them. I can try to find you some help, but if there is a support or sales support contact associated with your evaluation, that might be a better route. The best thing probably would be if support can let you run your evaluation with 6.7u2 for now.

6.7ep13 wasn't released with an ISO image; that's planned for the next regular patch (not express patch).

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Wed, 13 Nov 2019 01:38:33 GMT

Upgrade directly from 6.7u2 to the new patch in one step. Do not upgrade to 6.7u3 and then to the new patch as a second step.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Wed, 13 Nov 2019 00:42:16 GMT

VMware ESXi 6.7, Patch Release ESXi670-201911001 is out today and inclues the fix for this issue.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Fri, 08 Nov 2019 22:11:26 GMT

If you update directly from 6.7u2 to the upcoming patch that contains the fix, you'll never have the buggy bootloader installed and will never boot with it. I think that was your question.

Of course, if you have a machine that's already suffering from the bug (upgraded to 6.7u3 and won't boot), obviously you have to do something by hand to get out of that state. The easiest ways are (1) if possible, boot in legacy BIOS mode, or (2) boot an installer ISO from an unaffected release and use the installer to upgrade to it. I believe the upcoming patch will include an installer ISO. Patch releases normally don't, but I requested one for this patch to make it easier, so that you don't have to downgrade to 6.7u2 using its ISO and then upgrade to the patch.

In the procedure I gave for running 6.7u3 in spite of the bug, I suggested rEFIt exactly because it's old. rEFInd is newer, but that's not necessarily better, as it comes with a new enough EFI shell that it won't work on machines that are too old. The procedure is complex and has pitfalls, so I'm not surprised some people who tried it didn't get it to work. Fortunately it won't be needed anymore as soon as the new patch is out.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Fri, 08 Nov 2019 20:56:46 GMT

I said a patch that contains the fix is coming soon. That means it is not released yet, so no, you won't get the fix by upgrading to any currently released patch. Also the contents of already released patches or updates will not be retroactively changed. You will get the fix by upgrading to the patch that has the fix, when it is released. I can't give more details than that at this time.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Fri, 08 Nov 2019 18:29:03 GMT

A patch release that includes a fix for this issue is coming soon.

Re: Has anyone been able to invoke the VMware BOOTX64.efi from a BOOTX64.efi (grub.efi) menu?

TimMann — Mon, 14 Oct 2019 19:32:14 GMT

> I am actually inserting kickstart files within the iso image,. I've had this working with legacy mode for quite a while, with multiple entries in isolinux.cfg, but have been struggling to get this to work with UEFI.

> Can you point me to an example of what the efi/boot/boot.cfg would look like? Would I be putting the menu items into bootx64.conf, or maybe grub.conf? If so, can you tell me what they would look like? Any help you can provide would be greatly appreciated.

Hmm, I think you're saying you want to have multiple different kickstart files on an ISO image that you can select from using a menu. I see where the missing piece is here. An ESXi ISO image uses isolinux as the first stage for booting in legacy BIOS mode, and that has a menu feature. I gather you're using that. But for booting in UEFI mode, there's nothing on the ISO image that has a menu feature. The first stage is isobounce (which is installed as bootx64 in the ISO's second El Torito image), which loads an ISO9660 filesystem driver from the El Torito image and then chains to mboot (which is installed as bootx64 on the iso9660 filesystem). Neither isobounce nor mboot has a menu feature.

If you want to create something on the ISO that shows a menu when booted in UEFI mode, there are a few alternatives you could try. I haven't done this myself, though, so I'm not aware if there are any pitfalls or which is best.

Personally I'd probably try VMware menu.efi, but mostly that's because I wrote it and we use it a lot internally at VMware. We haven't advertised it for customer use, but it's available under the GPL as part of our bootloader (esx-boot) source disclosure. There's documentation at the top of the source code at esx-boot/menu.c at master · vmware/esx-boot · GitHub . It's not very easy to compile the esx-boot package that it lives in, though.

You could also try grub, rEFInd, or rEFIt.

Whatever you try as a menu program, you will have to get the ISO image to execute it instead of booting right into ESXi. I suppose you could modify the ISO9660 filesystem to rename its \efi\boot\bootx64.efi to something else (say mboot.efi), and put in the menu program you choose as \efi\boot\bootx64.efi. Alternatively, you could modify the second El Torito image to invoke your menu program -- that seems like it would be more work, though.

Re: Has anyone been able to invoke the VMware BOOTX64.efi from a BOOTX64.efi (grub.efi) menu?

TimMann — Fri, 11 Oct 2019 19:55:28 GMT

The workaround for the GRUB bug where it does not null-terminate device paths is in the 6.7 ESXi bootloader, so I think chaining from GRUB to our bootloader will work for disk and ISO image boots now.

However, as explained earlier in this thread, the reason that chaining from GRUB into VMware's bootloader during PXE boot does not work is a limitation in GRUB. GRUB calls Stop on the UEFI PXE prototol before handing off to our bootloader, so our bootloader cannot load any more files over the network. It's not possible to restart the protocol (I tried). I don't know of a way to work around this on our side. You could perhaps try modifying GRUB to not do that.

More simply, I suggest using something other than GRUB to create menus, also as discussed earlier in this thread.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Mon, 16 Sep 2019 20:42:04 GMT

If you need an urgent fix on supported hardware, please file an SR. Right now there is only a plan to get a fix into the next regular patch. Also there is not a plan to release an ISO with the next patch. Plans can be changed, but asking in the community forum isn't the most effective way to make that happen.

Note: I'm an engineer working on new development on our main branch. I'm not responsible for patch and update release plans. The most I do on those is to backport my fixes from the main branch when needed.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Thu, 12 Sep 2019 19:53:13 GMT

> I don't think so. I believe it will work to apply 6.7u3 *and* the subsequent patch (when the patch is released) without rebooting in between. I will check to be sure, though.

Yes, I checked with someone who's more familiar with the patch application process, and he said:

Customers who are in pre-67U3,

- They can apply 67 patch fix directly with patch deliverables via regular method of patching (release are cumulative and they will get all fixes of 67 U3 plus latest fix of patch).

We will test this path for sure during our patch testing.

Customers who are affected with UEFI boot issue on 67 U3 environments,

- They need to use workaround and they can directly apply the patch via regular deliverables.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Wed, 11 Sep 2019 00:19:35 GMT

It ought to be fixed in the next regular patch release along the 6.7 line.

> Will then a new update-from-esxi6.7-6.7_update03.zip file be provided to come to the update 3 releases?

I don't think so. I believe it will work to apply 6.7u3 *and* the subsequent patch (when the patch is released) without rebooting in between. I will check to be sure, though.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Mon, 09 Sep 2019 19:53:48 GMT

> When will you provide an offical bugfix

It ought to be fixed in the next regular patch release along the 6.7 line. I'm not allowed to give out details or dates for future releases. In the meantime we'll issue a KB article with the workarounds.

> But wouldn't this also mean the odds are fifty-fifty that it will successfully boot with a cold restart?

There's no reason it would be 50-50. From what we've seen so far, a few machines get the error every time, while most never do. Yours is the first machine I've heard of that gets the error sometimes but not always.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Sat, 07 Sep 2019 20:22:04 GMT

To answer what was changed to break this: It's rather obscure. To work around an issue in some UEFI firmware, we added a sanity check on which UEFI memory types can contain valid page tables. The check was too strict, causing failure if the firmware puts page tables in some memory types that are unexpected but not illegal for that usage.

Re: esxi cannot boot after updating to 6.7 u3! how to fix it?

TimMann — Wed, 28 Aug 2019 18:08:09 GMT

> I've updated from Build 13981272 to Build 14320388 through shell and had the same problem. Provided solution with EFI shell didn't work for me.

It would be helpful to know more details about what you did and where it failed, if you can provide them.

> What did work was:

> 1. Made bootable USB with Update2 (Build 13006603)

> 2. Booted from it and chose upgrade

> 3. Selected the disk with ESXI installation and chose upgrade with Datastores preservation

> 4. After the installation and first boot restarted and chose Repair option on boot screen (Shift+R)

> 5. There were 2 versions listed - Build 13006603 (Default) and Build 13981272

> 6. Pressed Y to change the default hypervisor to Build 13981272

> 7. Rebooted and all seems to be back the way it was before updating

Yes, that should work to get your older installation functioning again. To clarify this for others who are reading, build 13981272 is ESXi 6.7 EP 10. So the procedure above got back the old bootloader from 6.7u2, allowing 6.7 ep10 to boot again. As a side effect, the procedure also overwrites the unsuccessful 6.7u3 installation with a copy of 6.7u2.