VMTN

So this is a bug and will be fixed? I am not sure if this was intended to work like this now or if this was a "mistake"...

Thanks for the reply.

I am running ESXi on HP Z620 and Z640 workstations.

After this weekends patch and upgrade session, the Z640's are running and the Z620's give this error.

So far, I have not been able to find a legacy boot option in the bios...

And since the hardware is not supported, we cannot get any support.

Hopefully they get this fixed in a future update.

Hi all,

I've got the same issue with a Dell Precision T1650 and probably two Dell PowerEdge T20 (not tested on T20, but they have nearly the same hardware as T1650).

First I switched from UEFI to legacy boot and the update procedure tasks were running. After that I switched back to UEFI and ESXi booted with build 13981272 instead of the build of update 3.

I won't upgrade anymore before there will be a fix for this issue.

Kind Regards.

Same exact issue with HP Z230. I was able to enable legacy boot in the BIOS, boot up and revert to ESXi-6.7.0-20190604001-standard

HP Z230 does not allow legacy boot by default, only when F9 is pressed during POST will it actually show the legacy boot options.

Yes, this seems to be a BUG. As Tim said, he will be updating the thread once he identify the fix

Here is an alternative workaround you can try if switching to legacy BIOS mode doesn't work for you. It's not easy, but I tried to make the instructions detailed. If anyone tries this, please let me know how it went and if there is anything I can clarify further in the instructions.

First, a note on rollback: Any time you install or upgrade ESXi and the first attempted boot into the new installation fails, that installation is effectively marked as invalid and you do not get the chance to try to boot it again.  ESXi recovery automatically rolls back to the previous installation, if any.  So if you would like to apply a workaround to get 6.7u3 to boot (not just to go back to 6.7u2), be sure to apply it prior to the first boot into the new installation.  If possible, install in legacy BIOS mode and switch to UEFI mode only after applying the workaround.

Workaround option 1:

Switching the firmware boot mode from UEFI to legacy BIOS will allow most affected machines to boot.  On some machines, the option to boot in legacy BIOS mode may be called CSM.

Workaround option 2:

Some machines may not have the option to boot in legacy BIOS mode.  On such a machine, you can manually copy the 6.7u2 bootloader into the

system partition to replace the 6.7u3 bootloader.

1) Get a copy of the 6.7u2 bootloader.  On an ESXi installer ISO image, the bootloader is located at /efi/boot/bootx64.efi of the ISO9660 filesystem.  Copy this file to a USB drive.  Rename it to mboot64.efi.  Plug the USB drive into the affected machine.

(Note: do not use the EFI shell to copy \EFI\BOOT\BOOTx64.EFI directly from a CD or ISO image.  That would give you the wrong file, taken from the El Torito boot image instead of the ISO9660 filesystem.)

2) Boot the affected machine into the EFI shell (not into ESXi).  If your machine does not offer the EFI shell as a built-in boot option, try http://refit.sourceforge.net/ for a downloadable boot manager that includes an EFI shell.

3) Find the filesystem names that EFI has assigned to the system partition on the boot disk and the USB drive containing your 6.7u2 bootloader.  You can do that by requesting directory listings of each filesystem with the EFI "dir" command, working upward from fs0:, until you find the ones with the expected contents.  In the example below, fs0: is the system partition and fs5: is the USB drive.

   Shell> dir fs0:

   Directory of: fs0:\

     08/05/19  02:38a <DIR>            512 EFI

     08/05/19  02:38a                   30 syslinux.cfg

     08/05/19  02:38a               61,288 safeboot.c32

     08/05/19  02:38a               93,672 mboot.c32

             3 File(s)     154,990 bytes

             1 Dir(s)

   Shell> dir fs0:\efi

   Directory of: fs0:\EFI

     08/05/19  02:38a <DIR>            512 .

     08/05/19  02:38a <DIR>              0 ..

     08/05/19  02:38a <DIR>            512 BOOT

     08/05/19  02:38a <DIR>            512 VMware

             0 File(s)           0 bytes

             4 Dir(s)

   Shell> dir fs0:\efi\vmware

   Directory of: fs0:\

     08/05/19  02:38a <DIR>            512 .

     08/05/19  02:38a <DIR>            512 ..

     08/05/19  02:38a              172,224 mboot64.efi

     08/05/19  02:38a               94,432 safebt64.efi

             2 File(s)     266,656 bytes

             2 Dir(s)

   Shell> dir fs5:\

   Directory of: fs5:\

     03/26/19  01:52p              171,400 mboot64.efi

             1 File(s)     171,400 bytes

             0 Dir(s)

4) Copy your 6.7u2 mboot64.efi file onto the system partition, replacing the one that's already there.  Continuing the example above:

   Shell> copy fs5:\mboot64.efi fs0:\efi\vmware\mboot64.efi

   Overwrite fs0:\EFI\VMware\mboot64.efi? (Yes/No/All/Cancel):y

   copying fs5:\mboot64.efi -> fs0:\EFI\VMware\mboot64.efi

    - [ok]

Hi TimMann,

thank you for providing us the workarounds. But what about those people who have tried to update ESXi 6.7 unsuccessfully? When the installation is marked as invalid no one of these workarounds can be applied, right? How can we delete the mark to try to boot ESXi 6.7 U3 with the firmware boot mode of legacy BIOS?

And will this issue be fixed with a new update ZIP file so that we can install it and boot with UEFI mode?

Kind Regards.

I've updated from Build 13981272 to Build 14320388 through shell and had the same problem. Provided solution with EFI shell didn't work for me. What did work was:

1. Made bootable USB with Update2 (Build 13006603)

2. Booted from it and chose upgrade

3. Selected the disk with ESXI installation and chose upgrade with Datastores preservation

4. After the installation and first boot  restarted and chose Repair option on boot screen (Shift+R)

5. There were 2 versions listed - Build 13006603 (Default) and Build 13981272

6. Pressed Y to change the default hypervisor to Build 13981272

7. Rebooted and all seems to be back the way it was before updating

Thanks to those who tried "workaround option 2". If the instructions didn't work for you, please try to give some details about what you did and what state you ended up in.

> When the installation is marked as invalid no one of these workarounds can be applied, right? How can we delete the mark to try to boot ESXi 6.7 U3 with the firmware boot mode of legacy BIOS?

I didn't explain how to do that because I was thinking it would be more straightforward and have fewer pitfalls to get your previous installation to boot again, then reapply the update. On the other hand, reapplying the update could reapply the 6.7u3 bootloader, so you might end up needing to do the workaround twice -- once to get your previous installation to boot, then again immediately after the 6.7u3 re-upgrade, just before the first boot into 6.7u3. Ugh.

For the adventurous, you can find the boot.cfg file of the bootbank containing the 6.7u3 installation and change the line that reads bootstate=2 or bootstate=3 and change it to bootstate=1. (If you see bootstate=0, the bootbank is valid, probably your older installation.) That will give the bootbank that was marked invalid another chance to try to boot.

> And will this issue be fixed with a new update ZIP file so that we can install it and boot with UEFI mode?

Of course this bug will be fixed in a future update. I'm just trying to be helpful for folks who are affected right now.

> I've updated from Build 13981272 to Build 14320388 through shell and had the same problem. Provided solution with EFI shell didn't work for me.

It would be helpful to know more details about what you did and where it failed, if you can provide them.

> What did work was:

> 1. Made bootable USB with Update2 (Build 13006603)

> 2. Booted from it and chose upgrade

> 3. Selected the disk with ESXI installation and chose upgrade with Datastores preservation

> 4. After the installation and first boot  restarted and chose Repair option on boot screen (Shift+R)

> 5. There were 2 versions listed - Build 13006603 (Default) and Build 13981272

> 6. Pressed Y to change the default hypervisor to Build 13981272

> 7. Rebooted and all seems to be back the way it was before updating

Yes, that should work to get your older installation functioning again. To clarify this for others who are reading, build 13981272 is ESXi 6.7 EP 10. So the procedure above got back the old bootloader from 6.7u2, allowing 6.7 ep10 to boot again. As a side effect, the procedure also overwrites the unsuccessful 6.7u3 installation with a copy of 6.7u2.

I extracted the bootloader from the 6.7u2 ISO with 7zip and used rEFInd (which was recommended since the tool you linked was not maintained anymore) to start EFI shell. In my case USB was fs0: and installation was on fs3:

I don't remember exactly but IIRC it directly threw an error when I tried to boot.

This worked for me, although I did not need steps 4-7.  I'm back on Build 13981272.

This also worked for me. I had to take some drastic recovery measures before getting to this point though.

Thanks for posting!

Thanks a lot, Tim, Man ;-) Your (this) post saved my evening!

Simply replacing the efi binary solved the problem and I could successfully boot u3 after starting another vib-install.  Thanks for the hint regarding the invalid bootbank and auto-rollback.

And the ElTorito hint!  Instructions can't be clearer.

Btw. also C220 chipset here (Intel S1200v3RP) and the same error after u3 update - confirmed to be easily avoidable by your Workaround option #2.

While this workaround is all we need for the moment, I'd be highly interested in the real cause.  Meaning, what was changed in the mboot64.efi, that breaks C220 systems.  The CPU microcode upload?

If so, do we have the latest microcode despite this helpful but ugly workaround?  Guess yes, but better to ask ;-)

Like mentioned, any technical details are always welcome as well :-)

To answer what was changed to break this: It's rather obscure. To work around an issue in some UEFI firmware,  we added a sanity check on which UEFI memory types can contain valid page tables.  The check was too strict, causing failure if the firmware puts page tables in some memory types that are unexpected but not illegal for that usage.