1 2 3 4 5 Previous Next 61 Replies Latest reply on Nov 12, 2019 5:38 PM by TimMann Go to original post
      • 15. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
        Snify Lurker

        So this is a bug and will be fixed? I am not sure if this was intended to work like this now or if this was a "mistake"...

        Thanks for the reply.

        • 16. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
          bakerjw1 Lurker

          I am running ESXi on HP Z620 and Z640 workstations.

          After this weekends patch and upgrade session, the Z640's are running and the Z620's give this error.

          So far, I have not been able to find a legacy boot option in the bios...

           

          And since the hardware is not supported, we cannot get any support.

          Hopefully they get this fixed in a future update.

          • 17. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
            TheExpert Novice

            Hi all,

             

            I've got the same issue with a Dell Precision T1650 and probably two Dell PowerEdge T20 (not tested on T20, but they have nearly the same hardware as T1650).

             

            First I switched from UEFI to legacy boot and the update procedure tasks were running. After that I switched back to UEFI and ESXi booted with build 13981272 instead of the build of update 3.

             

            I won't upgrade anymore before there will be a fix for this issue.

             

            Kind Regards.

            • 18. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
              oliera Novice

              Same exact issue with HP Z230. I was able to enable legacy boot in the BIOS, boot up and revert to ESXi-6.7.0-20190604001-standard

              HP Z230 does not allow legacy boot by default, only when F9 is pressed during POST will it actually show the legacy boot options.

              1 person found this helpful
              • 19. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                ashishsingh1508 Enthusiast
                vExpert

                Yes, this seems to be a BUG. As Tim said, he will be updating the thread once he identify the fix

                1 person found this helpful
                • 20. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                  TimMann Enthusiast
                  VMware Employees

                  Here is an alternative workaround you can try if switching to legacy BIOS mode doesn't work for you. It's not easy, but I tried to make the instructions detailed. If anyone tries this, please let me know how it went and if there is anything I can clarify further in the instructions.

                   

                  First, a note on rollback: Any time you install or upgrade ESXi and the first attempted boot into the new installation fails, that installation is effectively marked as invalid and you do not get the chance to try to boot it again.  ESXi recovery automatically rolls back to the previous installation, if any.  So if you would like to apply a workaround to get 6.7u3 to boot (not just to go back to 6.7u2), be sure to apply it prior to the first boot into the new installation.  If possible, install in legacy BIOS mode and switch to UEFI mode only after applying the workaround.

                   

                  Workaround option 1:

                   

                  Switching the firmware boot mode from UEFI to legacy BIOS will allow most affected machines to boot.  On some machines, the option to boot in legacy BIOS mode may be called CSM.

                   

                  Workaround option 2:

                   

                  Some machines may not have the option to boot in legacy BIOS mode.  On such a machine, you can manually copy the 6.7u2 bootloader into the

                  system partition to replace the 6.7u3 bootloader.

                   

                  1) Get a copy of the 6.7u2 bootloader.  On an ESXi installer ISO image, the bootloader is located at /efi/boot/bootx64.efi of the ISO9660 filesystem.  Copy this file to a USB drive.  Rename it to mboot64.efi.  Plug the USB drive into the affected machine.

                   

                  (Note: do not use the EFI shell to copy \EFI\BOOT\BOOTx64.EFI directly from a CD or ISO image.  That would give you the wrong file, taken from the El Torito boot image instead of the ISO9660 filesystem.)

                   

                  2) Boot the affected machine into the EFI shell (not into ESXi).  If your machine does not offer the EFI shell as a built-in boot option, try http://refit.sourceforge.net/ for a downloadable boot manager that includes an EFI shell.

                   

                  3) Find the filesystem names that EFI has assigned to the system partition on the boot disk and the USB drive containing your 6.7u2 bootloader.  You can do that by requesting directory listings of each filesystem with the EFI "dir" command, working upward from fs0:, until you find the ones with the expected contents.  In the example below, fs0: is the system partition and fs5: is the USB drive.

                   

                     Shell> dir fs0:

                     Directory of: fs0:\

                       08/05/19  02:38a <DIR>            512 EFI

                       08/05/19  02:38a                   30 syslinux.cfg

                       08/05/19  02:38a               61,288 safeboot.c32

                       08/05/19  02:38a               93,672 mboot.c32

                               3 File(s)     154,990 bytes

                               1 Dir(s)

                   

                     Shell> dir fs0:\efi

                     Directory of: fs0:\EFI

                       08/05/19  02:38a <DIR>            512 .

                       08/05/19  02:38a <DIR>              0 ..

                       08/05/19  02:38a <DIR>            512 BOOT

                       08/05/19  02:38a <DIR>            512 VMware

                               0 File(s)           0 bytes

                               4 Dir(s)

                   

                     Shell> dir fs0:\efi\vmware

                     Directory of: fs0:\

                       08/05/19  02:38a <DIR>            512 .

                       08/05/19  02:38a <DIR>            512 ..

                       08/05/19  02:38a              172,224 mboot64.efi

                       08/05/19  02:38a               94,432 safebt64.efi

                               2 File(s)     266,656 bytes

                               2 Dir(s)

                   

                     Shell> dir fs5:\

                     Directory of: fs5:\

                       03/26/19  01:52p              171,400 mboot64.efi

                               1 File(s)     171,400 bytes

                               0 Dir(s)

                    

                  4) Copy your 6.7u2 mboot64.efi file onto the system partition, replacing the one that's already there.  Continuing the example above:

                   

                     Shell> copy fs5:\mboot64.efi fs0:\efi\vmware\mboot64.efi

                     Overwrite fs0:\EFI\VMware\mboot64.efi? (Yes/No/All/Cancel):y

                     copying fs5:\mboot64.efi -> fs0:\EFI\VMware\mboot64.efi

                      - [ok]

                  2 people found this helpful
                  • 21. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                    TheExpert Novice

                    Hi TimMann,

                     

                    thank you for providing us the workarounds. But what about those people who have tried to update ESXi 6.7 unsuccessfully? When the installation is marked as invalid no one of these workarounds can be applied, right? How can we delete the mark to try to boot ESXi 6.7 U3 with the firmware boot mode of legacy BIOS?

                     

                    And will this issue be fixed with a new update ZIP file so that we can install it and boot with UEFI mode?

                     

                    Kind Regards.

                    • 22. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                      SzDredd Lurker

                      I've updated from Build 13981272 to Build 14320388 through shell and had the same problem. Provided solution with EFI shell didn't work for me. What did work was:

                      1. Made bootable USB with Update2 (Build 13006603)

                      2. Booted from it and chose upgrade

                      3. Selected the disk with ESXI installation and chose upgrade with Datastores preservation

                      4. After the installation and first boot  restarted and chose Repair option on boot screen (Shift+R)

                      5. There were 2 versions listed - Build 13006603 (Default) and Build 13981272

                      6. Pressed Y to change the default hypervisor to Build 13981272

                      7. Rebooted and all seems to be back the way it was before updating

                      • 23. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                        TimMann Enthusiast
                        VMware Employees

                        Thanks to those who tried "workaround option 2". If the instructions didn't work for you, please try to give some details about what you did and what state you ended up in.

                         

                        > When the installation is marked as invalid no one of these workarounds can be applied, right? How can we delete the mark to try to boot ESXi 6.7 U3 with the firmware boot mode of legacy BIOS?

                         

                        I didn't explain how to do that because I was thinking it would be more straightforward and have fewer pitfalls to get your previous installation to boot again, then reapply the update. On the other hand, reapplying the update could reapply the 6.7u3 bootloader, so you might end up needing to do the workaround twice -- once to get your previous installation to boot, then again immediately after the 6.7u3 re-upgrade, just before the first boot into 6.7u3. Ugh.

                         

                        For the adventurous, you can find the boot.cfg file of the bootbank containing the 6.7u3 installation and change the line that reads bootstate=2 or bootstate=3 and change it to bootstate=1. (If you see bootstate=0, the bootbank is valid, probably your older installation.) That will give the bootbank that was marked invalid another chance to try to boot.

                         

                        > And will this issue be fixed with a new update ZIP file so that we can install it and boot with UEFI mode?

                         

                        Of course this bug will be fixed in a future update. I'm just trying to be helpful for folks who are affected right now.

                        • 24. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                          TimMann Enthusiast
                          VMware Employees

                          > I've updated from Build 13981272 to Build 14320388 through shell and had the same problem. Provided solution with EFI shell didn't work for me.

                           

                          It would be helpful to know more details about what you did and where it failed, if you can provide them.

                           

                          > What did work was:

                          > 1. Made bootable USB with Update2 (Build 13006603)

                          > 2. Booted from it and chose upgrade

                          > 3. Selected the disk with ESXI installation and chose upgrade with Datastores preservation

                          > 4. After the installation and first boot  restarted and chose Repair option on boot screen (Shift+R)

                          > 5. There were 2 versions listed - Build 13006603 (Default) and Build 13981272

                          > 6. Pressed Y to change the default hypervisor to Build 13981272

                          > 7. Rebooted and all seems to be back the way it was before updating

                           

                          Yes, that should work to get your older installation functioning again. To clarify this for others who are reading, build 13981272 is ESXi 6.7 EP 10. So the procedure above got back the old bootloader from 6.7u2, allowing 6.7 ep10 to boot again. As a side effect, the procedure also overwrites the unsuccessful 6.7u3 installation with a copy of 6.7u2.

                          • 25. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                            SzDredd Lurker

                            I extracted the bootloader from the 6.7u2 ISO with 7zip and used rEFInd (which was recommended since the tool you linked was not maintained anymore) to start EFI shell. In my case USB was fs0: and installation was on fs3:

                            I don't remember exactly but IIRC it directly threw an error when I tried to boot.

                            • 26. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                              cshep Lurker

                              This worked for me, although I did not need steps 4-7.  I'm back on Build 13981272.

                              • 27. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                                bakerjw1 Lurker

                                This also worked for me. I had to take some drastic recovery measures before getting to this point though.

                                Thanks for posting!

                                • 28. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                                  sharix Novice

                                  Thanks a lot, Tim, Man ;-) Your (this) post saved my evening!

                                   

                                  Simply replacing the efi binary solved the problem and I could successfully boot u3 after starting another vib-install.  Thanks for the hint regarding the invalid bootbank and auto-rollback.

                                  And the ElTorito hint!  Instructions can't be clearer.

                                   

                                  Btw. also C220 chipset here (Intel S1200v3RP) and the same error after u3 update - confirmed to be easily avoidable by your Workaround option #2.

                                   

                                  While this workaround is all we need for the moment, I'd be highly interested in the real cause.  Meaning, what was changed in the mboot64.efi, that breaks C220 systems.  The CPU microcode upload?

                                  If so, do we have the latest microcode despite this helpful but ugly workaround?  Guess yes, but better to ask ;-)

                                  Like mentioned, any technical details are always welcome as well :-)

                                  • 29. Re: esxi cannot boot after updating to 6.7 u3! how to fix it?
                                    TimMann Enthusiast
                                    VMware Employees

                                    To answer what was changed to break this: It's rather obscure. To work around an issue in some UEFI firmware,  we added a sanity check on which UEFI memory types can contain valid page tables.  The check was too strict, causing failure if the firmware puts page tables in some memory types that are unexpected but not illegal for that usage.