7 Replies Latest reply on Jul 15, 2020 12:31 AM by CreaThor

    vSphere 5.5 - Boot Menu - Scripted vsphere Installer

    ArrowSIVAC Enthusiast

      Sigh...  for support reasons of a software vendor I have to maintain a 5.5u2 cluster.

       

      Trying to stage for class / POC environment (hyperconvergence so has to work on hardware level), so I have to handle hardware

       

      Goal:

      1) Use kickstart structure answer files to load the nodes

      2) Create a boot menu from vSphere 5.5u2 (oem customized versions) to allow user to boot USB image over IPMI, select node they are on, and it feeds it the correct ks.cfg file

       

      I followed various guides / notes:

      vSphere Boot Menu Guides

       

      https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004582

      Template attached but details for fields and assumptions below.

       

       

      Create menu for the list of nodes to each customization answer file

      Documentation:

      http://www.virtuallyghetto.com/2015/06/how-to-create-custom-esxi-boot-menu-to-support-multiple-kickstart-files.html

      http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html

       

      But either it is a translation issue of my understanding of their notes, or issue with 5.5u2 or ???

       

      ###

       

      Step 1:  Create bootable USB vSphere OS key on 2GB key

       

      I named key label "esx5U2_auto_install"

       

      I followed instructions and used tool rufus

      https://tinkertry.com/rufus-takes-2-minutes-to-create-a-bootable-usb-flash-drive-for-esxi-installation

       

      I connected and used OEM vSphere cutomization ISO for vSphere 5.5u2

       

       

      Step 2:  Create answer file response files

       

      Example is node 1-4 cluster.

       

      on USB create directory "KS" and in that create files  "NODE1.cfg" -"NODE4.cfg"

       

      Example file for node1

      node1 ks.cfg

       

      # Accept vSphere license ULA

      vmaccepteula

       

      # Set root password

      rootpw password

       

      # Clear drive partition on first device in device list

      clearpart --firstdisk --overwritevmfs

       

      # Set installation target to be first drive it finds in device list

      install --firstdisk --overwritevmfs

       

      # Set static IP for vmnic0 per Spectrum Accelerate table

      network --bootproto=static --device=vmnic0 --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=node1 -- nameserver=172.20.12.100,172.20.13.100 vlanid=11

       

      # Reboot host

      reboot

       

       

      # Post Installation goodies

      # Enable SSH and Tech Suport modes

      vim-cmd hostsvc/enable_esx_shell

      vim-cmd hostsvc/start_esx_shell

      vim-cmd hostsvc/enable_ssh

      vim-cmd hostsvc/start_ssh

      esxcli network firewall ruleset set --ruleset-id sshClient --enabled yes

       

       

      Step 3: Create menu for nodes

       

       

      Edit file "isolinux.cfg" for above host menu options.  Save this file in root of USB key.

      Design is to boot over USB attached drive through the IMM (or local attached USB)

      DEFAULT menu.c32

      MENU TITLE 5.5u2 Boot Menu

      NOHALT 1

      PROMPT 0

      TIMEOUT 80

      #

      LABEL hddboot

        LOCALBOOT 0x80

        MENU LABEL ^Boot from local disk

      #

      LABEL IBM Spectrum Accelerate Node 1 Install

        KERNEL mboot.c32

        APPEND -c boot.cfg ks=usb:/KS/NODE1.cfg +++

        MENU LABEL ^1 Node1 Install

      #

      LABEL IBM Spectrum Accelerate Node 2 Install

        KERNEL mboot.c32

        APPEND -c boot.cfg ks=usb:/KS/NODE2.CFG +++

        MENU LABEL ^2 Node2 Install

      #

      LABEL IBM Spectrum Accelerate Node 3 Install

        KERNEL mboot.c32

        APPEND -c boot.cfg ks=usb:/KS/NODE3.CFG +++

        MENU LABEL ^3 Node3 Install

      #

       

      #############

       

       

       

      But...  When I boot it just runs normal script to install vSphere from basic answer file

       

      What I think I am missing is what do I need to do to the boot.cfg file to get it to call the menu (DEFAULT menu.c32)

       

       

      As usual, all help is appreciated.

       

       

       

      PS: I also tried to just effect the base load for just test of answer file (aka without fancy menu) by modification of "boot.cfg" as noted below...and it booted normal installer and ignored answers

      boot.cfg  - node 1 direct ks.cfg call

      bootstate=0

      title=Loading ESXi installer

      kernel=/tboot.b00

      kernelopt=ks=/KS/NODE1.cfg

      modules=/b.b00 --- /jumpstrt.gz --- /useropts.gz --- /k.b00 --- /chardevs.b00 --- /a.b00 --- /user.b00 --- /sb.v00 --- /s.v00 --- /brcm.v00 --- /misc_cni.v00 --- /net_bnx2.v00 --- /net_bnx2.v01 --- /net_cnic.v00 --- /net_tg3.v00 --- /scsi_bnx.v00 --- /scsi_bnx.v01 --- /brcdprov.v00 --- /net_bna.v00 --- /scsi_bfa.v00 --- /elxnet.v00 --- /emulex_c.v00 --- /ima_be2i.v00 --- /lpfc.v00 --- /scsi_be2.v00 --- /ianet_ci.v00 --- /net_igb.v00 --- /net_ixgb.v00 --- /lsiprovi.v00 --- /scsi_meg.v00 --- /scsi_mpt.v00 --- /scsi_mpt.v01 --- /concrete.v00 --- /filetran.v00 --- /fupb.v00 --- /fwupdate.v00 --- /hwckvm.v00 --- /ilfu.v00 --- /immpasst.v00 --- /pciinfo.v00 --- /soibms.v00 --- /mlnxprov.v00 --- /net_mlx4.v00 --- /net_mlx4.v01 --- /ima_qla4.v00 --- /net_qlcn.v00 --- /net_qlge.v00 --- /qlnative.v00 --- /qlogic_c.v00 --- /scsi_qla.v00 --- /ata_pata.v00 --- /ata_pata.v01 --- /ata_pata.v02 --- /ata_pata.v03 --- /ata_pata.v04 --- /ata_pata.v05 --- /ata_pata.v06 --- /ata_pata.v07 --- /block_cc.v00 --- /ehci_ehc.v00 --- /weaselin.t00 --- /esx_dvfi.v00 --- /xlibs.v00 --- /ipmi_ipm.v00 --- /ipmi_ipm.v01 --- /ipmi_ipm.v02 --- /misc_dri.v00 --- /mtip32xx.v00 --- /net_be2n.v00 --- /net_e100.v00 --- /net_e100.v01 --- /net_enic.v00 --- /net_forc.v00 --- /net_nx_n.v00 --- /net_vmxn.v00 --- /ohci_usb.v00 --- /rste.v00 --- /sata_ahc.v00 --- /sata_ata.v00 --- /sata_sat.v00 --- /sata_sat.v01 --- /sata_sat.v02 --- /sata_sat.v03 --- /sata_sat.v04 --- /scsi_aac.v00 --- /scsi_adp.v00 --- /scsi_aic.v00 --- /scsi_fni.v00 --- /scsi_hps.v00 --- /scsi_ips.v00 --- /scsi_lpf.v00 --- /scsi_meg.v01 --- /scsi_meg.v02 --- /scsi_mpt.v02 --- /scsi_mpt.v03 --- /scsi_qla.v01 --- /uhci_usb.v00 --- /tools.t00 --- /lnvcusto.v00 --- /xorg.v00 --- /imgdb.tgz --- /imgpayld.tgz

      build=

      updated=0

        • 1. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
          ArrowSIVAC Enthusiast

          So done quite a bit of poking around and think I am close but missing a few pieces.

           

          What is the boot mode is very important.  The server I have does support "Legacy" boot mode.  As such the vSphere installer displays and boots differently from install media.

          Legacy BootUEFI Boot

          Boot from ISO / USB..  notice menu .. get option to interact and choose boot option (and so kickstart answer)

          Boot from ISO / USB..  notice no menu .. just boots. No menu. No options.

          When I add / change customized files (see below)

          When I add / change customized files (see below)

           

           

           

           

          What I need to understand is the different files and what the boot order is UEFI vs Legacy boot..... Trying to sketch it out.

           

          As such. With UEFI only boot ability I made the following file changes: Example is paired down to just two nodes

          syslinux.cfgisolinux.cfgboot1.cfgNODE1.cfg

          default menu.c32

          prompt 0

          menu title Arrow|IBM|Lenovo|Brocade HyperConverge 5.5u2

          timeout 100

           

          label unetbootindefault

          menu label ^Boot from local disk

          kernel /ubnkern

          append initrd=/ubninit

           

          label ubnentry0

           

          menu label ^Node1

          #kernel /ubnkern

          kernel /MBOOT.C32

          append initrd=/ubninit -c boot1.cfg

           

          label ubnentry1

           

          menu label ^Node2

          #kernel /ubnkern

          kernel /MBOOT.C32

          append initrd=/ubninit -c boot2.cfg

           

          label ubnentry2

          EFAULT menu.c32

          MENU TITLE HyperConverge 5.5u2

          NOHALT 1

          PROMPT 0

          TIMEOUT 80

           

          LABEL hddboot

            LOCALBOOT 0x80

            MENU LABEL ^Boot from local disk

          LABEL NODE1

            KERNEL mboot.c32

            APPEND -c boot1.cfg +++

            MENU LABEL ^1 NODE1 Install

          LABEL NODE2

            KERNEL mboot.c32

            APPEND -c boot2.cfg +++

            MENU LABEL ^1 NODE2 Install

          bootstate=0

          title=Loading ESXi installer Node1

          kernel=/tboot.b00

          #kernelopt=runweasel

          kernelopt=ks=usb:/NODE1.cfg

          modules=/b.b00 --- /jumpstrt.gz <snip>--- /imgpayld.tgz

          build=

          updated=0

          ## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster

          # v0.01

          ####

           

           

          vmaccepteula

           

          #clear any existing partition information

          clearpart --alldrives --overwritevmfs

           

          #specify to install on first usb drive it finds

          install --firstdisk=usb --overwritevmfs

           

          # Set root password

          rootpw Pa55w0rd!

          reboot

           

          # set vmnics state in vswitch0 to vmnic0 active

          /bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0

          <snip>

          *Thought process was that I copy all those files in both / and /efi/boot to capture compute nodes weather they boot UEFI or Legacy.

           

          I thought that this would work because:  EFI boot hardware -> /efi/boot/bootx64.efi  -> Calls menu.c32 (binary)-> calls syslinux.cfg (options and user selection) Defines boot kernel "MBOOT.C32" and then initrd /ubninit with options file "boot#.cfg" -> calls boot#.cfg for kernel file "tboot.b00" and kernel options, kickstart file /NODE#.cfg  and modules outside initrd-> calls kickstartfile for auto answer files for OS installation.

           

           

          Question:

          1) Can UEFI boot mode servers still use a "menu" in vSphere installation?

          2) Can someone direct me to what the flow for the boot structure is such that I can debug the boot menu?

          1 person found this helpful
          • 2. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
            ArrowSIVAC Enthusiast

             

            First is a baseline that systems when they boot can boot "Legacy" (BIOS boot loader) or EFI The boot path they take impacts the flow of the boot order, and by necessity, the ability to call your installation scripts

             

            I documented much of this via a thread on the vmware forums https://communities.vmware.com/message/2596896#2596896

            * Notice that in EFI, vmware bypassed any parsing of isolinux.cfg and as such there is no menu option.  They converted "/mboot.c32" to "/efi/boot/bootx64.efi"

            (32bit EFI option also exists but same concept)

             

             

            Goal of this is to create a USB boot image/ key , which weather Legacy or EFI will boot up, allow user selection of "node ##" and then install via a user defined kickstart answer file

             

             

            #################################################

             

            Step 1: Create bootable USB vSphere OS key

             

            2GB key

             

            I named it "esx5U2_auto_install"

             

            I followed instructions and used tool rufus

            https://tinkertry.com/rufus-takes-2-minutes-to-create-a-bootable-usb-flash-drive-for-esxi-installation

             

            I connected and used Lenovo vSphere customization ISO for vSphere 5.5u2 per this location https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5097258

            * The above instructions are based on Lenovo, can be used for ANY OEM / or standard vSphere boot image

             

             

             

             

            Step 2: Create vSphere answer file for installation

             

            https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004582

            Template attached but details for fields and assumptions below.

             

             

            Create menu for the list of nodes to each customization answer file

            Documentation:

            http://www.virtuallyghetto.com/2015/06/how-to-create-custom-esxi-boot-menu-to-support-multiple-kickstart-files.html

            http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html

             

            Example:  node1 answer file called "NODE1.KS" which will install on first USB drive and install with basic hostname, IP, username, password, SSH and esxi shell enabled (for later scripting needs), DNS and NTP.  Keep file names shorter then eight characters and all in upper case to avoid ISO / USB format issues.

             

            Save these files in /   <root> of the usb key

            ## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster

            # v0.5

            ####

             

             

            vmaccepteula

             

            #clear any existing partition information

            clearpart --alldrives --overwritevmfs

             

            #specify to install on first usb drive it finds

            install --firstdisk=usb --overwritevmfs

             

            # Set root password

            rootpw Pa55w0rd!

            reboot

             

            # set vmnics state in vswitch0 to vmnic0 active

            /bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0

             

             

            # Set network settings for system to static IP

            network --bootproto=static --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=x385001.ibm.aessatl.arrow.com --nameserver=172.20.12.100,172.20.13.100 --addvmportgroup=1

             

            %post --interpreter=busybox

             

            # stop USB Arbitrator service to access USB device in ESXi Shell

            /etc/init.d/usbarbitrator stop

             

            %firstboot --interpreter=busybox

             

            # enable & start SSH

            vim-cmd hostsvc/enable_ssh

            vim-cmd hostsvc/start_ssh

             

            # enable & start ESXi Shell

            vim-cmd hostsvc/enable_esx_shell

            vim-cmd hostsvc/start_esx_shell

             

            # Suppress ESXi Shell warning

            esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

             

             

            Repeat / copy above and save for list of nodes or answer file options you wish.  These will be called later on based on menu as kernel options

             

             

             

            Step 3: Modify base vSphere installation to enable menu

             

            Example: nodes 1-4  boot options from separate kickstart answer files

             

            This is documented two paths… legacy and EFI as they are very different.

             

            From shell (DOS or Linux) in the directory root of the USB key created…..

            Legacy

            EFI

            # Make backup of isolinux.cfg

            cp /isolinux.cfg  isolinux.bak

            # modify isolinux.cfg to have menu

            edit isolinux.cfg

             

            DEFAULT menu.c32

            MENU TITLE Lenovo_ESXi6.0_20150420 Arrow HyperConverge Boot Menu

            NOHALT 1

            PROMPT 0

            TIMEOUT 80

            LABEL hddboot

            LOCALBOOT 0x80

            MENU LABEL ^Boot from local disk

             

            LABEL install node1

            KERNEL mboot.c32

            APPEND -c BOOT1.CFG

            MENU LABEL Lenovo_ESXi6.0_20150420 Node1 ^Installer

            LABEL install node2

            KERNEL mboot.c32

            APPEND -c BOOT2.CFG

            MENU LABEL Lenovo_ESXi6.0_20150420 Node2 ^Installer

            LABEL install node3

            KERNEL mboot.c32

            APPEND -c BOOT3.CFG

            MENU LABEL Lenovo_ESXi6.0_20150420 Node3 ^Installer

            LABEL install node4

            KERNEL mboot.c32

            APPEND -c BOOT4.CFG

            MENU LABEL Lenovo_ESXi6.0_20150420 Node4 ^Installer

             

            # modify boot.cfg which has kernel and module options to be node specific on kickstart file called

             

            cp boot.cfg BOOT1.CFG

            edit BOOT1.CFG

            bootstate=0

            title=Loading ESXi installer

            kernel=/tboot.b00

            #kernelopt=runweasel

            kernelopt=ks=usb:/NODE1.KS

            modules=/b.b00 --- /jumpstrt.gz <snip> --- /imgpayld.tgz

            build=

            updated=0

            # save files..  making both boot config and kickstart files for each "node" or menu option

            # We have to replace the simplistic EFI boot system from VMWare

            # EFI boot https://sourceforge.net/projects/refind/?source=typ_redirect

            # Documentation for menu http://www.rodsbooks.com/refind/configfile.html

            # rename vmware's boot file as it will be used later

            mv /efi/boot/bootx64.efi /efi/boot/esxboot.efi

             

            # Download rEFInd and unzip

            cp /tmp/refind-bin-0.10.3\refind/refind_x64.efi <usb vsphere>/efi/boot/bootx64.efi

             

            # Now with the EFI replaced we need a menu file

            # Note: the options in this menu file do NOT replace weasel but just append to kernel options in the base boot.cfg (hense why we never touch it when modifications made for legacy boot side)

            # We also say timeout is infiite…aka requires user to select. As well as not to scan for any file *.efi and add dynamically to the boot menu

            edit /efi/boot/refind.conf

            timeout 0

            scanfor manual

            menuentry "Lenovo_ESXi6.0_20150420 Node1" {

                loader \EFI\boot\esxboot.efi

                options "ks=usb:/NODE1.KS"

            }

            menuentry "Lenovo_ESXi6.0_20150420 Node2" {

                loader \EFI\boot\esxboot.efi

                options "ks=usb:/NODE2.KS"

            }

            # save files..  making both boot menu option and kickstart files for each "node" or menu option

            * Name files all in upper case https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026373

            1 person found this helpful
            • 3. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
              gilles974 Lurker

              Hello,

               

              it's a very interesting post.

              I'm trying to implement the same but it does not work...

              I have the boot menu but as if my refind.conf is not used...

               

              Do i need to reference the refind.conf somewhere? (you seems to mention the boot.cfg but i can't see in the boot.cfg you show)

               

              Thanks for your help!!!

               

              Regards

              • 4. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
                forcerk Novice

                It is the same for me. I do get the rEFInd boot menu but non of created menuentry.

                • 5. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
                  gilles974 Lurker

                  Hi,

                  sorry for not giving news on this.

                  It happen to be working as explained on physical hardware, unfortunately not on virtual machine.

                  I suspect that it's because the mkisof command i have does not respect the ISOxxxx norm but unfortunately i was never able to sort it of

                   

                  Regards


                  Gilles

                  • 6. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
                    justShyOf8 Novice

                    you mentioned appending the refind.conf to the boot.cfg, but haven't found any doc describing how to do that; I tried the following(didn't work):

                     

                    kernelopt=runweasel cdromBoot append -c refind.conf

                     

                    I keep getting the error

                     

                    Configuration error while parsing /EFI/VMware/boot.cfg

                    Fatal error: 15 (Not found)

                     

                    any help would be greatly appreciated

                    • 7. Re: vSphere 5.5 - Boot Menu - Scripted vsphere Installer
                      CreaThor Lurker

                      Hi everybody,

                       

                      Did anyone ever figure this out? I'm having the same issue as described before -- the installation successfully enters the rEFInd boot menu; however, all changes made to refind.conf are completely ignored.

                       

                      Any help is much appreciated!