VMware Cloud Community
ArrowSIVAC
Enthusiast
Enthusiast

vSphere 5.5 - Boot Menu - Scripted vsphere Installer

Sigh...  for support reasons of a software vendor I have to maintain a 5.5u2 cluster.

Trying to stage for class / POC environment (hyperconvergence so has to work on hardware level), so I have to handle hardware Smiley Happy

Goal:

1) Use kickstart structure answer files to load the nodes

2) Create a boot menu from vSphere 5.5u2 (oem customized versions) to allow user to boot USB image over IPMI, select node they are on, and it feeds it the correct ks.cfg file

I followed various guides / notes:

vSphere Boot Menu Guides

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=20045...

Template attached but details for fields and assumptions below.

Create menu for the list of nodes to each customization answer file

Documentation:

http://www.virtuallyghetto.com/2015/06/how-to-create-custom-esxi-boot-menu-to-support-multiple-kicks...

http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html

But either it is a translation issue of my understanding of their notes, or issue with 5.5u2 or ???

###

Step 1:  Create bootable USB vSphere OS key on 2GB key

I named key label "esx5U2_auto_install"

I followed instructions and used tool rufus

https://tinkertry.com/rufus-takes-2-minutes-to-create-a-bootable-usb-flash-drive-for-esxi-installati...

I connected and used OEM vSphere cutomization ISO for vSphere 5.5u2

Step 2:  Create answer file response files

Example is node 1-4 cluster.

on USB create directory "KS" and in that create files  "NODE1.cfg" -"NODE4.cfg"

Example file for node1

node1 ks.cfg

# Accept vSphere license ULA

vmaccepteula

# Set root password

rootpw password

# Clear drive partition on first device in device list

clearpart --firstdisk --overwritevmfs

# Set installation target to be first drive it finds in device list

install --firstdisk --overwritevmfs

# Set static IP for vmnic0 per Spectrum Accelerate table

network --bootproto=static --device=vmnic0 --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=node1 -- nameserver=172.20.12.100,172.20.13.100 vlanid=11

# Reboot host

reboot

# Post Installation goodies

# Enable SSH and Tech Suport modes

vim-cmd hostsvc/enable_esx_shell

vim-cmd hostsvc/start_esx_shell

vim-cmd hostsvc/enable_ssh

vim-cmd hostsvc/start_ssh

esxcli network firewall ruleset set --ruleset-id sshClient --enabled yes

Step 3: Create menu for nodes

Edit file "isolinux.cfg" for above host menu options.  Save this file in root of USB key.

Design is to boot over USB attached drive through the IMM (or local attached USB)

DEFAULT menu.c32

MENU TITLE 5.5u2 Boot Menu

NOHALT 1

PROMPT 0

TIMEOUT 80

#

LABEL hddboot

  LOCALBOOT 0x80

  MENU LABEL ^Boot from local disk

#

LABEL IBM Spectrum Accelerate Node 1 Install

  KERNEL mboot.c32

  APPEND -c boot.cfg ks=usb:/KS/NODE1.cfg +++

  MENU LABEL ^1 Node1 Install

#

LABEL IBM Spectrum Accelerate Node 2 Install

  KERNEL mboot.c32

  APPEND -c boot.cfg ks=usb:/KS/NODE2.CFG +++

  MENU LABEL ^2 Node2 Install

#

LABEL IBM Spectrum Accelerate Node 3 Install

  KERNEL mboot.c32

  APPEND -c boot.cfg ks=usb:/KS/NODE3.CFG +++

  MENU LABEL ^3 Node3 Install

#

#############

But...  When I boot it just runs normal script to install vSphere from basic answer file

What I think I am missing is what do I need to do to the boot.cfg file to get it to call the menu (DEFAULT menu.c32)

As usual, all help is appreciated.

PS: I also tried to just effect the base load for just test of answer file (aka without fancy menu) by modification of "boot.cfg" as noted below...and it booted normal installer and ignored answers

boot.cfg  - node 1 direct ks.cfg call

bootstate=0

title=Loading ESXi installer

kernel=/tboot.b00

kernelopt=ks=/KS/NODE1.cfg

modules=/b.b00 --- /jumpstrt.gz --- /useropts.gz --- /k.b00 --- /chardevs.b00 --- /a.b00 --- /user.b00 --- /sb.v00 --- /s.v00 --- /brcm.v00 --- /misc_cni.v00 --- /net_bnx2.v00 --- /net_bnx2.v01 --- /net_cnic.v00 --- /net_tg3.v00 --- /scsi_bnx.v00 --- /scsi_bnx.v01 --- /brcdprov.v00 --- /net_bna.v00 --- /scsi_bfa.v00 --- /elxnet.v00 --- /emulex_c.v00 --- /ima_be2i.v00 --- /lpfc.v00 --- /scsi_be2.v00 --- /ianet_ci.v00 --- /net_igb.v00 --- /net_ixgb.v00 --- /lsiprovi.v00 --- /scsi_meg.v00 --- /scsi_mpt.v00 --- /scsi_mpt.v01 --- /concrete.v00 --- /filetran.v00 --- /fupb.v00 --- /fwupdate.v00 --- /hwckvm.v00 --- /ilfu.v00 --- /immpasst.v00 --- /pciinfo.v00 --- /soibms.v00 --- /mlnxprov.v00 --- /net_mlx4.v00 --- /net_mlx4.v01 --- /ima_qla4.v00 --- /net_qlcn.v00 --- /net_qlge.v00 --- /qlnative.v00 --- /qlogic_c.v00 --- /scsi_qla.v00 --- /ata_pata.v00 --- /ata_pata.v01 --- /ata_pata.v02 --- /ata_pata.v03 --- /ata_pata.v04 --- /ata_pata.v05 --- /ata_pata.v06 --- /ata_pata.v07 --- /block_cc.v00 --- /ehci_ehc.v00 --- /weaselin.t00 --- /esx_dvfi.v00 --- /xlibs.v00 --- /ipmi_ipm.v00 --- /ipmi_ipm.v01 --- /ipmi_ipm.v02 --- /misc_dri.v00 --- /mtip32xx.v00 --- /net_be2n.v00 --- /net_e100.v00 --- /net_e100.v01 --- /net_enic.v00 --- /net_forc.v00 --- /net_nx_n.v00 --- /net_vmxn.v00 --- /ohci_usb.v00 --- /rste.v00 --- /sata_ahc.v00 --- /sata_ata.v00 --- /sata_sat.v00 --- /sata_sat.v01 --- /sata_sat.v02 --- /sata_sat.v03 --- /sata_sat.v04 --- /scsi_aac.v00 --- /scsi_adp.v00 --- /scsi_aic.v00 --- /scsi_fni.v00 --- /scsi_hps.v00 --- /scsi_ips.v00 --- /scsi_lpf.v00 --- /scsi_meg.v01 --- /scsi_meg.v02 --- /scsi_mpt.v02 --- /scsi_mpt.v03 --- /scsi_qla.v01 --- /uhci_usb.v00 --- /tools.t00 --- /lnvcusto.v00 --- /xorg.v00 --- /imgdb.tgz --- /imgpayld.tgz

build=

updated=0

8 Replies
ArrowSIVAC
Enthusiast
Enthusiast

So done quite a bit of poking around and think I am close but missing a few pieces.

What is the boot mode is very important.  The server I have does support "Legacy" boot mode.  As such the vSphere installer displays and boots differently from install media.

Legacy BootUEFI Boot

Boot from ISO / USB..  notice menu .. get option to interact and choose boot option (and so kickstart answer)

pastedImage_12.png

Boot from ISO / USB..  notice no menu .. just boots. No menu. No options.

pastedImage_9.png

When I add / change customized files (see below)

pastedImage_1.png

When I add / change customized files (see below)

pastedImage_0.png

What I need to understand is the different files and what the boot order is UEFI vs Legacy boot..... Trying to sketch it out.

pastedImage_3.png

As such. With UEFI only boot ability I made the following file changes: Example is paired down to just two nodes

syslinux.cfgisolinux.cfgboot1.cfgNODE1.cfg

default menu.c32

prompt 0

menu title Arrow|IBM|Lenovo|Brocade HyperConverge 5.5u2

timeout 100

label unetbootindefault

menu label ^Boot from local disk

kernel /ubnkern

append initrd=/ubninit

label ubnentry0

menu label ^Node1

#kernel /ubnkern

kernel /MBOOT.C32

append initrd=/ubninit -c boot1.cfg

label ubnentry1

menu label ^Node2

#kernel /ubnkern

kernel /MBOOT.C32

append initrd=/ubninit -c boot2.cfg

label ubnentry2

EFAULT menu.c32

MENU TITLE HyperConverge 5.5u2

NOHALT 1

PROMPT 0

TIMEOUT 80

LABEL hddboot

  LOCALBOOT 0x80

  MENU LABEL ^Boot from local disk

LABEL NODE1

  KERNEL mboot.c32

  APPEND -c boot1.cfg +++

  MENU LABEL ^1 NODE1 Install

LABEL NODE2

  KERNEL mboot.c32

  APPEND -c boot2.cfg +++

  MENU LABEL ^1 NODE2 Install

bootstate=0

title=Loading ESXi installer Node1

kernel=/tboot.b00

#kernelopt=runweasel

kernelopt=ks=usb:/NODE1.cfg

modules=/b.b00 --- /jumpstrt.gz <snip>--- /imgpayld.tgz

build=

updated=0

## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster

# v0.01

####

vmaccepteula

#clear any existing partition information

clearpart --alldrives --overwritevmfs

#specify to install on first usb drive it finds

install --firstdisk=usb --overwritevmfs

# Set root password

rootpw Pa55w0rd!

reboot

# set vmnics state in vswitch0 to vmnic0 active

/bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0

<snip>

*Thought process was that I copy all those files in both / and /efi/boot to capture compute nodes weather they boot UEFI or Legacy.

I thought that this would work because:  EFI boot hardware -> /efi/boot/bootx64.efi  -> Calls menu.c32 (binary)-> calls syslinux.cfg (options and user selection) Defines boot kernel "MBOOT.C32" and then initrd /ubninit with options file "boot#.cfg" -> calls boot#.cfg for kernel file "tboot.b00" and kernel options, kickstart file /NODE#.cfg  and modules outside initrd-> calls kickstartfile for auto answer files for OS installation.

Question:

1) Can UEFI boot mode servers still use a "menu" in vSphere installation?

2) Can someone direct me to what the flow for the boot structure is such that I can debug the boot menu?

ArrowSIVAC
Enthusiast
Enthusiast

First is a baseline that systems when they boot can boot "Legacy" (BIOS boot loader) or EFI The boot path they take impacts the flow of the boot order, and by necessity, the ability to call your installation scripts

I documented much of this via a thread on the vmware forums https://communities.vmware.com/message/2596896#2596896

pastedImage_0.png

* Notice that in EFI, vmware bypassed any parsing of isolinux.cfg and as such there is no menu option.  They converted "/mboot.c32" to "/efi/boot/bootx64.efi"

(32bit EFI option also exists but same concept)

Goal of this is to create a USB boot image/ key , which weather Legacy or EFI will boot up, allow user selection of "node ##" and then install via a user defined kickstart answer file

#################################################

Step 1: Create bootable USB vSphere OS key

2GB key

I named it "esx5U2_auto_install"

I followed instructions and used tool rufus

https://tinkertry.com/rufus-takes-2-minutes-to-create-a-bootable-usb-flash-drive-for-esxi-installati...

I connected and used Lenovo vSphere customization ISO for vSphere 5.5u2 per this location https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5097258

pastedImage_3.png

* The above instructions are based on Lenovo, can be used for ANY OEM / or standard vSphere boot image

Step 2: Create vSphere answer file for installation

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=20045...

Template attached but details for fields and assumptions below.

Create menu for the list of nodes to each customization answer file

Documentation:

http://www.virtuallyghetto.com/2015/06/how-to-create-custom-esxi-boot-menu-to-support-multiple-kicks...

http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html

Example:  node1 answer file called "NODE1.KS" which will install on first USB drive and install with basic hostname, IP, username, password, SSH and esxi shell enabled (for later scripting needs), DNS and NTP.  Keep file names shorter then eight characters and all in upper case to avoid ISO / USB format issues.

Save these files in /   <root> of the usb key

## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster

# v0.5

####

vmaccepteula

#clear any existing partition information

clearpart --alldrives --overwritevmfs

#specify to install on first usb drive it finds

install --firstdisk=usb --overwritevmfs

# Set root password

rootpw Pa55w0rd!

reboot

# set vmnics state in vswitch0 to vmnic0 active

/bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0

# Set network settings for system to static IP

network --bootproto=static --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=x385001.ibm.aessatl.arrow.com --nameserver=172.20.12.100,172.20.13.100 --addvmportgroup=1

%post --interpreter=busybox

# stop USB Arbitrator service to access USB device in ESXi Shell

/etc/init.d/usbarbitrator stop

%firstboot --interpreter=busybox

# enable & start SSH

vim-cmd hostsvc/enable_ssh

vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell

vim-cmd hostsvc/enable_esx_shell

vim-cmd hostsvc/start_esx_shell

# Suppress ESXi Shell warning

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

Repeat / copy above and save for list of nodes or answer file options you wish.  These will be called later on based on menu as kernel options

Step 3: Modify base vSphere installation to enable menu

Example: nodes 1-4  boot options from separate kickstart answer files

This is documented two paths… legacy and EFI as they are very different.

From shell (DOS or Linux) in the directory root of the USB key created…..

Legacy

EFI

# Make backup of isolinux.cfg

cp /isolinux.cfg  isolinux.bak

# modify isolinux.cfg to have menu

edit isolinux.cfg

DEFAULT menu.c32

MENU TITLE Lenovo_ESXi6.0_20150420 Arrow HyperConverge Boot Menu

NOHALT 1

PROMPT 0

TIMEOUT 80

LABEL hddboot

LOCALBOOT 0x80

MENU LABEL ^Boot from local disk

LABEL install node1

KERNEL mboot.c32

APPEND -c BOOT1.CFG

MENU LABEL Lenovo_ESXi6.0_20150420 Node1 ^Installer

LABEL install node2

KERNEL mboot.c32

APPEND -c BOOT2.CFG

MENU LABEL Lenovo_ESXi6.0_20150420 Node2 ^Installer

LABEL install node3

KERNEL mboot.c32

APPEND -c BOOT3.CFG

MENU LABEL Lenovo_ESXi6.0_20150420 Node3 ^Installer

LABEL install node4

KERNEL mboot.c32

APPEND -c BOOT4.CFG

MENU LABEL Lenovo_ESXi6.0_20150420 Node4 ^Installer

# modify boot.cfg which has kernel and module options to be node specific on kickstart file called

cp boot.cfg BOOT1.CFG

edit BOOT1.CFG

bootstate=0

title=Loading ESXi installer

kernel=/tboot.b00

#kernelopt=runweasel

kernelopt=ks=usb:/NODE1.KS

modules=/b.b00 --- /jumpstrt.gz <snip> --- /imgpayld.tgz

build=

updated=0

# save files..  making both boot config and kickstart files for each "node" or menu option

# We have to replace the simplistic EFI boot system from VMWare

# EFI boot https://sourceforge.net/projects/refind/?source=typ_redirect

# Documentation for menu http://www.rodsbooks.com/refind/configfile.html

# rename vmware's boot file as it will be used later

mv /efi/boot/bootx64.efi /efi/boot/esxboot.efi

# Download rEFInd and unzip

cp /tmp/refind-bin-0.10.3\refind/refind_x64.efi <usb vsphere>/efi/boot/bootx64.efi

pastedImage_4.png

# Now with the EFI replaced we need a menu file

# Note: the options in this menu file do NOT replace weasel but just append to kernel options in the base boot.cfg (hense why we never touch it when modifications made for legacy boot side)

# We also say timeout is infiite…aka requires user to select. As well as not to scan for any file *.efi and add dynamically to the boot menu

edit /efi/boot/refind.conf

timeout 0

scanfor manual

menuentry "Lenovo_ESXi6.0_20150420 Node1" {

    loader \EFI\boot\esxboot.efi

    options "ks=usb:/NODE1.KS"

}

menuentry "Lenovo_ESXi6.0_20150420 Node2" {

    loader \EFI\boot\esxboot.efi

    options "ks=usb:/NODE2.KS"

}

# save files..  making both boot menu option and kickstart files for each "node" or menu option

* Name files all in upper case https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10263...

gilles974
Contributor
Contributor

Hello,

it's a very interesting post.

I'm trying to implement the same but it does not work...

I have the boot menu but as if my refind.conf is not used...

Do i need to reference the refind.conf somewhere? (you seems to mention the boot.cfg but i can't see in the boot.cfg you show)

Thanks for your help!!!

Regards

Reply
0 Kudos
forcerk
Enthusiast
Enthusiast

It is the same for me. I do get the rEFInd boot menu but non of created menuentry.

Reply
0 Kudos
gilles974
Contributor
Contributor

Hi,

sorry for not giving news on this.

It happen to be working as explained on physical hardware, unfortunately not on virtual machine.

I suspect that it's because the mkisof command i have does not respect the ISOxxxx norm but unfortunately i was never able to sort it of :smileycry:

Regards


Gilles

Reply
0 Kudos
justShyOf8
Contributor
Contributor

you mentioned appending the refind.conf to the boot.cfg, but haven't found any doc describing how to do that; I tried the following(didn't work):

kernelopt=runweasel cdromBoot append -c refind.conf

I keep getting the error

Configuration error while parsing /EFI/VMware/boot.cfg

Fatal error: 15 (Not found)

any help would be greatly appreciated Smiley Happy

Reply
0 Kudos
CreaThor
Contributor
Contributor

Hi everybody,

Did anyone ever figure this out? I'm having the same issue as described before -- the installation successfully enters the rEFInd boot menu; however, all changes made to refind.conf are completely ignored.

Any help is much appreciated!

Reply
0 Kudos
jhg03a
Contributor
Contributor

I couldn't find a solution either for uefi and really wanted to get secure boot going as well, so I solved it for me and took notes for others.  It's not perfect or optimal but did the job for me and I don't think, as a process, it's tied to any particular VMware versioning.  In my testing it worked with both VMware ESXi 7u3 and 8 GA on a HPE Proliant Gen10 Plus V2.

https://gist.github.com/jhg03a/7eed0b0f97c91a46b9cf65e22da7ab9f

Thanks @ArrowSIVAC for getting me pointed in the right direction.

Reply
0 Kudos