Sigh... for support reasons of a software vendor I have to maintain a 5.5u2 cluster.
Trying to stage for class / POC environment (hyperconvergence so has to work on hardware level), so I have to handle hardware
Goal:
1) Use kickstart structure answer files to load the nodes
2) Create a boot menu from vSphere 5.5u2 (oem customized versions) to allow user to boot USB image over IPMI, select node they are on, and it feeds it the correct ks.cfg file
I followed various guides / notes:
vSphere Boot Menu Guides |
---|
Template attached but details for fields and assumptions below. Create menu for the list of nodes to each customization answer file Documentation: http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html |
But either it is a translation issue of my understanding of their notes, or issue with 5.5u2 or ???
###
Step 1: Create bootable USB vSphere OS key on 2GB key
I named key label "esx5U2_auto_install"
I followed instructions and used tool rufus
I connected and used OEM vSphere cutomization ISO for vSphere 5.5u2
Step 2: Create answer file response files
Example is node 1-4 cluster.
on USB create directory "KS" and in that create files "NODE1.cfg" -"NODE4.cfg"
Example file for node1
node1 ks.cfg |
---|
# Accept vSphere license ULA vmaccepteula # Set root password rootpw password # Clear drive partition on first device in device list clearpart --firstdisk --overwritevmfs # Set installation target to be first drive it finds in device list install --firstdisk --overwritevmfs # Set static IP for vmnic0 per Spectrum Accelerate table network --bootproto=static --device=vmnic0 --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=node1 -- nameserver=172.20.12.100,172.20.13.100 vlanid=11 # Reboot host reboot # Post Installation goodies # Enable SSH and Tech Suport modes vim-cmd hostsvc/enable_esx_shell vim-cmd hostsvc/start_esx_shell vim-cmd hostsvc/enable_ssh vim-cmd hostsvc/start_ssh esxcli network firewall ruleset set --ruleset-id sshClient --enabled yes |
Step 3: Create menu for nodes
Edit file "isolinux.cfg" for above host menu options. Save this file in root of USB key.
Design is to boot over USB attached drive through the IMM (or local attached USB)
DEFAULT menu.c32 MENU TITLE 5.5u2 Boot Menu NOHALT 1 PROMPT 0 TIMEOUT 80 # LABEL hddboot LOCALBOOT 0x80 MENU LABEL ^Boot from local disk # LABEL IBM Spectrum Accelerate Node 1 Install KERNEL mboot.c32 APPEND -c boot.cfg ks=usb:/KS/NODE1.cfg +++ MENU LABEL ^1 Node1 Install # LABEL IBM Spectrum Accelerate Node 2 Install KERNEL mboot.c32 APPEND -c boot.cfg ks=usb:/KS/NODE2.CFG +++ MENU LABEL ^2 Node2 Install # LABEL IBM Spectrum Accelerate Node 3 Install KERNEL mboot.c32 APPEND -c boot.cfg ks=usb:/KS/NODE3.CFG +++ MENU LABEL ^3 Node3 Install # |
#############
But... When I boot it just runs normal script to install vSphere from basic answer file
What I think I am missing is what do I need to do to the boot.cfg file to get it to call the menu (DEFAULT menu.c32)
As usual, all help is appreciated.
PS: I also tried to just effect the base load for just test of answer file (aka without fancy menu) by modification of "boot.cfg" as noted below...and it booted normal installer and ignored answers
boot.cfg - node 1 direct ks.cfg call |
---|
bootstate=0 title=Loading ESXi installer kernel=/tboot.b00 kernelopt=ks=/KS/NODE1.cfg modules=/b.b00 --- /jumpstrt.gz --- /useropts.gz --- /k.b00 --- /chardevs.b00 --- /a.b00 --- /user.b00 --- /sb.v00 --- /s.v00 --- /brcm.v00 --- /misc_cni.v00 --- /net_bnx2.v00 --- /net_bnx2.v01 --- /net_cnic.v00 --- /net_tg3.v00 --- /scsi_bnx.v00 --- /scsi_bnx.v01 --- /brcdprov.v00 --- /net_bna.v00 --- /scsi_bfa.v00 --- /elxnet.v00 --- /emulex_c.v00 --- /ima_be2i.v00 --- /lpfc.v00 --- /scsi_be2.v00 --- /ianet_ci.v00 --- /net_igb.v00 --- /net_ixgb.v00 --- /lsiprovi.v00 --- /scsi_meg.v00 --- /scsi_mpt.v00 --- /scsi_mpt.v01 --- /concrete.v00 --- /filetran.v00 --- /fupb.v00 --- /fwupdate.v00 --- /hwckvm.v00 --- /ilfu.v00 --- /immpasst.v00 --- /pciinfo.v00 --- /soibms.v00 --- /mlnxprov.v00 --- /net_mlx4.v00 --- /net_mlx4.v01 --- /ima_qla4.v00 --- /net_qlcn.v00 --- /net_qlge.v00 --- /qlnative.v00 --- /qlogic_c.v00 --- /scsi_qla.v00 --- /ata_pata.v00 --- /ata_pata.v01 --- /ata_pata.v02 --- /ata_pata.v03 --- /ata_pata.v04 --- /ata_pata.v05 --- /ata_pata.v06 --- /ata_pata.v07 --- /block_cc.v00 --- /ehci_ehc.v00 --- /weaselin.t00 --- /esx_dvfi.v00 --- /xlibs.v00 --- /ipmi_ipm.v00 --- /ipmi_ipm.v01 --- /ipmi_ipm.v02 --- /misc_dri.v00 --- /mtip32xx.v00 --- /net_be2n.v00 --- /net_e100.v00 --- /net_e100.v01 --- /net_enic.v00 --- /net_forc.v00 --- /net_nx_n.v00 --- /net_vmxn.v00 --- /ohci_usb.v00 --- /rste.v00 --- /sata_ahc.v00 --- /sata_ata.v00 --- /sata_sat.v00 --- /sata_sat.v01 --- /sata_sat.v02 --- /sata_sat.v03 --- /sata_sat.v04 --- /scsi_aac.v00 --- /scsi_adp.v00 --- /scsi_aic.v00 --- /scsi_fni.v00 --- /scsi_hps.v00 --- /scsi_ips.v00 --- /scsi_lpf.v00 --- /scsi_meg.v01 --- /scsi_meg.v02 --- /scsi_mpt.v02 --- /scsi_mpt.v03 --- /scsi_qla.v01 --- /uhci_usb.v00 --- /tools.t00 --- /lnvcusto.v00 --- /xorg.v00 --- /imgdb.tgz --- /imgpayld.tgz build= updated=0 |
So done quite a bit of poking around and think I am close but missing a few pieces.
What is the boot mode is very important. The server I have does support "Legacy" boot mode. As such the vSphere installer displays and boots differently from install media.
Legacy Boot | UEFI Boot |
---|---|
Boot from ISO / USB.. notice menu .. get option to interact and choose boot option (and so kickstart answer) | Boot from ISO / USB.. notice no menu .. just boots. No menu. No options. |
When I add / change customized files (see below) | When I add / change customized files (see below) |
What I need to understand is the different files and what the boot order is UEFI vs Legacy boot..... Trying to sketch it out.
As such. With UEFI only boot ability I made the following file changes: Example is paired down to just two nodes
syslinux.cfg | isolinux.cfg | boot1.cfg | NODE1.cfg |
---|---|---|---|
default menu.c32 prompt 0 menu title Arrow|IBM|Lenovo|Brocade HyperConverge 5.5u2 timeout 100 label unetbootindefault menu label ^Boot from local disk kernel /ubnkern append initrd=/ubninit label ubnentry0 menu label ^Node1 #kernel /ubnkern kernel /MBOOT.C32 append initrd=/ubninit -c boot1.cfg label ubnentry1 menu label ^Node2 #kernel /ubnkern kernel /MBOOT.C32 append initrd=/ubninit -c boot2.cfg label ubnentry2 | EFAULT menu.c32 MENU TITLE HyperConverge 5.5u2 NOHALT 1 PROMPT 0 TIMEOUT 80 LABEL hddboot LOCALBOOT 0x80 MENU LABEL ^Boot from local disk LABEL NODE1 KERNEL mboot.c32 APPEND -c boot1.cfg +++ MENU LABEL ^1 NODE1 Install LABEL NODE2 KERNEL mboot.c32 APPEND -c boot2.cfg +++ MENU LABEL ^1 NODE2 Install | bootstate=0 title=Loading ESXi installer Node1 kernel=/tboot.b00 #kernelopt=runweasel kernelopt=ks=usb:/NODE1.cfg modules=/b.b00 --- /jumpstrt.gz <snip>--- /imgpayld.tgz build= updated=0 | ## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster # v0.01 #### vmaccepteula #clear any existing partition information clearpart --alldrives --overwritevmfs #specify to install on first usb drive it finds install --firstdisk=usb --overwritevmfs # Set root password rootpw Pa55w0rd! reboot # set vmnics state in vswitch0 to vmnic0 active /bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0 <snip> |
*Thought process was that I copy all those files in both / and /efi/boot to capture compute nodes weather they boot UEFI or Legacy.
I thought that this would work because: EFI boot hardware -> /efi/boot/bootx64.efi -> Calls menu.c32 (binary)-> calls syslinux.cfg (options and user selection) Defines boot kernel "MBOOT.C32" and then initrd /ubninit with options file "boot#.cfg" -> calls boot#.cfg for kernel file "tboot.b00" and kernel options, kickstart file /NODE#.cfg and modules outside initrd-> calls kickstartfile for auto answer files for OS installation.
Question:
1) Can UEFI boot mode servers still use a "menu" in vSphere installation?
2) Can someone direct me to what the flow for the boot structure is such that I can debug the boot menu?
First is a baseline that systems when they boot can boot "Legacy" (BIOS boot loader) or EFI The boot path they take impacts the flow of the boot order, and by necessity, the ability to call your installation scripts
I documented much of this via a thread on the vmware forums https://communities.vmware.com/message/2596896#2596896
* Notice that in EFI, vmware bypassed any parsing of isolinux.cfg and as such there is no menu option. They converted "/mboot.c32" to "/efi/boot/bootx64.efi"
(32bit EFI option also exists but same concept)
Goal of this is to create a USB boot image/ key , which weather Legacy or EFI will boot up, allow user selection of "node ##" and then install via a user defined kickstart answer file
#################################################
Step 1: Create bootable USB vSphere OS key
2GB key
I named it "esx5U2_auto_install"
I followed instructions and used tool rufus
I connected and used Lenovo vSphere customization ISO for vSphere 5.5u2 per this location https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5097258
* The above instructions are based on Lenovo, can be used for ANY OEM / or standard vSphere boot image
Step 2: Create vSphere answer file for installation
Template attached but details for fields and assumptions below.
Create menu for the list of nodes to each customization answer file
Documentation:
http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html
Example: node1 answer file called "NODE1.KS" which will install on first USB drive and install with basic hostname, IP, username, password, SSH and esxi shell enabled (for later scripting needs), DNS and NTP. Keep file names shorter then eight characters and all in upper case to avoid ISO / USB format issues.
Save these files in / <root> of the usb key
## Kickstart vSphere 5.5u2 Answer file for Node1 of cluster # v0.5 #### vmaccepteula #clear any existing partition information clearpart --alldrives --overwritevmfs #specify to install on first usb drive it finds install --firstdisk=usb --overwritevmfs # Set root password rootpw Pa55w0rd! reboot # set vmnics state in vswitch0 to vmnic0 active /bin/esxcli network vswitch standard policy failover set --active-uplinks vmnic0 --vswitch-name vSwitch0 # Set network settings for system to static IP network --bootproto=static --ip=172.20.11.197 --netmask=255.255.255.0 --gateway=172.20.11.1 --hostname=x385001.ibm.aessatl.arrow.com --nameserver=172.20.12.100,172.20.13.100 --addvmportgroup=1 %post --interpreter=busybox # stop USB Arbitrator service to access USB device in ESXi Shell /etc/init.d/usbarbitrator stop %firstboot --interpreter=busybox # enable & start SSH vim-cmd hostsvc/enable_ssh vim-cmd hostsvc/start_ssh # enable & start ESXi Shell vim-cmd hostsvc/enable_esx_shell vim-cmd hostsvc/start_esx_shell # Suppress ESXi Shell warning esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1 |
Repeat / copy above and save for list of nodes or answer file options you wish. These will be called later on based on menu as kernel options
Step 3: Modify base vSphere installation to enable menu
Example: nodes 1-4 boot options from separate kickstart answer files
This is documented two paths… legacy and EFI as they are very different.
From shell (DOS or Linux) in the directory root of the USB key created…..
Legacy | EFI | |||
# Make backup of isolinux.cfg cp /isolinux.cfg isolinux.bak # modify isolinux.cfg to have menu edit isolinux.cfg
# modify boot.cfg which has kernel and module options to be node specific on kickstart file called cp boot.cfg BOOT1.CFG edit BOOT1.CFG
# save files.. making both boot config and kickstart files for each "node" or menu option | # We have to replace the simplistic EFI boot system from VMWare # EFI boot https://sourceforge.net/projects/refind/?source=typ_redirect # Documentation for menu http://www.rodsbooks.com/refind/configfile.html # rename vmware's boot file as it will be used later mv /efi/boot/bootx64.efi /efi/boot/esxboot.efi # Download rEFInd and unzip cp /tmp/refind-bin-0.10.3\refind/refind_x64.efi <usb vsphere>/efi/boot/bootx64.efi # Now with the EFI replaced we need a menu file # Note: the options in this menu file do NOT replace weasel but just append to kernel options in the base boot.cfg (hense why we never touch it when modifications made for legacy boot side) # We also say timeout is infiite…aka requires user to select. As well as not to scan for any file *.efi and add dynamically to the boot menu edit /efi/boot/refind.conf
# save files.. making both boot menu option and kickstart files for each "node" or menu option |
* Name files all in upper case https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10263...
Hello,
it's a very interesting post.
I'm trying to implement the same but it does not work...
I have the boot menu but as if my refind.conf is not used...
Do i need to reference the refind.conf somewhere? (you seems to mention the boot.cfg but i can't see in the boot.cfg you show)
Thanks for your help!!!
Regards
It is the same for me. I do get the rEFInd boot menu but non of created menuentry.
Hi,
sorry for not giving news on this.
It happen to be working as explained on physical hardware, unfortunately not on virtual machine.
I suspect that it's because the mkisof command i have does not respect the ISOxxxx norm but unfortunately i was never able to sort it of :smileycry:
Regards
Gilles
you mentioned appending the refind.conf to the boot.cfg, but haven't found any doc describing how to do that; I tried the following(didn't work):
kernelopt=runweasel cdromBoot append -c refind.conf
I keep getting the error
Configuration error while parsing /EFI/VMware/boot.cfg
Fatal error: 15 (Not found)
any help would be greatly appreciated
Hi everybody,
Did anyone ever figure this out? I'm having the same issue as described before -- the installation successfully enters the rEFInd boot menu; however, all changes made to refind.conf are completely ignored.
Any help is much appreciated!
I couldn't find a solution either for uefi and really wanted to get secure boot going as well, so I solved it for me and took notes for others. It's not perfect or optimal but did the job for me and I don't think, as a process, it's tied to any particular VMware versioning. In my testing it worked with both VMware ESXi 7u3 and 8 GA on a HPE Proliant Gen10 Plus V2.
https://gist.github.com/jhg03a/7eed0b0f97c91a46b9cf65e22da7ab9f
Thanks @ArrowSIVAC for getting me pointed in the right direction.