5.5.0 upgrade to U3 fails with Errno 39

[HeadlessChicken]

Hello All,

Is anyone able to offer any assistance with a 5.5.0 upgrade problem , Errno 39 at 24% ?

I had a PSOD when I booted ESX - bora/vmkernel/core/bootModule.c:122

KB for this indicated reinstall ESX over the top.  That didn't work as ESX would not install, said it was corrupted.

However, ESX had previously been upgraded from 5.5.0 to u2.  I used the recovery option when ESX boots and everything was fine when it then reverted to old version, 5.5.0 2068190...  ESX is up and VM's are working (phew).

So - I decided to upgrade then to u3, even if at the least it gives me this recovery option again.

U3 upgrade fails though with "OSError: [Errno 39] Directory not empty: 'vmfs/volumes/f43c8f72-55a88ebf-96e1-0c7041e9414c/System Volume Information'

KB for THAT problem indicates remove state.XXXXX, but in /bootbank there is no state.XXXX (there is no /var/log/vua.log either).

The vfms volume referred to in the error is actually /altbootbank not /bootbank. In /altbootbank is ->

~ # cd /altbootbank/
/vmfs/volumes/f43c8f72-55a88ebf-96e1-0c7041e9414c # ls -alR
.:
total 257
drwxr-xr-x    1 root     root           512 Jan  5 11:10 ..
drwxr-xr-x    1 root     root             8 Jan  4 21:07 System Volume Information
-rwx------    1 root     root           196 Jan  5 01:13 imgdb.tgz

./System Volume Information:
total 513
drwxr-xr-x    1 root     root             8 Jan  4 21:07 .
drwxr-xr-x    1 root     root             8 Jan  1  1970 ..
-rwx------    1 root     root            76 Jan  4 21:07 IndexerVolumeGuid
-rwx------    1 root     root            12 Jan  4 21:07 WPSettings.dat
/vmfs/volumes/f43c8f72-55a88ebf-96e1-0c7041e9414c #

If I move imgdb.tgz somewhere else, it only wants to reinstall ESX not upgrade.

If I copy all of /bootbank to /altbootbank, I get the Errno 39 at 24% again.

Any clues anyone?

Thanks

HC.

[HeadlessChicken]

I was hoping this problem would magically disappear, but it hasn't.. so if anyone has any ideas - I'd love to hear them!

Thanks.

[swinster]

@HeadlessChicken - firstly, I understand this is a nearly 8-year-old post that refers to an EXSi version long since gone, however, I happened across this post multiple times when hitting exactly the same issue when upgrading an ESXi 6.7 host. The VMware KB article https://kb.vmware.com/s/article/2007163 or other posts such as https://knowaretech.com/2021/03/25/upgrading-a-vmware-esxi-host-fails-with-the-error/, whilst they refer to the same error number, the issue resolution is slightly different.

TL;DR - I believe the boot media used for ESXi had been mounted in a Windows system at some point which write the "System Volume Information" folder to the basic disk partition it can access (namely the symlinked folders of `/bootbank`, `/altbootbank` and `/store`). I think these folders are therefore unexpected by ESXi and this is why the upgrade fails. The simple resolution is to remove this folder and its contents entirely, for example:

rm /bootbank/System\ Volume\ Information\ -r
rm /altbootbank/System\ Volume\ Information\ -r
rm /store/System\ Volume\ Information\ -r

The reason I noted this is that I use a dual SD card module with mirrored redundancy but recently swapped the M/B and the mirror was broken. Of course, I mounted the cards in a handy Windows laptop to see what was on them to determine the correct card that contained the latest state, and in the act of doing so, I believe the "System Volume Infomation" folder was written to these three partitions.

The other alternative (which I ended up also doing) was to simply re-install and re-configure ESXi from scratch. As I had the ability to easily create and break the SD card mirror, I could ensure that I always had a card that I could boot from whilst I played with the other card. If things went wrong (and they did, several times), I could re-create the mirror from the working card and try again.

Hopefully, this may at least help someone else that sees the same issue. 

[swinster]

FWIW, on a fresh install of ESXi, the /altbootbank only contains a single file - boot.cfg. Whilst this file also exists within the /bootbank folder, they are slightly different in their content (likely due to them indicating which partition to boot from - although I have not researched this so do not understand exactly what that differences in values mean). I suspect, therefore, that simply copying the boot.cfg file from /bootbank to /altbootbank might end up causing so funnies. 

Just for reference, below are the contents of both boot.cfg files from a freshly installed ESXi 6.7U3. I'll ;leave it to you to note the differences and determine if they might make a difference or not.

/bootbank/boot.cfg

bootstate=0
title=Loading VMware ESXi
timeout=5
prefix=
kernel=b.b00
kernelopt= installerDiskDumpSlotSize=2560 no-auto-partition
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- chardevs.b00 --- user.b00 --- procfs.b00 --- uc_intel.b00 --- uc_amd.b00 --- uc_hygon.b00 --- vmx.v00 --- vim.v00 --- sb.v00 --- s.v00 --- dell_sha.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- dellemc_.v00 --- dellemc_.v01 --- lpfc.v00 --- i40en_en.v00 --- i40en.v00 --- icen.v00 --- igbn.v00 --- ixgben_e.v00 --- ixgben.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- qlnative.v00 --- qcnic.v00 --- qedentv_.v00 --- qedentv.v00 --- qedf.v00 --- qedi.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- sfvmk.v00 --- ata_liba.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- brcmfcoe.v00 --- char_ran.v00 --- ehci_ehc.v00 --- elxiscsi.v00 --- elxnet.v00 --- hid_hid.v00 --- iavmd.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- iser.v00 --- lpnic.v00 --- lsi_msgp.v01 --- lsi_msgp.v02 --- misc_cni.v00 --- misc_dri.v00 --- mtip32xx.v00 --- ne1000.v00 --- nenic.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cdc_.v00 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_fcoe.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_libf.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_usbn.v00 --- net_vmxn.v00 --- nfnic.v00 --- nhpsa.v00 --- ntg3.v00 --- nvme.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- ohci_usb.v00 --- pvscsi.v00 --- qflge.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_isc.v00 --- scsi_lib.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- shim_isc.v00 --- shim_isc.v01 --- shim_lib.v00 --- shim_lib.v01 --- shim_lib.v02 --- shim_lib.v03 --- shim_lib.v04 --- shim_lib.v05 --- shim_vmk.v00 --- shim_vmk.v01 --- shim_vmk.v02 --- smartpqi.v00 --- uhci_usb.v00 --- usb_stor.v00 --- usbcore_.v00 --- vmkata.v00 --- vmkfcoe.v00 --- vmkplexe.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- xhci_xhc.v00 --- elx_esx_.v00 --- btldr.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- esxupdt.v00 --- weaselin.t00 --- lsu_hp_h.v00 --- lsu_inte.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- lsu_smar.v00 --- native_m.v00 --- rste.v00 --- vmware_e.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- dell_con.v00 --- xorg.v00 --- imgdb.tgz --- state.tgz
build=6.7.0-19898906
updated=1

/altbootbank/boot.cfg

bootstate=3
title=Loading VMware ESXi
timeout=5
prefix=
kernel=b.b00
kernelopt= installerDiskDumpSlotSize=2560
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- chardevs.b00 --- user.b00 --- procfs.b00 --- uc_intel.b00 --- uc_amd.b00 --- uc_hygon.b00 --- vmx.v00 --- vim.v00 --- sb.v00 --- s.v00 --- dell_sha.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- dellemc_.v00 --- dellemc_.v01 --- lpfc.v00 --- i40en_en.v00 --- i40en.v00 --- icen.v00 --- igbn.v00 --- ixgben_e.v00 --- ixgben.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- qlnative.v00 --- qcnic.v00 --- qedentv_.v00 --- qedentv.v00 --- qedf.v00 --- qedi.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- sfvmk.v00 --- ata_liba.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- brcmfcoe.v00 --- char_ran.v00 --- ehci_ehc.v00 --- elxiscsi.v00 --- elxnet.v00 --- hid_hid.v00 --- iavmd.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- iser.v00 --- lpnic.v00 --- lsi_msgp.v01 --- lsi_msgp.v02 --- misc_cni.v00 --- misc_dri.v00 --- mtip32xx.v00 --- ne1000.v00 --- nenic.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cdc_.v00 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_fcoe.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_libf.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_usbn.v00 --- net_vmxn.v00 --- nfnic.v00 --- nhpsa.v00 --- ntg3.v00 --- nvme.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- ohci_usb.v00 --- pvscsi.v00 --- qflge.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_isc.v00 --- scsi_lib.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- shim_isc.v00 --- shim_isc.v01 --- shim_lib.v00 --- shim_lib.v01 --- shim_lib.v02 --- shim_lib.v03 --- shim_lib.v04 --- shim_lib.v05 --- shim_vmk.v00 --- shim_vmk.v01 --- shim_vmk.v02 --- smartpqi.v00 --- uhci_usb.v00 --- usb_stor.v00 --- usbcore_.v00 --- vmkata.v00 --- vmkfcoe.v00 --- vmkplexe.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- xhci_xhc.v00 --- elx_esx_.v00 --- btldr.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- esxupdt.v00 --- weaselin.t00 --- lsu_hp_h.v00 --- lsu_inte.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- lsu_smar.v00 --- native_m.v00 --- rste.v00 --- vmware_e.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- dell_con.v00 --- xorg.v00 --- imgdb.tgz
build=6.7.0-19898906
updated=2