bootbank cannot be found at path /bootbank

[bvanarse]

Hi

In vCenter for one of the ESXi server I am getting  error as "bootbank cannot be found at path /bootbank"

Reboot is temporary solution for this.

Out of two devices there is none USB device at boot time

vmhba32 usb-storage link-n/a usb.vmhba32 () USB

And below is the error details :-

SCSI Level: 2 Is Pseudo: false Status: dead timeout

Is RDM Capable: false Is Removable: true

Can some one please suggest permanent solution if any.

Thanks &  Regards

[bspagna89]

Hi,

I've seen an issue where bootbank errors were showing up on hosts after upgrading to 6.0.. This was particular to HP and there's a KB article referencing it. Before we proceed, I think we need to know a little bit more about your environment..

1. What kind of servers/model?

2. Were there any changes done? upgrades/firmware/etc..?

3. Are you booting (or trying to) from a SD card or USB stick? Do you have a method of checking the health of that device? (HP and Dell both have a way to check in DRAC/iLO).

4. Were there any changes done to the BIOS of the server?

Let me know..

-Brian

[RajeevVCP4]

• If you accidentally mask a boot device, the /bootbank partition on your ESXi system might become unavailable. The /bootbank partition contains core hypervisor code and is critical for the functioning of the ESXi

Please provide out put for these commands.

• Use the esxcfg-scsidevs -a command to determine which adapter is used for the boot device.
• Run the following command to obtain a list of all claim rules for the boot device adapter. The following example uses vmhba1 adapter:
• esxcli corestorage claimrule list|grep vmhba1

[Dee006]

Hi,

Welcome to VMware Communities,

Are you using your USB drive to boot the ESXi host?If yes try to change the USB drive and enable/disable the USB after host reboot.

Reference Link:

Important information about the new ESXi 6.5 USB driver vmkusb, and the legacy USB drivers (2147650)...

Unable to connect to host with vSphere Client or vCenter - My VMware blog

[BenLiebowitz]

I have to agree with bspagna89​, this sounds like the same issue we had with our HP Blades. 

Please confirm what hardware you're running on and what version of ESXi so we can help troubleshoot your issue.

[philanimk]

Hi BenLiebowitz​ and bspagna89​

I am seeing the same alarm in my environment.

Overview

We have 3x HPE BL 660c blades with an HPE 3Par (StorServ 7200) providing storage. Blades one and two of these are Gen 8, blade 3 is Gen9. We are running ESXi 5.5 in our datacenter. I ran esxcfg-scsidevs -a as suggested in this thread and below is the output from the three blades:

Blade1:

====

~ # esxcfg-scsidevs -a

vmhba0  lpfc              link-up   fc.50060b0000c26201:50060b0000c26200    (0:33:0.2) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba1  lpfc              link-up   fc.50060b0000c26203:50060b0000c26202    (0:33:0.3) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

~ # exit

====

Blade 2:

====

~ # esxcfg-scsidevs -a

vmhba0  lpfc              link-up   fc.50060b0000c26205:50060b0000c26204    (0:33:0.2) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba1  lpfc              link-up   fc.50060b0000c26207:50060b0000c26206    (0:33:0.3) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba2  hpsa              link-n/a  sas.5001438028499260                    (0:34:0.0) Hewlett-Packard Company Smart Array P220i

~ # exit

====

Blade 3:

====

~ # esxcfg-scsidevs -a

vmhba0  hpsa              link-n/a  sas.50014380339fd040                    (0:34:0.0) Hewlett-Packard Company Smart Array P220i

vmhba32 bnx2fc            link-up   fcoe.50060b0000c2623d:50060b0000c2623c  (0:33:0.2) Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter

vmhba33 bnx2fc            link-up   fcoe.50060b0000c2623f:50060b0000c2623e  (0:33:0.3) Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter

~ # exit

====

What changed

The StorServ is having additional hard drives installed and HPE recommended we upgrade the OS on the StorServ prior to this. One of the requirements of the OS upgrade was a change in Host Persona Configuration; previously this was set to "Generic". On the StorServ the following steps were carried out for Blades 1 and 2:

1. Under the Host edit menu, Unexport all virtual volumes
2. Remove the Paths from the host
3. Set the Host OS to "ESX 4.x/5/x"
4. Add the paths
5. Export virtual volumes to the hypervisor

Since the above changes were made, we are now seeing there are alarms from VMware:

====

[VMware vCenter - Alarm alarm.HostErrorAlarm] Issue detected on w.x.y.z in datacenter: Bootbank cannot be found at path '/bootbank' (2017-05-15T07:32:52.348Z cpu57:319614292)

====

Please let me know if you need more information and thank you for answering our questions.

/philani

[bspagna89]

Hi,

This sounds a lot like the issue we were running into with our HP G8/G9 Blades. Have a look at this KB below .. So far upgrading the iLO has resolved this issue for us...

HP Gen 8/9 lose access to device backing boot filesystem (2144283) | VMware KB

Let me know.

-Brian

[philanimk]

Hi bspagna89​,

Thanks for this. I'll go through everything and let you know of results once done.

/philani

[philanimk]

Hi bspagna89​,

I've upgraded to iLO 2.4 and I'm still getting notifications from VMware. Could it have to do with the installation mounts being volumes which are resident on the 3Par (not SD cards or memory sticks)?

[pbalderos]

Has anyone seen this issue on Dell power edge servers? I don't think its limited to HP as we are having the same issue. Any resolution?

[wkwicker]

Yes, I am having this issue on Dell M520, M620 and M630 blades.  This just seemed to start and has happened to about 6 systems.  I am running ESXi 5.5 and most of the 6 systems have all the firmware up to date.  One of the M630's is brand new but after a reboot has not done it again.

[planewalker]

December 28, at 4 am. One of BL460g8 with the installed version 3116895 "froze" with the same error. Firmware 2.40 Dec 02 2015

[TEST1981]

gotta same issue in here.

ILO 2.55

BIOS P89 v2.0
we have one other exact configuration blade in cluster on BIOS P89 v 2,52 and there is no issue.

[planewalker]

The similar found in a logs vcenter

Target: 10.0.5.96

Stateless event alarm

Alarm Definition:

([Event alarm expression: Host error] OR [Event alarm expression: Host warning])

Event details:

Issue detected on 10.0.5.96 in ALF-DATACENTER: Bootbank cannot be found at path '/bootbank'

(2017-12-27T19:03:01.257Z cpu24:206300420)

[lagasss]

Same problem here. Dell R430 running Esxi 6.0 Dell image on an SD card.

[SajjadAxia]

Same problem here as well on ESXI 6.5 with SD card boot on a HP DL120 G9... All the latest bios / fw have been applied.

[jiridvorak1]

Today during restart

"P[OST Error: 324-HPE Dual microSD Device Error - Both microSD cards have failed. Action: Replace the HPE Dual microSD device."

And system prevents to boot - HP DL360 G9

Seems this is known issue of HP internal USB of bad series which need to be replaced

Check

https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=7796454&docLocale=en_US&docId=emr_na-c0536...

[jiridvorak1]

So this is what need to be replaced on every HP DL9.....otherwise you are going to big issue (as me today)

[SajjadAxia]

How can you even check if you have that part without physically opening the server? iLO only says Embedded Flash/SD-CARD

[jiridvorak1]

to be honest,i think that opening can is only way to get SN to get patched from HP.According all information it seems,that all 001/002 are eligble for change.So to prevent issue i woudl recommend make it asap.