Configuration ist not preserved after reboot on free Esxi 6.0

[SiegfriedKurz]

Hello,

last August, I've updated our (Free) Esxi 5.5 to version 6.0.

First, everything seems to be working fine.

But after a while, I found out that configuration changes were not preserved after a reboot of the host.

Configuration changes are very rare in our company, for that it took a while to notice the problem, so I am not 100% sure, that this problem stems from the update.

Hardware Information:

ProLiant ML350p Gen8

iLO Self-Test Results:

Supported host

Power Management Controller         Version 3.3

....

Embedded Flash/SD-CARD         Controller firmware revision 2.10.00 /8 GB SD-CARD in internal slot

CPLD - PAL1         ProLiant ML350p Gen8 SAS Programmable Logic Device version 0x0C

CPLD - PAL0         ProLiant ML350p Gen8 System Programmable Logic Device version 0x4D

....

software information:

vmware -vl

VMware ESXi 6.0.0 build-9313334

VMware ESXi 6.0.0 Update 3

esxcfg-info -e

boot type: visor-usb

After doing some reading in the vmware kb I found out:

[root@esxi-hit:~] ls -la /*boot*

lrwxrwxrwx    1 root     root             4 Apr 23 13:19 /bootbank -> /tmp

-r--r--r--    1 root     root        332055 Jul 27  2018 /bootpart.gz

[root@esxi-hit:/etc/rc.local.d] vmkfstools -P /altbootbank

Could not open /altbootbank

Error: No such file or directory

[root@esxi-hit:/etc/rc.local.d] grep bootbank  /var/log/esxupdate.log

2019-04-23T06:44:12Z esxupdate: HostImage: INFO: Installer <class 'vmware.esximage.Installer.BootBankInstaller.BootBankInstaller'> was not initiated - reason: altbootbank is invalid: Error in loading boot.cfg from bootbank /bootbank: Error parsing bootbank boot.cfg file /bootbank/boot.cfg: [Errno 2] No such file or directory: '/bootbank/boot.cfg'

So, it seems that there is something wrong with the "Embedded Flash/SD-CARD"

What can I do to get rid of the problem?

Thanks in advance.

best regards from germany

Siegfried

[johncol]

You could try powering-off the ESXi host, reseat the SD Card and check then, are you using Nimble Storage (NCM) by any chance?

[SiegfriedKurz]

Hello johncol,

>You could try powering-off the ESXi host, reseat the SD Card and check then,

I'll try this in the evening.

> are you using Nimble Storage (NCM) by any chance?

No, should I?

I've tried to install it with:

[root@esxi-hit:/var/log] esxcli software vib install -d http://update.nimblestorage.com/esx6/ncm

but I got:

[StatelessError]

The transaction is not supported: VIB Nimble_bootbank_nimble-ncs_6.0.0-600005 cannot be live installed.

So, You think it ist a hardware problem?

But the iLO Self-Test Results shows OK for the SD-Card:

SD-Card Status: Present: 8 GB

best regards from germany

Siegfried

[johncol]

No, my question about Nimble was that there is a known issue with that Nimble Controller.HP and bootbank----> tmp

Try the re-seat of the card and a reboot first

[SiegfriedKurz]

Hello johncol,

> Try the re-seat the card and a reboot first

I did that, it didn't help.

best regards from germany

Siegfried

[SureshKumarMuth]

Have you tried this KB

VMware Knowledge Base

[johncol]

thats a boot from SAN kb issue, here its an SD card boot media...

Is altbootbank pointing to tmp as well?  Could you Shift R on reboot to see if there is an option to revert back to an earlier build, would the 5.5 be the last version you had installed? You could be looking at backing up the configuration here and doing a reinstall with the same version unfortunatley

[SiegfriedKurz]

Hello Sureshkumar M,

> Have you tried this KB

Yes, when I tried the first command, I got this result:

[root@esxi-hit:~] localcli --plugin-dir=/usr/lib/vmware/esxcli/int/ boot system restore --bootbanks

VmkCtl: Did not find a valid boot device, symlinking /bootbank to /tmp

best regards from germany

Siegfried

[SiegfriedKurz]

Hello johncol,

> Is altbootbank pointing to tmp as well? 

Unfortunately, there is no /altbootbank:

[root@esxi-hit:~] ls -la /*boot*

lrwxrwxrwx    1 root     root             4 Apr 24 14:33 /bootbank -> /tmp

-r--r--r--    1 root     root        332055 Jul 27  2018 /bootpart.gz

best regards from germany

Siegfried

[SureshKumarMuth]

The symlinks are not proper..perhaps you have to reinstall your ESXi host to fix the issue...

[SiegfriedKurz]

Hello Sureshkumar,

> perhaps you have to reinstall your ESXi host to fix the issue...

OK, but unfortunately, I've never done a reinstall of an Esxi-Host, so I have some questions:

If the SD-Card is the problem, where comes the config in /tmp from?

How can I be sure that the SD-Card is ok?

Is it possible to test the SD-Card before the installation?

Should I probably try to install Esxi to an USB-Stick?

Best regards from germany

Siegfried

[SureshKumarMuth]

Generally, state.tgz is the file which contain all the information of your ESXi config and it is located in /bootbank  which in turn stored in the  boot device (SD card).Since bootbank is pointing to /tmp any changes what you do is getting stored in bootbank --> /tmp and during the reboot it is not saved in the SD card and config changes are not stored permanently, this is my understanding. Now the Filesytem structure got changed somehow so the partitions are not listed as expected. Reinstallation will fix all these issues and will bring back the default folder structure.

I dont think this is an issue with your SD card, you may try to take a current config backup and reinstall the host and restore the config backup.

VMware Knowledge Base

[SiegfriedKurz]

Hello Sureshkumar,

> I dont think this is an issue with your SD card, you may try to take a current config backup and reinstall the host and restore the config backup.

Unfortunately, I've never done a reinstall of an ESXi.

OK, I got a config backup, but I won't need it, because theres is only IP-Configuration which has to be changed.

When I perform the reinstall, is there any danger of wiping out my datastore?

I think, the most important thing is that I have to use the option:

====================================

.....

(X) Install ESXi, preserving VMFS Datastore

.....

====================================

I've got a backup of all important VMs on our Synology NAS.

I'll use this installation image for my "ProLiant ML350p Gen8":

VMware-ESXi-6.0.0-Update3-9313334-HPE-preGen9-600.9.8.5.4-Sep2018.iso

Is there anything more I have to consider?

best regards from germany

Siegfried