Scratch location won't be honored/changed after boot SD card cloning

[ctucci]

Running ESXi v7/VCSA v7...

Using Dell Internal Dual SD card for ESXi boot. One of the cards failed and I replaced it with exact same make and model but it must be a different batch because Dell BIOS complains the new card is smaller than the card I am replacing, thus it won't let me re-build to it. It does indeed seem the new card is a few MB smaller than the original card. I have two of these new cards though which do match of course so want to migrate ESXi install to those.

I was able to clone my original still working card 1 to new card 1 using an HDD block cloning software on windows. ESXi actually boots off this new cloned card and Dell BIOS then copies that card to new card 2 and says mirror is functional again.

ESXi boots back fully, connects to vCenter, everything works, EXCEPT it complains that there is no core dump location set anymore. However, the scratch config advanced setting for that host stills is filled with the datastore I had scratch going to, but the "current" scratch location advanced setting reads "/scratch". If I change the config scratch location to another folder or other datastore, it takes the change, I go to maintenance mode and reboot, and system comes back up still saying current location is "/scratch". Not only that, but the config location is now set back to the original datastore folder it was on. So, something is not letting me chage scratch setting anymore and something is not honoring the configured location, and it's defaulting to "/scratch". Everything else works fine, it's just this problem not letting me set or change scratch away from "/scratch". The datastore that scratch was going to and it claims is set to still, is still there, available working, and unchanged during this entire process.

If I boot ESXi back up with the original card 1, all works normally and scratch can be changed.

Is there a way to reset scratch settings somehow? Something is messed up during the clone I guess with that? It's just strange that every time I reboot it persists to the original configured scratch location, even if I change it, and even so it doesn't even use that location, it sets current to "/scratch" and shows the error on vCenter about no core dump location being configured.

[msripada]

Am not sure if there is any error in hostd log when configuring..

Ideally the scratch should be in ./tmp/scratch instead of /scratch..

There was an issue in 6.x due to elx vib VMware Knowledge Base  and am not sure if you are seeing this issue in your new

thanks,

MS

[continuum]

ESXi 7 stores the path to the partition that will be used for /scratch on next boot in /etc/vmware/locker.conf

That file will be automatically backuped every now and then into the tarball /bootbank/state.tgz

During next boot state.tgz will be expanded into /etc and the conf-files will be read during boot.

So when you want to create a clone of a ESXi boot media make sure that you first check if your current config will work on a cloned system.

Check

/etc/vmware/locker.conf

/etc/vmware/esx.conf

If the config is universal run backup.sh to update the state.tgz in the bootbank.

Then create the clone.

[ctucci]

Definitely will try that tomorrow, with the backup.sh. I was assuming every time the setting gets changed it automatically gets written to bootbank. If it's not right away, then where is that setting getting stored? Because on my original boot SD, I was always able to and still can change scratch and it takes it after a reboot and persists, etc... works properly.

Is there a better way to maybe clone the original SD (bigger) to new SD (slightly smaller)? I used gparted live to attempt to copy the original to new and when it tries to copy the partition table over, it errors out and says the last partition on the original SD is too big to fit on the new SD. The three partitions before the last are called BOOT, BOOTBANK 1 and BOOTBANK 2 and those can fit easily, on new SD so not sure what this last approx. 12GB partition is. gparted live won't even show filesystem for that one, just says unknown. Because of that, I can't even downsize that last partition slightly on the original card to then make 1:1 clone to new card easier.

[continuum]

> Is there a better way to maybe clone the original SD (bigger) to new SD (slightly smaller)?

Oh dear ... I tried the same recently and it was a real pain. It is possible but you basically have to recreate a new 4th partition (VMFSL).

Even resizing the 2 FAT partitions is not trivial at all

Honestly - the short answer (and the best one if you consider your nerves as a valuable resource ) is simply : buy a new SD-card that matches size or is bigger.

[ctucci]

The backup.sh didn't make a difference. I think the correct scratch value was already in the bootbank anyway because I didn't change it from what it has been set to for a long time. I just simply cloned the card and when booting the new card, esxi just wasn't honoring what was listed in the config scratch advanced option. Almost like it thought I never set an option, but it was still there for sure in that locker.conf file.

What your post about re-creating the VMFS-L volume did make me think to try was installing a fresh copy of ESXi on the new, smaller SD card. This naturally and correctly created partitions esxi needs on the smaller card, including that 4th partition. Then I used gparted live to copy and paste only BOOT, BOOTBANK 1, and BOOTBANK 2 from the original card to the new card. So essentially, using the installer to just create a valid partition scheme, then copying over the bootbanks. This worked, esxi boots off the new smaller SD card with all config still present, and scratch partition using the correct value that is configured.

So, something about purely trying to clone the bigger card to smaller, messes up the way esxi tries to read and set the scratch partition to the configured value from esxi/vcenter.

Thanks for the help!

[ctucci]

Unfortunately, that log didn't provide anything other than the same error that shows in vcenter, that the scratch location isn't set.

[continuum]

Found this yesterday ...

ESXi selects one of these scratch locations during startup in this order of preference:

1. The location configured in the /etc/vmware/locker.conf configuration file, set by the ScratchConfig.ConfiguredScratchLocation configuration option, as in this article.
2. A Fat16 filesystem of at least 4 GB on the Local Boot device.
3. A Fat16 filesystem of at least 4 GB on a Local device.
4. A VMFS datastore on a Local device, in a .locker/ directory.
5. A ramdisk at /tmp/scratch/.

see VMware Knowledge Base

If your clone-attempts truncate the VMFSL partition the partition probably is unmountable at boot-time.

So option 4 is expected to fail ...

[ctucci]

That's what I thought at first, but the whole time condition #1 there was met. A valid path on a datastore was defined in that option and locker.conf reflected that valid path, yet esxi still tried to use the /scratch ramdisk. And after rebooting even if I changed the path to another datastore, it would revert to the old datastore path (which was still accessible anyway) that was set when the original install was done on original SD last year. THat's why I was hopeful your backup.sh suggestion would work because maybe that path was only in vcenter and not getting written to the bootbank for some reason before cloning, but it didn't seem to make a difference.

I know what I was trying to do isn't officially supported/the right way but still weird it wouldn't honor the configured location for scratch.