Configured scratch location doesn't work after upgrade to ESXi 6.5

[lukaszc]

Hi,
My configured scratch location for some reason doesn't work after upgrading to ESXi 6.5.

I set up parameter ScratchConfig.ConfiguredScratchLocation to a persistent storage. The folder exists and worked before the upgrade.

After each reboot ScratchConfig.CurrentScratchLocation is always set to /scratch while ScratchConfig.ConfiguredScratchLocation is /vmfs/volumes/volume01/scratches/host01

I'm not sure if this is related but to upgrade ESXi I had to provide boot parameter preferVmklinux=TRUE as my USB drive wasn't visible and later I run these commands:

esxcli system settings kernel set -s preferVmklinux -v FALSE

esxcli system module set --enabled=false –m vmkusb

Thanks for your help in advance,

Lukasz

[rm_bk]

18 months later this problem still isn't fixed.  I've got three HPE Gen9 blades on fully patched 6.7U3 working fine and a fourth blade that utterly refuses to persist a custom scratch location across a reboot.  elx-libelxima is at 12.0.1188 on all of them.   Has there been any news on this?

[PatrickDLong]

+1 for "still an issue"  I just upgraded a ton of HPE diskless servers from 6.7U3 to 7.0.1 and no matter HOW I try to set the scratch config on SOME of them it reverts upon reboot. These are all identical h/w servers, installed in an identical way, upgraded via identical methods (image profile) with identical versions of the HPE custom installer - and some of them will persist the setting across reboots and others will not.  Maddening!  I've tried all the known resolutions - upgraded ELX drivers, disabled/uninstalled ELX drivers, set scratch in the html GUI, set it via Set-VMHostAdvancedConfiguration (now deprecated) set it via Get-AdvancedSetting, made sure bootbanks were synchronized with /sbin/auto-backup.sh prior to rebooting,-read all the related KB's, changed the value when the host is in maint mode and NOT in maint mode, deleted and recreated the destination scratch folder, pointed to other folders entirely -and yet *nothing* works to resolve this for the random servers that will NOT persist this setting across reboots.  For some of the upgraded servers, every time I reboot my advanced settings show:

ScratchConfig.ConfiguredScratchLocation     /vmfs/volumes/5axxxx0b-6cxxxx4e-8f5b-daxxxxxxx018/.locker-hostname

ScratchConfig.CurrentScratchLocation     /tmp/_osdata4ivqeeqn  (effectively /tmp/scratch for those of you still on 6.x)

I'm at my wits end with this issue that has been present in so many ESXi releases.  How can you write a value to the configuration of a host, have that command respond that it's written successfully, but then this written change is GONE on reboot as if you had never written it? How can 0's and 1's be this non-deterministic?

[continuum]

1. set scratch location via the webinterface - wait a few seconds
2. verifify if the change made it into the file /etc/vmware/locker.conf

3. now run backup.sh
4. set host in maintenance mode

5. reboot

Thats the only way that works for me.

Verify if the changes really made it into state.tgz by downloading state.tgz and extracting it. Then you can check what is written in locker.conf

Ulli

[PatrickDLong]

@continuum Thank you for your response.  I have just retried  per the steps you have given and no luck. 

I created a NEW directory in same shared storage location as all my other .locker locations for all other hosts, /vmfs/volumes/5axxxx0b-6cxxxx4e-8f5b-daxxxxxxx018/.locker-TESTING and added it to the ScratchConfig.ConfiguredScratchLocation advanced setting via web interface; waited a few seconds.

Checked /etc/vmware/locker.conf and the changes ARE reflected there. Only one line that looks like this, my specified location value plus a <space> zero appended:
/vmfs/volumes/5axxxx0b-6cxxxx4e-8f5b-daxxxxxxx018/.locker-TESTING 0

ran /sbin/auto-backup.sh
Files /etc/vmware/dvsdata.db and /tmp/auto-backup.2225731//etc/vmware/dvsdata.db differ
Saving current state in /bootbank
Clock updated.
Time: 14:21:08 Date: 02/12/2021 UTC

After reboot, host shows ScratchConfig.CurrentScratchLocation as /tmp/_osdatai6lvxssn, NOT the /vmfs/volumes/5axxxx0b-6cxxxx4e-8f5b-daxxxxxxx018/.locker-TESTING that I specified.

I extracted state.tgz from the bootbank and it shows /vmfs/volumes/5axxxx0b-6cxxxx4e-8f5b-daxxxxxxx018/.locker-TESTING 0

Additional info that may be helpful - this system is a diskless system booting off microSD card - the two bootbank partitions appear to be located on the SD disk as expected and as such are persistent; obviously we do not want high-volume writes (scratch) going to the SD card but rather to high-endurance storage device like shared datastore on SAN  There is a new kb https://kb.vmware.com/s/article/2149444 where these may be placed in /tmp/ after reboot due to storage-path-claim service issues but I am not experiencing issues with the bootbanks, only the scratch location - for some reason this system just continues to place my scratch location in /tmp RAMdisk instead of any persistent location that I specify. 

[root@Lxxxxxxxx:~] cd bootbank
[root@Lxxxxxxxx:/vmfs/volumes/601c4757-f9272e20-3e4f-7a94036000d0] cd ..
[root@Lxxxxxxxx:~] cd altbootbank
[root@Lxxxxxxxx:/vmfs/volumes/e28ed3b2-ac4c1b5a-9ab3-0188fa271c64] cd ..
[root@Lxxxxxxxx:~] cd /scratch
[root@Lxxxxxxxx:/tmp/_osdatai6lvxssn]

Any other thoughts or suggestions? It accepts my setting, it saves my setting, it just does not actually SET my setting and I'm flummoxed. I do have an SR created, going through the standard "send us your logs" hoops now. 

[PatrickDLong]

My issue has been escalated to VMware Engineering. - will update here when I have news to share.

[PatrickDLong]

@continuumThis issue turned out to be related to https://kb.vmware.com/s/article/81434  "Slow storage device discovery causes bootbank/scratch to not get mounted"  Even though both of my bootbanks WERE loading (USB-based micro-SD boot device on motherboard), it turns out I needed to introduce a delay into the storage-path-claim service discovery process due to the large number of SAN volumes attached to these hosts.  On many hosts the volume where I had configured the scratch location was not being enumerated/discovered in the default amount of time, so I added devListStabilityCount=30 to the kernelopt= line in boot.conf in bootbank and altbootbank and voila - scratch partition was discovered appropriately and used after every subsequent reboot. 

Additional background info available in https://kb.vmware.com/s/article/2149444  Bootbank loads in /tmp/ after reboot of ESXi 7.0 Update 1 host