Changing scratch location destroyed my datastore

[jakorsme]

I installed 7.0 to usb and brought up esxi host

I configured a data store and created a vm which ran fine.

Then I went to change scratch to persistent storage in a datastore rather than the auto configured /tmp.

I changed it in the gui and that's where my problem began.

I designated my only datastore (the one holding my vm) as the scratch location without apparently designating a subdirectory.

So, it moved scratch fine and rebooted and came up. But, now my datastore and vm is gone.

The datastore's device is there, but is attached to LOCKER and OSDATA.

And, no matter what I do, I can't get scratch back on /tmp to see if esxi did something to corrupt my datastore.

[jakorsme]

Did I post this in the wrong category?

[Ardaneh]

As far as I know, there was an issue in ESXi 7.0 related to VMFS-L and locker partition. the issue made the locker partition corrupted. So please try to upgrade your ESXi to 7 Update 2c. this is from VMware:

"PR 2763986: If you configure the scratch partition on the root directory of a VMFS volume, upgrade to ESXi 7.0.x fails and you might lose data on the VMFS volume"

"ESXi 7.0 introduced the VMFS-L based ESX-OSData partition, which takes on the role of the legacy scratch partition, locker partition for VMware Tools, and core dump destination. If you configure the scratch partition on the root directory of a VMFS volume during an upgrade to ESXi 7.0.x, the system tries to convert the VMFS files into VMFS-L format to match the OS-Data partition requirements. As a result, the OS-Data partition might overwrite the VMFS volume and cause data loss.

This issue is resolved in this release (ESXi 7.0 Update 2c | 24 AUG 2021 | Build 18426014). The fix adds a check if the filesystem is VFAT before converting it into OS-DATA. For more information, see VMware knowledge base article 83647."

[jakorsme]

This sounds close in terms of the resultant datastore corruption, except that this was not an upgrade, it was a brand new install of 7.0U2a on supported hardware and the datastore was new and vmfs version 6. The issue occurred when I changed the scratch partition to the datastore's root. 

For instance, from the relase notes:"This issue is resolved in this release. The fix adds a check if the filesystem is VFAT before converting it into OS-DATA. For more information, see VMware knowledge base article 83647" 

It sounds like this might be an incomplete fix, as in my circumstance, the filesystem was not VFAT, but was VMFS6, so the fix would not have worked and the corruption would have still occurred.

I did also notice that it did convert the datastore's filesystem from VMFS6 to VMFSL. So maybe, even if I'd designated a sub-directory rather than root for scratch, it might have still converted the filesystem and corrupted my vm on that datastore anyway?

[pkvmw]

Hi @jakorsme,

as you've read in KB https://kb.vmware.com/s/article/83647 a fix (further checks) was added to ESXi 7.0 U2c, and you installed 7.0 U2a (so not containing the fix). As your VMFS6 is not VFAT, it would have not converted it to OS-DATA. 

Beside this, don't put the scratch location to a datastore root. To my knowledge it isn't best-practise.

Regards,
Patrik

[Ardaneh]

There is a possibility that your problem is related to the bug which was in the version you've installed in your environment and the root cause of the problem is when the scratch location is set up incorrectly to the root of a datastore, so I think it's not completely related to an upgrade situation.

There is always a possibility that some bugs were found with just a single cause, but there can be more causes to that bug. so I recommend it to upgrade your environment and try to create your datastore again and check if your problem was solved.

I believe after you upgrade your environment to the fixed version, there will be no probability of datastore corruption if you choose a subdirectory for scratch. we are using the same version and for the scratch path, we always use a subdirectory for each ESXi host and we don't have any problem.

I hope you could solve the problem. Please share the result

[jakorsme]

Yes - I think one simply can't put it on root or it repartitions the disk and erases your VMs. Pretty bad Vmware.

So - I think at this point, I have to reformat the disk to vmfs6 and then move scratch off the usb again, but this time make absolutely sure it's to a subdirectory. Then start putting VMs back. Then I can upgrade at some point.