Attempting to add a VMFS partition in ESXi v8 results in "Failed to create VMFS datastore"

[swinster]

Hi,

I am trying to add a new datastore on a large local RAID disk for ESXi v8, but it fails with the error :

Failed to create VMFS datastore RAID6_Local - Failed to update disk partitions for /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38

The array used to contain a single VMF partition, however, when I installed ESXi v8, I somehow targeted it for the OSDATA partition, so I managed to re-partition the disk and wipe everything and then add a standalone VMFSL datastore for the OSDATA :(. I thought it would add a new directory, but obviously not. Now, I simply want to get new systems up and running again. 

Currently, the volume looks like this:

Clicking "Finish" to add the new partition fails with the error shown above

The only posts I have read that suggest a resolution point to re-partition the entire drive which is not really going to be a thing. 

[a_p_]

Not sure why you want to have an OSDATA partition on this LUN, but that's up to you.
The GUI will only let you create a datastore on an empty LUN. If you want to have an additional VMFS datastore on the LUN, you'l. have to create the partition, and format it manually from the command line.

André

[swinster]

Thanks for the guidance on the CLI. I will check this out tomorrow.

To be honest, creating the OSDATA partition on the array was not meant to be done. This is a home lab, and the LUN is an array attached to a physical RAID card in the host, but it is the only local persistent disk in the host. The OS boot partitions are on a mirror SD card (yes, I realise these are being phased out of support). It was a cockup that led to writing the OSDATA partition to this array, but now the **bleep** up is complete, I might as well go with it. As far as I could see from https://kb.vmware.com/s/article/85685, the OSDATA partition should be created on a physical device, not USB or SD Card, and going forward, the entire ESXi system should be on a persistent disk. I will see if I can get a small SSD for the job, but for now, it is what it is.

[swinster]

Unfortunately, the partition/disk seems to be set to read-only, and I cannot delete the partition. I went out a purchased a second SSD so I could create a new ESXi installation and ensure that the partition on the large local persistent disk (seen above) would be able to be removed. but it doesn't want to play ball.

Here is what I have tried:

ls -ltrh /vmfs/devices/disks/
total 5757200368
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048:8 -> naa.6f4ee0802f1512002b459235011d6ded:8
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048:7 -> naa.6f4ee0802f1512002b459235011d6ded:7
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048:6 -> naa.6f4ee0802f1512002b459235011d6ded:6
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048:5 -> naa.6f4ee0802f1512002b459235011d6ded:5
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048:1 -> naa.6f4ee0802f1512002b459235011d6ded:1
lrwxrwxrwx 1 root root 36 Jan 3 20:11 vml.02000000006f4ee0802f1512002b459235011d6ded504552432048 -> naa.6f4ee0802f1512002b459235011d6ded
lrwxrwxrwx 1 root root 38 Jan 3 20:11 vml.020000000061866da0ac21fa002ad086944b997d38504552432048:7 -> naa.61866da0ac21fa002ad086944b997d38:7
lrwxrwxrwx 1 root root 36 Jan 3 20:11 vml.020000000061866da0ac21fa002ad086944b997d38504552432048 -> naa.61866da0ac21fa002ad086944b997d38
-rw------- 1 root root 765.7G Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded:8
-rw------- 1 root root 119.9G Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded:7
-rw------- 1 root root 4.0G Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded:6
-rw------- 1 root root 4.0G Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded:5
-rw------- 1 root root 100.0M Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded:1
-rw------- 1 root root 893.8G Jan 3 20:11 naa.6f4ee0802f1512002b459235011d6ded
-rw------- 1 root root 128.0G Jan 3 20:11 naa.61866da0ac21fa002ad086944b997d38:7
-rw------- 1 root root 3.5T Jan 3 20:11 naa.61866da0ac21fa002ad086944b997d38

The disk in question is `naa.61866da0ac21fa002ad086944b997d38`, and the partition I want to remove is `naa.61866da0ac21fa002ad086944b997d38:7`.

partedUtil partinfo "/vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38" 7
Partition Number: 7
Start sector: 2048
End sector: 268435455
Partition Type GUID: 4EB2EA3978554790A79EFAE495E21F8D
Partition Unique GUID: 2C3FB6286A2545B5AB41D6981A0C4EA7
Partition Filesystem Type: vmfsl
Partition attributes: 0

However, trying to do this using `partedUtil` results in:

partedUtil delete "/vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38" 7
Error: Read-only file system during write on /dev/disks/naa.61866da0ac21fa002ad086944b997d38
Unable to delete partition 7 from device /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38

Checking out some previous posts, I also tried the following:

Setting the `PreventVMFSOverwrite` to `0`.

esxcfg-advcfg --get /Disk/PreventVMFSOverwrite
Value of PreventVMFSOverwrite is 1

esxcfg-advcfg --set 0 /Disk/PreventVMFSOverwrite
Value of PreventVMFSOverwrite is 0

Then, attempting to use `dd` to re-write the beginning of the partition table:

dd if=/dev/zero of="/vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38" bs=512 count=34 conv=notrunc
dd: /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38: Read-only file system

I looked to see if there was a core dump partition assigned to the disk, but as this is effectively a new ESXi instalation, there isn't:

esxcli system coredump partition get
Active:
Configured:

So ultimately, I'm stumped.

It seems I might need a third party until to complete this 😞

[a_p_ · Accepted solution]

I was curious, and reproduced this in my lab.

It looks like ESXi somehow protects the "old" OSDATA/VMFSL partition, even though a new one has been created with the new installation. After some trial&error I found a way to get rid of the old partition, and was able to create a new VMFS datastore on the device.
Please note that this may not be officially supported, but it worked in my lab:

1. Format the old OSDATA partition with a VMFS filesystem:
   vmkfstools -C vmfs6 -S OLD-OSDATA /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38:7
2. Reset the partition table
   partedUtil mklabel /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38 gpt
3. Rescan the storage (not the upper-case "-V")
   vmkfstools -V
4. Verify the result
   partedUtil getptbl /vmfs/devices/disks/naa.61866da0ac21fa002ad086944b997d38

After that I was able to create a new datastore from the GUI (which must be reloaded due to caching).

André

[swinster]

@a_p_ , very good! The commands helped remove the old VMFSL partition, but for some reason, even after re-loading the browser, I could not create a new data store, albeit the disk was reported as being 100% free space. In the GUI, I had to again "clear the partition table", and then I could re-create a data store. 

Many thanks for your help for getting me to this stage.

[AshishParmar]

Hi,

I tried this solutions but its showing access denied, below is what I am getting when logged in through SSH to the console.

Any advise for this please ?

__

[Admin@MTC-G10-01:~] vmkfstools -C vmfs6 -S OLD-OSDATA /vmfs/devices/disks/naa.6
00508b1001c4208427f2159257a1e56
create fs deviceName:'/vmfs/devices/disks/naa.600508b1001c4208427f2159257a1e56', fsShortName:'vmfs6', fsName:'OLD-OSDATA'
deviceFullPath:/dev/disks/naa.600508b1001c4208427f2159257a1e56 deviceFile:naa.60 0508b1001c4208427f2159257a1e56
ATS on device /dev/disks/naa.600508b1001c4208427f2159257a1e56: not supported
.
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs6 file system on "naa.600508b1001c4208427f2159257a1e56" with blockS ize 1048576, unmapGranularity 1048576, unmapPriority default and volume label "O LD-OSDATA".
Failed to create VMFS on device naa.600508b1001c4208427f2159257a1e56
/vmfs/devices/disks/naa.600508b1001c4208427f2159257a1e56: Permission denied. (Have you set the partition type to 0xfb?)
Error: Permission denied

[a_p_]

You need to append the partition number (e.g. ":n") to the device name.

André

[brian1948]

Deleting an VMFSL partition was resolved quickest with booting into RHEL <any-REL> boot USB Flash device (<5m).

• Use Rufus to create the boot image from the RHEL ISO to a bootable USB Flash.
• Select Recovery shell.
• fdisk /dev/sdc
  • d to delete the partition
  • w to write/save the changes.

This worked well, when the ESXi tools were failing (ESXi Host Client, partedUtil, esxi fdisk, esxi dd, vmkfstools and objtool).