vSAN - Clear Disk Partitions

[JeremeyWise]

Had vSAN get totally corrupt.  After days of trying to get it back up ..  ... giving up.  I got the VMs off but now trying to clean out so I could create fresh.    But to do that, I need to simply wipe out partition data on disk.

This is example host:  Has 120GB SSD for vSAN cache and 1TB disk for vSAN Data

GUI fails to delete

Name

Identifier

LUN

Type

Capacity

Datastore

Operational State

Hardware Acceleration

Drive Type

Adapter

Owner

Physical Location

Sector Format

Perennially Reserved

Transport

Local ATA Disk (t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________)

t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________

disk

931.51 GB

vsanDatastore

Attached

Not supported

Flash

vmhba0

HPP

512n

No

Block Adapter

Local ATA Disk (t10.ATA_____KINGSTON_SA400S37120G___________________50026B77838F133D____)

t10.ATA_____KINGSTON_SA400S37120G___________________50026B77838F133D____

disk

111.79 GB

vsanDatastore

Attached

Not supported

Flash

vmhba0

HPP

512n

No

Block Adapter

SSH to host to try to wipe disk so vSAN wizard will run again.

[root@thor:/dev/disks] ls -alh |grep 2926
-rw------- 1 root root 931.5G Feb 7 15:44 t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________
-rw------- 1 root root 2.0M Feb 7 15:44 t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________:1
-rw------- 1 root root 931.5G Feb 7 15:44 t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________:2
lrwxrwxrwx 1 root root 73 Feb 7 15:44 vml.01000000003139313036413830323932362020202020202020574443202057 -> t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________
lrwxrwxrwx 1 root root 75 Feb 7 15:44 vml.01000000003139313036413830323932362020202020202020574443202057:1 -> t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________:1
lrwxrwxrwx 1 root root 75 Feb 7 15:44 vml.01000000003139313036413830323932362020202020202020574443202057:2 -> t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________:2
[root@thor:/dev/disks] dd if=/dev/zero of=t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________
dd: can't open 't10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________': Function not implemented
[root@thor:/dev/disks] partedUtil delete t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________ 1
Error: Read-only file system during write on /dev/disks/t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________
Unable to delete partition 1 from device t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________

[root@thor:/dev/disks] partedUtil delete t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________ 2
Error: Read-only file system during write on /dev/disks/t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________
Unable to delete partition 2 from device t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________19106A802926________

[root@thor:/dev/disks]

Besides cracking each server and pulling drives 😛   Which would really be bad...  Are there other ideas on wiping disks.

Thanks

[TheBobkin]

@JeremeyWise The devices are returning read-only for reason why did isn't working and also why vSAN Disk Management is unable to remove the partitions. I have observed this from disks in odd crashed states which is sometimes resolved by cold-booting the host but other times this is just a symptom of a broken disk (and isn't detected after reboot or remains unwritable).

It is within the realm of possibility that while vSAN can't use/manage the devices properly that some vSAN process is preventing overwrite - if same state after cold-boot, you could try temporarily disabling the vSAN modules to rule this out: https://kb.vmware.com/s/article/66996

[JeremeyWise]

I ended up having to boot LiveLinux USB to clear partitions.  Even after doing this.. I was able to run wizard to add disk back to vSAN .. but when services came back up.. it seemed to still think it was working with old VSAN definition.  I could see "vSANDatastore"  but I could not create folder / save data to the volume.  I disabled all services again..  and was right back where vSAN all shut off...  no VMFS volume showing.. but..  partitions again on the servers that I could not clean out.   Back to trying to cycle each server one at a time to wipe disks manually.

Quesiton:

1) Is there any steps / guide on how to clean out all vSAN and start over? 

[JeremeyWise]

Update and new ask:

After manual wipe.  Created "fresh new vSAN"  but it failed..  Two hosts keep getting all data and third one does not then vSAN goes into a bunch of errors about not being balanced.   I then disable VSAN serviecs again and trying to start over.

I put all servers into maintennece mode one at a time and reboot and this allows me now to format disk to clean out old vSAN.

Running Fresh setup for fourth time.. Trying to find pattern on why the one host fails

I notice two hosts have disk creation logs 

Then the one host which has issues just hangs at 20% for hours

Attached is logs from fresh boot -> delete partions ->  Then run vSAN setup wizard.

Only notable items is that both disk "...Failed to get freeMB from UUID. Roll back...." and "..Cannot parse UUID from /vsantraces..."

Open to suggestions: