deeploy
Contributor
Contributor

Impossible to restore or extend VMFS

Initial situation :
2 x 500Go SSD
1 SSD is taken for VMS with datastore linked to SSD 1.
Second is taken for the ESXI (probably)

List VM :

Database
Dev
Prod
API
web
API-ALL

Objective :
While we don't have enough space on the first SSD, we wanted to use the second SSD space disk.

Manip :
Via ESXI UI :
Stockage ; SSD 2 ; actions ; Update partitions ; select VMFS 469,56 Go ; delete partition ; save

Datastore ; main ; Raise capacity ; add extend to datastore VMFS ; select SSD 2 ; maximum allocation ; finish
- error message : Error extend datastore VMFS main - Impossible to update host configuration

Via ssh ESXI :
I tried what commands are here.
Commands history :

partedUtil getUsableSectors "/vmfs/devices/disks/t10.NVMe____WDC_CL_SN720_SDAQNTW2D512G2D2000__________BCE3B8488B441B00"
34 1000215182


partedUtil getptbl "/vmfs/devices/disks/t10.NVMe____WDC_CL_SN720_SDAQNTW2D512G2D2000__________BCE3B8488B441B00"
gpt
62260 255 63 1000215216
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

partedUtil add command doesn't work anymore :
reference https://kb.vmware.com/s/article/1036609

Tried :

partedUtil setptbl "/vmfs/devices/disks/t10.NVMe____WDC_CL_SN720_SDAQNTW2D512G2D2000__________BCE3B8488B441B00" gpt "3 15472640 1000215182 AA31E02A400F11DB9590000C2911D1B8 0"
gpt
0 0 0 0
3 15472640 1000215182 AA31E02A400F11DB9590000C2911D1B8 0
Error: Read-only file system during write on /dev/disks/t10.NVMe____WDC_CL_SN720_SDAQNTW2D512G2D2000__________BCE3B8488B441B00
SetPtableGpt: Unable to commit to disk

==================================================================================

From here, there was a problem with the database VM. It goes read-only mode for the entire system. The autocompletion with tab doesn't work, neither the apt-get update. Impossible to start the mysql service. Everytime problem with the read-only mode.

According to that problem, some solutions are :
https://askubuntu.com/questions/197459/how-to-fix-sudo-unable-to-open-read-only-file-system
https://askubuntu.com/questions/822887/boot-failed-failed-to-start-remount-root-and-kernel-file-syst...
https://askubuntu.com/questions/287021/how-to-fix-read-only-file-system-error-when-i-run-something-a...

I've tested everything but nothing worked. Disclaimer, I tested it without rescue mode. Like it's a VM I couldn't boot with an ISO (even configuring correctly the boot through the parameters console. And impossible to enter in the grub mode (even with the shift key). It's too quick to boot and shift at the same. It's simply doesn't work.

None of the other VMs seems to have been impacted (still sometimes I got some error relating to threads). But the VM seems to work more or less fine, and they are on read and write normal mode.

Many reboots were done for the VMs (Like the fsck command tutorial recommanded it).
And 2 reboots of ESXI were done too.

=====================================================================================

Hypothesis :

- We have only less than 10Go of free ssd space on the SSD1. Is it possible that read-only mode comes because of lack of space ?
- Is it possible the VMFS delete process occured the ESXi and therefore we got problem on the database VM ? If so, why the read-only locked system only for that VM and not the others ?

=====================================================================================

3 Solutions :
1 - Delete database VM, but way too much time to reconfigure everything (more than 14 databases with different access)
2 - Find a solution to remount read and write mode (impossible to access the rescue mode boot)
3 - Find a solution to extend and/or restore the VMFS deleted, and enjoy the free space for other VMS

Thanks in advance.

 

 

0 Kudos
0 Replies