Unable to expnad a datastore

[stopeject]

I started using this server with a 32TB RAID5. I created a 32TB datastore. Later I added two more drives to the array, so it expanded to 54TB. I can't find the way to expand my datastore to 54TB or create another one.

https://scrn.li/Gn8k5Dt3wELhrN

The device browser shows that the RAID array is fully occupied by a partition

https://scrn.li/4lqZFoZ2Aew3A0

What can I do about it short of destroying the existing datastore, which would be a major hassle?

Thank you.

[Alex_Romeo]

Hi,

can't you create a new LUN instead of expanding the existing datastore? If there are no particular reasons, it seems to me even more logical.

ARomeo

[stopeject]

I can't because the existing partition takes the entire disk. The partition increased in size to match the new disk size, but the associated datastore did not.

[Alex_Romeo]

VMware Knowledge Base

Resolution

To increase the capacity of a VMFS datastore:

1. Open vCenter session to the Datacenter/Cluster.
2. Select the Hosts and Clusters view and click the Rescan All link.
3. Switch to the Datastores and Datastore Clusters view.
4. Select each volume you wish to increase and select the Properties link.
5. Ensure the Extent shows in the Extents window, and reflects the current free space reported by the storage array.
6. Click the Increase button and select the extent to resize, then click Next.
7. Ensure the Total Capacity reflects the new capacity.

Alternatively, you may directly connect to ESXi host and perform a rescan.

How to Expand VMware ESXi Datastore Capacity

ARomeo

[stopeject]

Yes, this is where I started. If I pick the "Add an extent to existing VMFS datastore" it offers to put it on the boot SD card, as it is the only disk with unpartitioned space.

If I pick the "Expand an existing VMFS datastore extent " I get "No devices with free space message".

I think the problem is that my partition takes the entire RAID, but its size isn't in sync with the datastore size. I don't know what to do about it.

[a_p_]

If the datastore expansion didn'd work correctly in the first attempt, and only the partition size has been modified, you may need to grow the VMFS from the command line (see e.g. https://kb.vmware.com/s/article/2002461​, step 13). Please ensure that you read, and understand the whole procedure, and do the checks before running the growfs command. This is to ensure that the partition is in the proper state for the expansion.

André

[stopeject]

The problem is that the partition already shows 54TB, while the datastore shows 32TB.

Both GUI and the command line seem to agree

[user@Absolut:~] vmkfstools -P "/vmfs/volumes/RAID5"

VMFS-6.81 (Raw Major Version: 24) file system spanning 1 partitions.

File system label (if any): RAID5

Mode: public

Capacity 35184372088832 (33554432 file blocks * 1048576), 14460171321344 (13790294 blocks) avail, max supported file size 70368744177664

UUID: 5b56f434-018cd946-a551-e4434b023e66

Partitions spanned (on "lvm"):

        naa.6d0946606d80620022e9afb75fde9e1d:1

Is Native Snapshot Capable: NO

[user@Absolut:~]  partedUtil get "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d:1"

7294342 255 63 117183606785

[user@Absolut:~] partedUtil get "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d"

7294342 255 63 117183610880

1 2048 117183608832 0 0

[user@Absolut:~] partedUtil getptbl "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d"

gpt

7294342 255 63 117183610880

1 2048 117183608832 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Do I need to shrink the partition to match the datastore size for the extra space to become usable?

[a_p_]

The problem is that the partition already shows 54TB, while the datastore shows 32TB.

That's why I mentioned step 13 from the KB article. The command in this step will resize VMFS Datastore.

André

[stopeject]

Unfortunately it doesn't work

vmkfstools --growfs "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d:1" "/vmfs/devices/disks/naa.6d0946606d80

620022e9afb75fde9e1d:1"

Address temporarily unmapped

Error: Bad address

[a_p_]

That doesn't look so good.

Can you please share the output of:

partedUtil getUsableSectors "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d"

to see whether it matches tha partition information?

André

[stopeject]

partedUtil getUsableSectors "/vmfs/devices/disks/naa.6d0946606d80620022e9afb75fde9e1d"

34 117183610846

Thanks for your help. My expertise on the low level disk operations is a bit rusty and a stand alone ESXi can't be backed up properly, so I'm trying to be conservative with what I'm doing.

[a_p_]

The last sector doesn't match the value reported by the partition, but since the partition is a little bit smaller than possible, this should not cause the issue.

If that was a test system, I'd say let's fix the partition information to see whether this helps, but as for now, I'd really like to have continuum​'s opinion, i.e. whether he has seen this error message before.

André

[continuum]

Hi Andre

I wish we had the email address of one of the VMFS engineers to ask for help in a case like this.

I think that any operation that changes the partitiontable or the VMFS filesystem is an unacceptable risk if the user has no full backup of the VMs.

Given the size of the datastore a full backup can take days and so the only suggestion I have at this time is to declare the 2 added disks as collateral damage and call it a day.

If you try any operation please create a vmfs header dump first and send it my way - from the dump we could create a list of dd-commands to be able to extractt he content even if any operation fails and results in an unreadable datastore.

Anyway - dont try any further stunts without a full backup.

Wonder how VMware gets away with an attitude that denies all documentation for essential functions  like this ...

The case is interesting though ... I would be interested in a closer look ...

Ulli

[stopeject]

Thanks. I figured moving further is too risky. Will have to play a long game of moving VMs to a different server (when I'll have it) and rebuilding this one. This is a downside of using a free version of a commercial software.

[continuum]

This has nothing to do with your use  of the free version.

This can happen to users with an expensive licence too.

Anyway - if you are interested I would like to look at it myself via Teamviewer or Anydesk ...

Ulli

[stopeject]

The free version has limited backup options, it restricts my options. Thanks for your remote assistance offer, I'll try to set something up.

[fl0pp]

Did you solve this issue? I have kind of the same problem.
A HPE Proliant ML350 Gen 9 running VMware 6.5 U1, with a datastore of 17,3TB.

Last week we bought some more disks and where going to expand the datastore according to VMware Knowledge Base, but run into problems.

Expanding the RAID was not a problem, it reports:

logicaldrive 1 (25.5 TB, RAID 5, OK)

The datastore reports 17,3TB usable size:

vmkfstools -P -h /vmfs/volumes/HDD/

VMFS-6.81 (Raw Major Version: 24) file system spanning 1 partitions.

File system label (if any): HDD

Mode: public

Capacity 16 TB, 1.3 TB available, file block size 1 MB, max supported file size 64 TB

UUID: 59dbbc5c-0854c15d-7565-70106fc72818

Partitions spanned (on "lvm"):

        naa.600508b1001ccd0c7ac243199dc6e601:1

Is Native Snapshot Capable: NO

In the web GUI in Storage => Devices => Local HP Disk reports 25,47TB capacity.

The problems started at step 13 in VMware Knowledge Base

vmkfstools --growfs /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601:1 /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601:1

Underlying device has no free space

Error: No space left on device

I searched the internet, and tried to re-mount the datastore, reboot, click "refresh"/"rescan" in the web interface, and re-run the growfs command, with no luck.
Then I stumbled upon VMware ESXi 6.5 Update 2 Release Notes

"PR 1986020: ESXi VMFS5 and VMFS6 file systems might fail to expand

ESXi VMFS5 and VMFS6 file systems with certain volume sizes might fail to expand due to an issue with the logical volume manager (LVM). A VMFS6 volume fails to expand with a size greater than 16 TB. This issue is resolved in this release."

Then we upgraded to 6.5.0 U3, but are stuck with another error:

vmkfstools --growfs /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601:1 /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601:1

Address temporarily unmapped

Error: Bad address

partedUtil get /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601

3404761 255 63 54697488560

1 2048 54697488526 0 0

partedUtil get /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601:1

3404761 255 63 54697486479

partedUtil getUsableSectors /vmfs/devices/disks/naa.600508b1001ccd0c7ac243199dc6e601

34 54697488526

vmkernel.log when I run growfs:

2019-11-10T19:22:23.957Z cpu3:74931)Vol3: 743: Failed to expand file system HDD (Address temporarily unmapped), but logical volume was successfully extended. Will try to expand file system nexttime(s) volume is opened.

2019-11-10T19:22:23.957Z cpu3:74931)Vol3: 1025: Unable to register file system HDD for APD timeout notifications: Already exists

2019-11-10T19:22:23.957Z cpu3:74931)LVM: 10683: No unused capacity on device naa.600508b1001ccd0c7ac243199dc6e601:1

2019-11-10T19:22:23.957Z cpu3:74931)LVM: 10596: Error adding space (0) on device naa.600508b1001ccd0c7ac243199dc6e601:1 to volume 59dbbc5c-fc632e3e-823c-70106fc72818: Underlying device has no free space

2019-11-10T19:22:23.961Z cpu3:74931)Vol3: 743: Failed to expand file system HDD (Address temporarily unmapped), but logical volume was successfully extended. Will try to expand file system nexttime(s) volume is opened.

Any idea how I should expand the datastore?

[stopeject]

I haven't made any progress with this issue. I'll probably leave it as it is for now, and will try to replicate the VMs and rebuild the entire datastore when I'll have enough storage and time to do it. Your problem seems a bit different from mine, my datastore isn't limited to 16TB.