14 Replies Latest reply on Nov 10, 2018 5:43 PM by ThompsG

    Can't grow VMFS Datastore

    facundocanepa Lurker

      Hello

       

      I wanted to expand the Datastore because it's running out of free space and the raid1 where it resides had about 50 GB unasigned (I let those 50 GB unused when I initially created the VMFS)

      Since I couldn't do it on the GUI (ncrease datastore capacity greyed out), I followed this instructions to do it on the CLI: https://michlstechblog.info/blog/esxi-expand-datastore-from-command-line/

       

      I could expand the partition and all went well until the part of growing the VMFS.

       

      Initial part:

       

      [root@esx:~] partedUtil get /vmfs/devices/disks/eui.b0269f4100d00000

      30377 255 63 488017920

      1 128 364644400 0 0

       

      [root@esx:~] partedUtil getUsableSectors /vmfs/devices/disks/eui.b0269f4100d00000

      34 488017886

       

      [root@esx:~] partedUtil resize /vmfs/devices/disks/eui.b0269f4100d00000 1 128 488017886

       

      [root@esxi:~] vmkfstools -V

       

       

      If I do as described, using the partition indicator ":1", I get a "not found" message.

      I if remove the ":1" from the second argument (destination?), I get a "Failed to get info from head device path"

       

      [root@esx:/dev/disks] vmkfstools --growfs /vmfs/devices/disks/eui.b0269f4100d00000:1 /vmfs/devices/disks/eui.b0269f4100d00000:1

      Not found

      Error: No such file or directory

      [root@esx:/dev/disks] vmkfstools --growfs /vmfs/devices/disks/eui.b0269f4100d00000:1 /vmfs/devices/disks/eui.b0269f4100d00000

      Failed to get info from head device path /dev/disks/eui.b0269f4100d00000.

      Error: No such file or directory

       

      I'm stucked here and don't know what to do

      The partition size is 232 GB but the Datastore is only 173 GB,,,

      Here some screenshots

      1.png

       

      2.png

       

      3.png

       

       

      Could you help me???

      Thanks in advance.

        • 1. Re: Can't grow VMFS Datastore
          rajen450m Hot Shot
          vExpert

          Hi,

          How  many ESXi hosts are this datastore connected and let us know the ESXi version to understand better.

          This can be due to inconsistent ESXi versions in the cluster.

           

          Regards,

          • 2. Re: Can't grow VMFS Datastore
            facundocanepa Lurker

            Hi Raj

             

            There is only one ESXi host using this datastore. It's a single server with two locally attached raid1 arrays: 2x 250GB SSD and 2x 3TB HDD.

            This ESXi is running 6.5.0 Build 4887370.

            The Datastore I'm trying to expand is the one located in the SSD array.

             

            Regards,

            Facundo

            • 3. Re: Can't grow VMFS Datastore
              rajen450m Hot Shot
              vExpert

              HI,

              Then its a bug that causing you this, you are running pretty old ESXi 6.5 version.

              You are still on the 6.5.0 A version, which was immediate released after GA.

              Please try to upgrade your host to the latest and try to expand.

               

              Regards,

              • 4. Re: Can't grow VMFS Datastore
                vFouad Enthusiast
                VMware Employees

                Looks like you almost got it all done!

                 

                Check out this VMware Knowledge Base

                 

                You missed step 11:

                Grow the VMFS Datastore in to the new space using the vmkfstools --growfs command, specifying the partition containing the target VMFS Datastore twice.

                 

                vmkfstools --growfs "/vmfs/devices/disks/device:partition" "/vmfs/devices/disks/device:partition"

                 

                For example:

                 

                vmkfstools --growfs "/vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:5" "/vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:5"

                • 5. Re: Can't grow VMFS Datastore
                  facundocanepa Lurker

                  Hi

                   

                  Thanks for the suggestion.

                  I'll get the server updated to the latest 6.7 version.

                  I'll report back here if that solved the issue.

                   

                  Regards,

                  Facundo

                  • 6. Re: Can't grow VMFS Datastore
                    facundocanepa Lurker

                    Hi all and thanks for the hints.

                     

                    rajen450m

                     

                    I've updated the Server to ESXi to 6.7.0 Build 10176752

                    The update went fine and I even got other issues solved (autostart of certain VMs that on 6.5 didn't work, now it does), but the storage problem remains the same.

                     

                    1.png

                    Partition Size: 232 GB

                    2.png

                     

                    Datastore size: 173 GB

                    3.png

                    Expanding the datastore: no devices with free space...

                    4.png

                     

                    vFouad

                     

                    I've actually tried that. The output looks like this:

                     

                    [root@ESXI:~] vmkfstools --growfs /vmfs/devices/disks/eui.b0269f4100d00000:1 /vmfs/devices/disks/eui.b0269f4100d00000

                    Failed to get info from head device path /dev/disks/eui.b0269f4100d00000.

                    Error: No such file or directory

                    [root@ESXI:~] vmkfstools --growfs /vmfs/devices/disks/eui.b0269f4100d00000:1 /vmfs/devices/disks/eui.b0269f4100d00000:1

                    Not found

                    Error: No such file or directory

                     

                    Additional info:

                     

                    [root@ESXI:~] vmkfstools -P /vmfs/volumes/OS-SSD/

                    VMFS-5.61 (Raw Major Version: 14) file system spanning 1 partitions.

                    File system label (if any): OS-SSD

                    Mode: public

                    Capacity 186562641920 (177920 file blocks * 1048576), 43003150336 (41011 blocks) avail, max supported file size 69201586814976

                    Disk Block Size: 512/512/0

                    UUID: 57f2a0d5-d3b64608-70db-0cc47ac81928

                    Partitions spanned (on "lvm"):

                            eui.b0269f4100d00000:1

                    Is Native Snapshot Capable: YES

                    [root@ESXI:~]

                     

                    [root@ESXI:~] partedUtil get /vmfs/devices/disks/eui.b0269f4100d00000

                    30377 255 63 488017920

                    1 128 488017886 0 0

                     

                     

                    So for some reason it still won't grow the FS

                     

                    I even tried to reduce the size of the partition by a few sectors, and event to it's original size (because originally I could see the non-partitioned, free space on the GUI devices view, maybe that's what VMware needs in order to expand a Datastore?), but this is anyway not working:

                    [root@ESXI:~] partedUtil resize /vmfs/devices/disks/eui.b0269f4100d00000 1 128 488017852

                    Error: Read-only file system during write on /dev/disks/eui.b0269f4100d00000

                     

                    Shit.

                    Any clues???

                    Thanks

                    • 7. Re: Can't grow VMFS Datastore
                      vFouad Enthusiast
                      VMware Employees

                      "I even tried to reduce the size of the partition by a few sectors, and event to it's original size (because originally I could see the non-partitioned, free space on the GUI devices view, maybe that's what VMware needs in order to expand a Datastore?), but this is anyway not working"

                       

                      To quote lost in space "Danger Will Robinson!"

                       

                      I wouldn't try to shrink the partition, generally bad things happen if you manage to do that....

                       

                      Can you confirm that the host you are using can see /vmfs/devices/disks/eui.b0269f4100d00000:1

                      # ls -lathr /vmfs/devices/disks/eui.b0269f4100d00000:1

                       

                       

                      • 8. Re: Can't grow VMFS Datastore
                        Christian452 Lurker

                        Increasing the local datastore on the installation disk is not possible through the GUI and might not be supported. At you can find a walkthrough on how to grow the local datastore on ESXi 4.x With ESXi 5.0 you will need to use "partedUtil resize ..." instead of fdisk (delete an recreate the partition) to modify the partition size though. Apkjunky

                        • 9. Re: Can't grow VMFS Datastore
                          a.p. Guru
                          Community WarriorsUser ModeratorsvExpert

                          [root@ESXI:~] partedUtil get /vmfs/devices/disks/eui.b0269f4100d00000

                          30377 255 63 488017920

                          1 128 488017886 0 0

                          The output doesn't show the correct partition type!? What I'd expect is partition type 251 (VMFS) rather than 0.

                          Please run the command with the "getptbl" option, and post the output.

                          partedUtil getptbl /vmfs/devices/disks/eui.b0269f4100d00000

                          André

                          • 10. Re: Can't grow VMFS Datastore
                            patrickds Expert

                            I don't see any remaining space on your logical disks in those screenshots.

                            Maybe you left free space on your RAID set, which should first be added to the logical disk through the Raid Manager?

                            • 11. Re: Can't grow VMFS Datastore
                              Rajeev Sharma Enthusiast

                              The only reason is that entire space from SSD Storage attached to your host is used as Datastore and no free space is left.

                               

                              Browse your datastore and look at size of all files in there.

                              Cheers Rajeev
                              VCP6-DCV, VTSP-SV, VSP-SV, MCSE, CCNA
                              • 12. Re: Can't grow VMFS Datastore
                                ThompsG Master

                                Hi there,

                                 

                                The screenshot that shows the partition with 232.71 GB is good. This shows that you have only one partition on the disk and the partition is of type VMFS5. This means the "partedUtil resize" command worked and extended the partition to use all the space... well that was the case until you shrunk the partition. Hoping this didn't cause any issues but we will move one

                                 

                                After running the "partedUtil resize" you will not be able to extend this drive from the GUI and hence the reason you are seeing the message "No devices with free space". This because the GUI is expands both the partition and volume (filesystem). Because the partition is already expanded the wizard is a little lost because the two operations normally go hand in hand.

                                 

                                So the next part is to expand the VMFS volume to fill the partition and this is what looks like it is not working from the command line. As another poster has mentioned you need to do this using the "vmkfstools --growfs"

                                 

                                Q: Sorry to be a pain but can you confirm the command you are running from the console. It should be as follows based on the information that you provided. :

                                 

                                vmkfstools --growfs /vmfs/devices/disks/eui.b0269f4100d00000:1 /vmfs/devices/disks/eui.b0269f4100d00000:1
                                

                                 

                                NOTE: It is important that you are running this command against the partition 1 (one - :1).

                                 

                                In the first example you were missing the :1 from the "device" which would generate the "Failed to get info from head device path /dev/disks/eui.b0269f4100d00000." In the second, I just want to confirm you have the :1 definitely on both devices because missing the :1 from the first device gives the message "Not found" as you listed.

                                 

                                This might be too late now so sorry about that.

                                 

                                Kind regards

                                • 13. Re: Can't grow VMFS Datastore
                                  Dave_the_Wave Hot Shot

                                  Curious, how risky or reliable is all this?

                                   

                                  Is it as such a norm that it's every day life of a vm provider?

                                   

                                  I know everyone will say "always backup".

                                   

                                  If I have to back it all up anyways, I may as well vMotion all the VMs off to another host, do a new-install on the host with the upgraded datastore, and then vMotion them back to the new home.

                                   

                                  I've done this scenario a couple of times, because it really didn't take that long, I think it was 2 hours each way.

                                   

                                  I chose the method that worked with a planned timeframe. The procedure mentioned here looks so "you're on your own".

                                  • 14. Re: Can't grow VMFS Datastore
                                    ThompsG Master

                                    I guess it depends on the length of string. I've done the procedure a number of times when the GUI has let me down and there wasn't any other real option. Backups were in place and tested, so the risk was an acceptable one for me at the time. Everybody has to make the call in regards to risk for themselves and it can be a danger just taking the advise from a blog or community forum (or even vendor support) without understanding the full implications.

                                     

                                    I do agree that I would prefer to do this via the GUI where the chances of stuffing something up are drastically reduced. If there is the option of another host, or the option of standing up new disk, or anything else - then that is definitely the path I would take. I don't believe the original poster has those options otherwise we wouldn't be having this conversation

                                     

                                    Depending on time that systems can be offline, I would even be tempted to export the VMs out using VMware Converter and then import in again on the correctly sized data stores.

                                     

                                    The procedure mentioned here looks so "you're on your own".

                                     

                                    I think this statement is true of anything that is gotten from the community and should not be forgotten or stressed enough. Its just people trying to help people out but its not support