VMware Cloud Community
johnzapf
Contributor
Contributor
Jump to solution

There is no more space for virtual disk ...

I'm having this same issue. 22 terabytes and it says it's full. I've got it running two times and went in and deleted 7-9 terabytes of data so far and it still says its 100% full, I don't understand.

0 Kudos
2 Solutions

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

As mentioned before, changes are only written into the snapshot (sesparse) file, so even deleting data from within the guest OS will cause modification that will be written into the sesparse file.

Since you say that you are deleting data, I wonder - in case snapshot consolidation does not work - if it would be an option for you to discard todays changes on that large drive!? Just asking.

André

 

View solution in original post

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Just to confirm: It is ok for you to lose todays changes?

If the answer is yes, then what you may do is:

  1. shut down the VM (unless already done)
  2. delete the following snapshot files in /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
    AIBS2019_1-000001-ctk.vmdk
    AIBS2019_1-000001-sesparse.vmdk
    AIBS2019_1-000001.vmdk
  3. edit (using e.g. the vi editor) the VM's configuration (.vmx) file in the VM's base directory, and change the virtual disk's file name
    from: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1-000001.vmdk
    to: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1.vmdk
    (since the VM seems to have multiple virtual disks, make sure that you only change this one!
  4. reload the VMX by following the steps in https://kb.vmware.com/s/article/1026043
  5. create a new snapshot
  6. run "Delete All" from the Snapshot Manager (wait for the process to complete)
  7. edit the VM's Hard Disk settings and change the "Diskmode" from "Dependent" to "Independent-Persistent"
    (this will exclude the virtual disks from snapshots)

André

 

View solution in original post

30 Replies
johnzapf
Contributor
Contributor
Jump to solution

I don't understand it seems to be filling up as fast as I'm deleting data I am lost at what's going on?

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

and when I do it the vm on windows shows there is 11.2 TB's free. so why does esxi think its full?

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

There is no more space for virtual disk '/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1-000001.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session

And I know there is fourteen terabytes free on that drive I don't know what's going on!

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

It looks like you ran out of disk space on the datastore, due to an active snapshot.

Please run the following commands from the ESXi host's command line and post the commands' output.

cd /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
df -h
ls -lisa

André

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

 

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@ESXi-SERVER:~] cd /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 7.3T 4.3T 3.0T 59% /vmfs/volumes/8TB
VMFS-6 21.8T 21.8T 0.0B 100% /vmfs/volumes/23TB
VMFS-6 237.8G 59.7G 178.1G 25% /vmfs/volumes/256-SSD
VMFS-6 109.8G 1.4G 108.3G 1% /vmfs/volumes/ESXi-HOST-SSD
VMFSOS 119.8G 4.0G 115.8G 3% /vmfs/volumes/OSDATA-5fc10f7f-a6bbffdf-c4cf-b8ca3a6f7dfc
vfat 4.0G 279.6M 3.7G 7% /vmfs/volumes/BOOTBANK1
vfat 4.0G 204.6M 3.8G 5% /vmfs/volumes/BOOTBANK2
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019] ls -lisa
total 23434965248
167772164 128 drwxr-xr-x 1 root root 73728 Feb 19 18:23 .
37748740 128 drwxr-xr-x 1 root root 73728 Nov 30 2020 ..
469765380 6144 -rw------- 1 root root 5636608 Feb 19 18:23 AIBS2019_1-000001-ctk.vmdk
461376772 349503488 -rw------- 1 root root 446743482368 Feb 19 19:20 AIBS2019_1-000001-sesparse.vmdk
465571076 0 -rw------- 1 root root 388 Feb 19 18:23 AIBS2019_1-000001.vmdk
452988164 6144 -rw------- 1 root root 5636608 Feb 19 11:05 AIBS2019_1-ctk.vmdk
171966468 23085449216 -rw------- 1 root root 23639499997184 Feb 19 11:30 AIBS2019_1-flat.vmdk
176160772 0 -rw------- 1 root root 572 Feb 19 06:01 AIBS2019_1.vmdk
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019]

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

This is crazy, the drive is not full.

Screenshot 2024-02-19 112835.jpg

 

This is what is actually being used on that drive. 

Screenshot 2024-02-19 114307.jpg

I don't think it's taking a snapshot. There's no snapshots on any of the servers to delete. All snapshots are empty.

That server just shut down again there's no room left and I don't understand why... This is killing me i've spent 4 hours on this so far today I need to get this back up and running. Anybody has any idea please let me know thank you.

 

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Your guest OS may report free disk space, but the datastore ran out of disk space.

You've created a ~21.5 TB virtual disk on a ~21.8 TB datastore. leaving only ~300 GB free disk space.
That disk space is now used by the snapshot.
Please remember that once a snapshot is created, all new writes are stored in that snapshot, i.e. it can grow up to the provisioned size of its base disk.

What you could try - not sure whether it will work with almost no free disk space - is to delete the snapshot from the Snapshot Manager while the VM is powered off.

André

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

Screenshot 2024-02-19 115848.jpg

 

there is no snapshot that I can see. And this has been running like this for 5 years now not sure why it has been filling up like this?

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

There is a snapshot, even though it does not show up in the Snapshot Manager for whatever reason.

469765380 6144 -rw------- 1 root root 5636608 Feb 19 18:23 AIBS2019_1-000001-ctk.vmdk
461376772 349503488 -rw------- 1 root root 446743482368 Feb 19 19:20 AIBS2019_1-000001-sesparse.vmdk
465571076 0 -rw------- 1 root root 388 Feb 19 18:23 AIBS2019_1-000001.vmdk
452988164 6144 -rw------- 1 root root 5636608 Feb 19 11:05 AIBS2019_1-ctk.vmdk
171966468 23085449216 -rw------- 1 root root 23639499997184 Feb 19 11:30 AIBS2019_1-flat.vmdk
176160772 0 -rw------- 1 root root 572 Feb 19 06:01 AIBS2019_1.vmdk

The second line shows the snapshot, and the last line the date/time when the snapshot has (most likely) been created.

The usual way to get rid of such a snapshot would be to create another snapshot, and then click "Delete All" in the Snaphot Manager.
However, with no free disk space this may not work.
Do you see an option "Consolidate" in the VM's context menu?

André

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

Screenshot 2024-02-19 121126.jpg

 And not sure what that is it shows up as the 21 terabytes also.

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

yes that option is there, But it won't run insufficient disk space.

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

This makes the most sense. So it must have started doing a snapshot this morning at 6AM and of course it's not going to have room to do that, that drive HAD 14TB's on it. But I don't understand I don't see snapshots under any of my servers I didn't know that was happening. I definitely need to turn that off in the future I do not have room for snapshots. Now we need to figure out how to delete that one stop it and delete it?

And I have been deleting terabytes of data off that drive and it's filling up as fast as I'm deleting the data. Have already removed 7TB's and its full again. 

 

 

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

As mentioned before, changes are only written into the snapshot (sesparse) file, so even deleting data from within the guest OS will cause modification that will be written into the sesparse file.

Since you say that you are deleting data, I wonder - in case snapshot consolidation does not work - if it would be an option for you to discard todays changes on that large drive!? Just asking.

André

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

Like I said above consolidate doesn't work there's no space. Just need to get this server back UP don't care about the data i've deleted I need to figure out how to stop the snapshot process how to clear it and how to get this server running ASAP. 

 And I need to turn off snapshots so they don't try to do this again. It's now one o'clock in the afternoon I've been working on this for 7 hours I have a head breakfast yet I haven't done anything this is very very very important to get this back up.

 

And I just don't understand what's taking up all this space. I have no snapshots on any of my vm's there should be fourteen terabytes free on this drive. Need to figure out how to get this cleared out and back UP and running and I'm not having any luck. I'm losing a lot of money and a lot of data right now I need to get this back up and running I don't care about the data I've deleted.

And I don't care about snapshots. I have never used one in my life. I have backup software that I can restore things from.

Disk consolidation for VM AIBS2022 has failed: Insufficient disk space on datastore

 

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Just to confirm: It is ok for you to lose todays changes?

If the answer is yes, then what you may do is:

  1. shut down the VM (unless already done)
  2. delete the following snapshot files in /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
    AIBS2019_1-000001-ctk.vmdk
    AIBS2019_1-000001-sesparse.vmdk
    AIBS2019_1-000001.vmdk
  3. edit (using e.g. the vi editor) the VM's configuration (.vmx) file in the VM's base directory, and change the virtual disk's file name
    from: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1-000001.vmdk
    to: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1.vmdk
    (since the VM seems to have multiple virtual disks, make sure that you only change this one!
  4. reload the VMX by following the steps in https://kb.vmware.com/s/article/1026043
  5. create a new snapshot
  6. run "Delete All" from the Snapshot Manager (wait for the process to complete)
  7. edit the VM's Hard Disk settings and change the "Diskmode" from "Dependent" to "Independent-Persistent"
    (this will exclude the virtual disks from snapshots)

André

 

a_p_
Leadership
Leadership
Jump to solution

... just in case the VM is still running, you may as well copy today's changed data to e.g. a network shared folder, and copy it back after the above procedure.

André

0 Kudos
a_p_
Leadership
Leadership
Jump to solution

WAIT: I just realized that the flat file's time stamp is "11:30".

Did you try to run a snapshot consolidation at that time?

In this case parts of the snapshot may have already been merged into the base disk, and deleting the snapshot by following the steps I mentioned may result in data corruption!

André

 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

the VM will not run, no space, that is the issue. VM is not running. 

0 Kudos
johnzapf
Contributor
Contributor
Jump to solution

I keep trying to consolidate it won't work there's no space there's absolutely zero space it's using it all.

0 Kudos