goldeneye_007
Contributor
Contributor

VM datastore running out of disk but not sure why.

Jump to solution

I have a datastore that keeps running out of disk space but I am unsure why.

No VM disks are thin provisioned. There are two VM's

1 has a 500GB disk

1 has a 100GB disk, and 2 500GB didsks.

The total is 1.6TB. The disk size is 3TB. Yet it consistently says it is out of space.

The sys_vol is on another disk and is 500GB in size and is 470GB free.

Any ideas what is going on? Is there some log file I should be looking for?

I am presently running disk consolidation on one of the VM's but I am dubious it

will help.

Thanks.

0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership

It's only the base virtual disk which is preallocated with thick provisioning. Snapshot will start with a small size, but can grow up to the provisioned size, depending on the VM's write activity.

Please take a look at the diagram at the end of https://kb.vmware.com/s/article/1015180​ which explains how snapshots work.

André

View solution in original post

0 Kudos
12 Replies
sjesse
Leadership
Leadership

If your running disk consolidation your probably had snapshots, which can be up to the same size of the disk per snapshot.

0 Kudos
goldeneye_007
Contributor
Contributor

Thanks. However, why would they continue to grow if there have been no new snapshots made?

I wonder if my backup software is creating snapshots then deleting them and reclamation is not working?

Also, is it better to delete snapshots or run consolidate disks or

delete snapshot and then consolidate?

0 Kudos
a_p_
Leadership
Leadership

There may be snapshots even though the Snapshot Manager doesn't list any.

An easy way to find out about it, is to open the VM's settings. If the virtual disk's sizes are grayed out and/or the virtual disks' names look like "<vmname>-00000x.vmdk", then the VM has active snapshots, which you may want to consolidate.

Please note that even with thick provisioned virtual disks, consolidation - with the VM powered on - will consume some additional disk space, depending on the VM's disk write activity.

Another great way to find out whether a VM has active snapshot (along with other things) is regularly run RVTools

André

0 Kudos
sjesse
Leadership
Leadership

When you take a snapshot everything before it is frozen, and that doesn't change, its the delta that grows.

0 Kudos
goldeneye_007
Contributor
Contributor

Why should it grow if all disk space is pre-allocated?

I can see snap A + snap B disk growth but not a continuous growth of snap B from that point.

0 Kudos
a_p_
Leadership
Leadership

It's only the base virtual disk which is preallocated with thick provisioning. Snapshot will start with a small size, but can grow up to the provisioned size, depending on the VM's write activity.

Please take a look at the diagram at the end of https://kb.vmware.com/s/article/1015180​ which explains how snapshots work.

André

View solution in original post

0 Kudos
goldeneye_007
Contributor
Contributor

I got it. Thanks for the explanation.

If I have a backup system that makes snapshots but then deletes them,

should not the space be reclaimed by VMware? I read some things about reclaim

from CLI but is there a way to check it using either the webgui of the server or vCenter?

I have been unable to locate it.

One final question if I may. I did not receive any notification that disk space was low.

Is there a setting I am missing?

0 Kudos
sjesse
Leadership
Leadership

Your initial disks may be, but snapshot aren't, can't be. If you take a snapshot, delete a bunch of stuff, then fill that disk that deleted stuff needs to be somewhere since you took a snapshot before than.Its why there are recommendations to limit there long term use, like they mention in

VMware Knowledge Base

like you mentioned I've seen backup software miss removing them so running something like the vcheck reports

vCheck vSphere - Samples - VMware {code}

or something else that can look at snapshots and orphaned snapshots is a good idea. If you have veeam one they have a tool to find the orphaned ones

Orphaned VM Snapshots - Veeam ONE Reporter Guide

0 Kudos
a_p_
Leadership
Leadership

If nothing goes wrong, image based backup applications delete/consolidate the snapshot they take for the backup after the job is done. However, it sometimes happens that a snapshot cannot be deleted (for whatever reason).

You can determine whether a snapshot exists with the options that I already mentioned. You can also see it from the VM's summary page, if the provisioned space is a multiple of what you've configured for the VM. Personally I use RVTools - which can even be scripted - and is free to use.


André

0 Kudos
goldeneye_007
Contributor
Contributor

Okay cool.

I learned a lot from your posts.

I was unaware that snapshots grow if left more than 72hours.

I usually leave 1 whenever there is some major change to a server

just to ensure a quick wy back to a known working point without going

through a full restore.

Thanks.

0 Kudos
sjesse
Leadership
Leadership

Take a look at this for setting up an alarm, you can have it email you if it runs over a certain threshold.

https://uncomplicatingit.com/vmware/setting-up-custom-datastore-thresholds-in-vcenter-6-5/

0 Kudos
goldeneye_007
Contributor
Contributor

It is already enabled.

Perhaps the email notification is not working.

0 Kudos