Snapshot Size and Space Management

[MikeGIllinois]

Hi, guys.

     As I learned today deleting a snapshot can take fifteen or twenty minutes. From what I've read this can take a lot longer if there snapshot is extra large.

     Here are my questions.

     When I provisions a VM I might say that I am allocating 100 gigs to the machine. The actual op system may take 40 gigs. Does this mean I have 60 gigs of space left? If I take too many snapshots, say, for example, 15 of them and each snapshot takes 4 gigs, will I crash the VM because I have filled up my available space? If so, do I get any kind of warning?

     My snapshots seem to be about 4 gigs each. Is that "normal" or does the size depend on the op system and other factors?

     I don't see a way to browse my datastore and find out how much free space is available to each VM, if that concept is even relevant here. <-:

     Thank you, as always.

Mike Gallery

[vmroyale · Accepted solution]

Hello.

As I learned today deleting a snapshot can take fifteen or twenty minutes. From what I've read this can take a lot longer if there snapshot is extra large.

Yes, it basically depends on the number of snapshots and the size of the snapshots.

When I provisions a VM I might say that I am allocating 100 gigs to the machine. The actual op system may take 40 gigs. Does this mean I have 60 gigs of space left?

If you are using thin provisioning, then yes the 60 GB of remaining space would initially remain.  Over time, the amount of space consumed will likely increase, due to the way writes consume space and deletes don't reclaim them.  Are you using NFS or VMFS for your datastores?

If I take too many snapshots, say, for example, 15 of them and each snapshot takes 4 gigs, will I crash the VM because I have filled up my available space? If so, do I get any kind of warning?

The snapshots are stored on the VMFS or NFS volume, and don't depend on the virtual disk (VMDK) files.  What you have to worry about is filling the datastore.  This can happen, and it wouldn't be a good thing.  If you are using vCenter you can set alerts for when the datastores get to a certain capacity.

My snapshots seem to be about 4 gigs each. Is that "normal" or does the size depend on the op system and other factors?

Size depends entirely on the amount of writes going on in the virtual machine.  The snapshot is basically a differencing disk, so any new writes get written to it.  Snapshots can grow to the same size as the VMDK file.

I don't see a way to browse my datastore and find out how much free space is available to each VM, if that concept is even relevant here. <-:

Check your datastores for available space.

Good Luck!

[DSTAVERT]

Make sure you understand what snapshots are and how they should and shouldn't be used. Snapshots are intended to be very short term and deleted afterwards. They are NOT backups.

Best Practice http://kb.vmware.com/kb/1025279
Understanding http://kb.vmware.com/kb/1015180

[MikeGIllinois]

Hi. First, thank you for your comprehensive answers. They are MOST helpful.

When I provisions a VM I might say that I am allocating 100 gigs to the machine. The actual op system may take 40 gigs. Does this mean I have 60 gigs of space left?

If you are using thin provisioning, then yes the 60 GB of remaining space would initially remain.  Over time, the amount of space consumed will likely increase, due to the way writes consume space and deletes don't reclaim them.  Are you using NFS or VMFS for your datastores?

     *I'm still learning, Brian. I'm not using thin provisioning. Therefore, the amount of space I am using does NOT grow on demand, right? I set up a 140 gig drive size and, therefore, assume I have 140 gigs of space to use. Windows Server, the VM, reports that I have 140 gigs of available space on C:. Yet, when I browse the datastore with the VSPhere client and look in the virtual machine, I see:

Server 2008 64 Bit Doub.vmdk                 146,800,600

Server 2008 64 Bit Doub-000002.vmdk    122,537,000

Server 2008 64 Bit Doub-000001.vmdk      5,145,600

Server 2008 64 Bit Doub-snapshot1.vmsn   4,199,587

Server 2008 64 Bit Doub-snapshot2.vmsn   4,199,587

Server 2008 64 Bit.vswp                               4,194,304

Server 2008 64 Bit.vmdk                               1,328,128

The above file names are approximate

and the list is not complete, of course. But, as a Windows guy, I am thinking that I have set aside 140 gigs of space on my host hard drive and this list tells me I am using more than that. I'm not understanding something, obviously.     <-: By my Windows way of thinking I have about 300 gigs of files (per the above list) on a 140 gig volume. Do I really have the 140 gigs available that Windows SBS 2008 tells me I do? I'd be pleased to read up on this if I knew where to look.

This is a test lab environment and I have my virtual on a high end workstation. There is no SAN in place so I'm not using ISCSi. Is my file system, therefore, VMFS in this case as opposed to NFS?"

What you have to worry about is filling the datastore.

"I guess once I understand how much space I'm really using on my 1.36 terabyte store I'll know if I'm filling the datastore. In my elementary environment I don't have VCenter, - just ESXi and the VSphere Client."

Check your datastores for available space.

"How? <-: When I look at the VSphere client I see

Provisioned Storage 296 gigs

Not Shared Storage 273 gigs

Used Storage 273 gigs

I'm not understanding how VMWare utilizes hard drive space and how to determine how much space is free. I wouldn't presume to ask you for a primer on this. If you can refer me to an article(s) that helps me grasp this I would be most grateful."

Thank you for your time and energy.

Mike Gallery

Good Luck!

[MikeGIllinois]

Your links were very helpful. Thank you. I get it - use snapshots and then delete them within 24 hours. I did some reading and see that snapshot deletion is quite the complex process.

Does the deletion sequence matter? I have some old snapshots here in my test environment. If I have 3 or 4 old ones can I delete the oldest ones first or is best practice to delete them all? Best practics is clearly to delete them much more promptly than I have been.

Thanks again for your help.

Mike Gallery

[vmroyale]

Yet, when I browse the  datastore with the VSPhere client and look in the virtual machine, I  see:

Server 2008 64 Bit Doub.vmdk                 146,800,600

Server 2008 64 Bit Doub-000002.vmdk    122,537,000

Server 2008 64 Bit Doub-000001.vmdk      5,145,600

Server 2008 64 Bit Doub-snapshot1.vmsn   4,199,587

Server 2008 64 Bit Doub-snapshot2.vmsn   4,199,587

Server 2008 64 Bit.vswp                               4,194,304

Server 2008 64 Bit.vmdk                               1,328,128

Here is how the above files break down:

Server 2008 64 Bit Doub.vmdk:

the virtual machine's 140 GB drive that you assigned

Server 2008 64 Bit Doub-000002.vmdk:

snapshot file #2 of "Server 2008 64 Bit Doub.vmdk"

Server 2008 64 Bit Doub-000001.vmdk:

snapshot file #1 of "Server 2008 64 Bit Doub.vmdk"

Server 2008 64 Bit Doub-snapshot1.vmsn:

snapshot state file #1, part of the snapshotting process - goes with snapshot file #1

Server 2008 64 Bit Doub-snapshot2.vmsn:

snapshot state file #2, part of the snapshotting process - goes with snapshot file #2

Server 2008 64 Bit.vswp:

the VM's swap file - equal to the size of the VM's memory

Server 2008 64 Bit.vmdk:

another VMDK file

The above file names are approximate and  the list is not complete, of course. But, as a Windows guy, I am  thinking that I have set aside 140 gigs of space on my host hard drive and this list tells me I am using more than that. I'm not understanding  something, obviously.    

You are indeed using much more than the 140 GB of space you set aside for the VM's primary disk.  Each snapshot file requires additional space, and can grow to the same size as the original VMDK.  The .vswp file is also created, when you power on the VM which will (generally speaking) be the same size as the memory assigned to the VM.  There are also log files, snapshot state files, etc that take up additional space on the VMFS datastore.  Using more space than what is assigned to a VM's disk file is to be expected and is perfectly normal.

<-: By my Windows way of thinking I have  about 300 gigs of files (per the above list) on a 140 gig volume. Do I  really have the 140 gigs available that Windows SBS 2008 tells me I do?  I'd be pleased to read up on this if I knew where to look.

Windows does have the free space that it reports.  Windows only knows the virtual disk that it is assigned.  It has no insight beyond the VMDK file into the VMFS volume.  The VMDK is a virtual disk that Windows sees as a hard disk.  There are two dimensions, if you will, to this.  The Windows file system which is inside the VMDK file and the VMware File System (VMFS) which is where the VMDK and other virtual machine files are located.  ESXi handles the VMFS volumes.  Basically, there are two filesystems at work here.  One is only for the VM's use and one is only for the ESXi host's use.

This  is a test lab environment and I have my virtual on a high end  workstation. There is no SAN in place so I'm not using ISCSi. Is my file  system, therefore, VMFS in this case as opposed to NFS?"

Yes, you are using VMFS volumes.

"I  guess once I understand how much space I'm really using on my 1.36  terabyte store I'll know if I'm filling the datastore. In my elementary  environment I don't have VCenter, - just ESXi and the VSphere Client."

In your vSphere client, connected to an ESXi host, select the host and then choose the Configuration tab.  Select the Storage option under the Hardware section, and from here you can view your datastores and the space information on them.

I'm  not understanding how VMWare utilizes hard drive space and how to  determine how much space is free.

Check out the "VMware Virtual Machine File System: Technical Overview and Best Practices" white paper.  It has a lot of good information in it.

[MikeGIllinois]

The data below is from Configuration/DataStores. I have 725.37 gigs free on on 1.36 Terabyte drive. The used data consists, therefore, of ESXi and all the files associated with my VM's.

Identification              Device                  Capacity      Free                     Type                     Last Update                Hardware Acceleration

1.36 TB Datastore      Local ATA Disk      1.36 TB      725.37                   vmfs3                     8/3/2011                    Unknown

Server 2008 64 Bit Doub.vmdk                 146,800,600

Server 2008 64 Bit Doub-000002.vmdk    122,537,000

Server 2008 64 Bit Doub-000001.vmdk      5,145,600

Server 2008 64 Bit Doub-snapshot1.vmsn   4,199,587

Server 2008 64 Bit Doub-snapshot2.vmsn   4,199,587

Server 2008 64 Bit.vswp                               4,194,304

Server 2008 64 Bit.vmdk                               1,328,128

Here is how the above files break down:

Server 2008 64 Bit Doub.vmdk                 146,800,600 - the virtual machine's 140 GB drive that you assigned

Server 2008 64 Bit Doub-000002.vmdk    122,537,000 - snapshot file #2 of "Server 2008 64 Bit Doub.vmdk"

Server 2008 64 Bit Doub-000001.vmdk      5,145,600 - snapshot file #1 of "Server 2008 64 Bit Doub.vmdk"

Server 2008 64 Bit Doub-snapshot1.vmsn   4,199,587 - snapshot state file #1, part of the snapshotting process - goes with snapshot file #1

Server 2008 64 Bit Doub-snapshot2.vmsn   4,199,587 - snapshot state file #2, part of the snapshotting process - goes with snapshot file #2

Server 2008 64 Bit.vswp                               4,194,304 - the VM's swap file - equal to the size of the VM's memory

Server 2008 64 Bit.vmdk                               1,328,128 - another VMDK file

vmware-1.log                                                   1,537.71 KB

vmware.log                                                        547.18 KB

vmware-2.log                                                   282.95 KB

Server 2008 64 Bit-Doub.nvram                         8.48 KB

Server 2008 64 Bit-Doub.vmx                             3.56 KB

Server 2008 64 Bit-Doub.vmsd                            1.21 KB

Wll, first I would like to thank you for taking the time to explain these files to me in such detail. <-: would you mind notating the six other files in my directory which I bolded?

I think what I have learned here is this:

When I create a new VM and claim 200 gigs of a 1 terabyte drive for the op system I am automatically claiming more space on my terabyte drive than just 200 gigs..

That extra space will consist of, at minimum, a swap file, log files, and undeleted snapshots.

Once a snapshot is created it should be deleted with 72 hours because VMWare will use it, instead of the parent VMDK, for the op system. This is not the intended purpose of snapshots and even through they can apparently grow to a size of 32 children and one parent this is messy. A snapshot is used to test a configuration and roll back to it. Snapshot deletion consumes a lot of time because snapshot data has to be reincorported into the original VMDK.

In an earlier message you said, "What you have to worry about is filling the datastore.  This can happen, and it wouldn't be a good thing."

Now I think I see how this can happen. If I had, for instance, a 1 terabyte hard drive supporting an ESXi host machine and I put 5 180 gig virtuals on it I via non-thin provisioning I would then have claimed 900 gigs of that terabyte drive leaving me with 100 gigs. Assuming that each op system uses a 4 gig swap file for a total of 20 gigs now I am down to 80 free gigs (not counting what ESXi uses and what log files and other smaller files use). So, after creating a few swap files and not deleting them the swap files start to grow. One day I have zero bytes left and this sounds like it's a pretty mess deal to clean up. Is this how it happens?

This article shows the complexity of snapshot relationships to their parent.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101518...

Thank you for the article on the VMWare Virtual File System.

Thank you again, for your help. I see that this thread has been viewed 55 times which means that others really do learn from the discussions out here.

Mike Gallery

[vmroyale]

Glad to help out!

vmware-1.log - VM log file that has been rotated
vmware.log - active VM log file
vmware-2.log - VM log file that has been rotated
Server 2008 64 Bit-Doub.nvram - the VM's BIOS
Server 2008 64 Bit-Doub.vmx - the virtual machine's configuration file (all the settings for the VM are in here)
Server 2008 64 Bit-Doub.vmsd - snapshot metadata structure file

When I create a new VM and claim 200 gigs of a 1 terabyte drive for the  op system I am automatically claiming more space on my terabyte drive  than just 200 gigs..

Yes, unless you used thin provisioning, which would not claim the entire 200 GB immediately.  The space is still provisioned, it is just not consumed.  With thin provisioning, you could overalloace space on the VMFS volume.  This isn't necessarily a bad thing, but it is something to be aware of and to monitor closely if used.

Once a snapshot is created it should be deleted with 72 hours because  VMWare will use it, instead of the parent VMDK, for the op system. This  is not the intended purpose of snapshots and even through they can  apparently grow to a size of 32 children and one parent this is messy. A  snapshot is used to test a configuration and roll back to it. Snapshot  deletion consumes a lot of time because snapshot data has to be  reincorported into the original VMDK.

There are no hard and fast rules here, but the logic is generally sound.  A VM that has very few writes could sit in snapshot mode for weeks or months and not grow significantly.  I think your logic of snapshots being used to test and roll back quickly is sound.  I don't personally like to get into multiple snapshots as it gets confusing, especially if I have to start explaining it to a customer.

Now I think I see how this can happen. If I had, for instance, a 1  terabyte hard drive supporting an ESXi host machine and I put 5 180 gig  virtuals on it I via non-thin provisioning I would then have claimed 900  gigs of that terabyte drive leaving me with 100 gigs. Assuming that  each op system uses a 4 gig swap file for a total of 20 gigs now I am  down to 80 free gigs (not counting what ESXi uses and what log files and  other smaller files use). So, after creating a few swap files and not  deleting them the swap files start to grow. One day I have zero bytes  left and this sounds like it's a pretty mess deal to clean up. Is this  how it happens?

Snapshots or thin provisioning are generally what will lead to full VMFS volumes.  There should only be on .vswp file per VM, so that scenario is unlikely.  There also needs to be some free space on VMFS volumes for metadata.  This is discussed in http://kb.vmware.com/kb/1003412  Again, monitoring the space is crucial when there are long-running "busy" snapshots or when thin provisioning is in use.