From an email I just sent to my managed service provider which might explain what I'm trying to convey better:
The volume was at 302gb of space. On the morning after the lock up, I thought I remember seeing the volume overprovisioned in the provisioned size value of over 500GB. The screen shot from xxx though shows 489 provisioned (again over allocated). My DBA didn’t create another snapshot. He copied the DB of about 9GB and started a job that would have caused the active original DB of 9GB to grow but not to have eaten up the more than 25GB of free that should have been there (302GB – the 276 in use by all VMDK’s VSWP files, logs etc but not the snapshot files). The snapshot filled that space and then some. My point is that vsphere 5, in this case, in order to accommodate even 1 snapshot, seemed to have automatically instituted thin provisioning principles to over allocate a volume (up to a point). I’m still asking the question about “is this different in vsphere 5…I say it is and which requires looking under the hood more on what is really going on with a volume and not just looking at a VM and checking the san status of what total size is and how much free space is left. In your email:
“. If you leave snapshots around, they can grow up to the size of the original disk. That’s why you’re seeing thin provisioning; it’s for the snapshots. The disks are thick provisioned, but there wasn’t enough space to thick provision the snapshots (if that’s possible).”
If you create a snapshot and don’t create another…how can the original snapshot grow? That is what is being impliedand what would explain having the volume run out of space…xxx actions didn’t fill up the space…but if the snapshot was dynamically and automatically updated/increased due to xxx copying a file that was there at the time of snapshots and or the original DB growing due to a job that was running…then I “get it” now. But to me, I still say this is new functionality in vsphere 5 that didn’t exist in 4.1. I’d rather have VMware tell me that there is NOT enough space to accommodate a snapshot than use this thing provisioning angle (which is a surprise). Now it will let you but unless you look more closely, you won’t know how dangerous a situation you may be in…the very risk you take when you explicitly choose a thin provisioning route on your san volume and/or VMDK’s. It will let you overprovision storage but at any moment, something could tip the scales resulting in zero space left.
So I think I can distill my "question" down to the following:
So I’m clearer now but still need a couple of basic questions answered so I fully understand:
- Is this a new feature of vsphere 5 to automatically use thin provisioning with snapshots, allowing you in many cases to create a snapshot on a thick provisioned volume that would in the past or ordinarily exceed the amount of free space? Does 4.1 and earlier allow this? I didn't think so to my recollection...if not enough free space the snapshot was denied.
- Where are the changes to the snapshot(s) being written? To a hidden “delta’ file that was exposed in previous versions? In some dynamic thin provisioned space?
- Is there a way to turn off this thin provisioning behavior? What if we just wanted “old school” behavior to NOT allow any snapshot if there isn’t enough free space to accommodate the amount of changed blocks in the snapshot about to be taken?
If there is a "hidden" delta file, then I can see the value of your script to expose them...
