VDR backup disk space never decreasing?

[MKguy]

I've been testing VDR 1.0.1 for a while now, it works fine for me so far, but there is one possible serious issue I'm pondering about now:

Old restore points are being deleted according to my configured retention policy:

8/6/2009 1:16:37 PM: Executing Reclaiming

8/6/2009 1:16:37 PM: Reclaiming Destination /SCSI-0:1/...

8/6/2009 1:16:54 PM: Deleted restore point 7/30/2009 10:09:35 AM for virtual machine MY_VM1

http://...

8/6/2009 1:19:59 PM: Task completed successfully

8/6/2009 1:19:59 PM: Duration: 00:03:22

Seems like everything goes according to plan here, no issue.

However the actual disk space used by VDR never decreases on my VDR box, even though it delted a few dozen restore points already.

Thinking about the way VDR backups work, this actually seems quite natural and logical to me:

VDR creates an initial full backup on the first backup, and then only incremental ones for all subsequent backups. So if you want to restore from a restore point from yesterday, you need to have the data of the initial + all other incremental backups ever made for that VM, regardless of whether you still have the actual restore points of all those incremental backups. That way (by that design), it is impossible for VDR to ever decrease the used disk space, unless you delete the most recent restore point or the whole VM from VDR of course.

This means it should make no difference to the disk space whether your retention policy keeps one or one hundred restore points. If you restore the most recent restore point from any of those policies, VDR still has to process all the data of all previous backups, applying changes one after another as those backups happened. "Merging" such a block-level disk backup seems impossible to me, or does it really do something like that?

Someone please clarify if this theory is applicable, or if am I horribly mistaken here?! if I am in fact mistaken, why is my used disk space not only rising steadily?

[RParker]

As with ESX (though less problematic than before) you have to rescan the HBA. Have you tried simply rescanning to see if the space is reclaimed?

Also how did you provision the target, is it thin provisioned? If it is, I don't think the space is reclaimed there either, once a file grows to a certain size I don't think it shrinks.. unless you migrate the VM.

However the actual disk space used by VDR never decreases on my VDR box, even though it delted a few dozen restore points already.

And did you mark these for deletion and wait until AFTER the integrity scan was done, because that's when the actual deletions occur....

[MKguy]

3 of the 5 VMs are thick provisioned, 2 are thin provisioned.

The VDR VM uses a simple thick provisioned VMDK for the dedupe store. No RDM LUN or some other fancy stuff. I'm checking what the VDR plugin is reporting as free space and im running a plain perl script in the VDR VM that writes the disk space to a textfile on an hourly basis. I never saw a decrease here.

Some deleted restore points are already from older than a week, so they should have been deleted for good by the daily or the last weekly integrity check. I'll try a manual integrity check tomorrow.

My problem is also that I don't understand how the actual data can be deleted if you use incremental backups the way VDR does.

[adisarro]

Think of it this way....it hashes each block and keeps a database of where the data for that hash is stored.

When it's deleting old restore points, it gets rid of the pointers for that restore point. so if it had any data that it was the only one referring to, that data would be deleted. If it didn't have any unique data, you wouldn't save any space.

[MKguy]

I ran a manual integrity check today too, it didn't change the used disk space at all. Can anyone confirm that they in fact see a reduction of the used disk space after reclaiming/integrity tasks?

adisarro:

This is conceivable with a file-level approach, but not with a block-level backup like VDR, at least I can't wrap my mind around how you can ensure integrity that way.

[adisarro]

Ok lets look at it a different way.

Dedupe only stores unique data. So if you delete one restore point it can only delete that unique data, not the things shared with other backups.

So if you delete day 1, it can only delete the differences between day 1 and day 2, if those differences haven't shown up again.

So if you delete day 1 it would most likely be offset by the new stuff from today's backup, the differences between today and yesterday are probably a similar amount of data as the differences between day 1 and day 2.

So unless you delete a bunch of early restore points, and you're data changes a lot each day, you probably won't notice much change in datastore size.

[RParker]

Think of it this way....it hashes each block and keeps a database of where the data for that hash is stored.

OK, we got the whole dedupe process and restore points, but the bottom line is if you DO delete those early restore points and all relevant data with it, and there should be NO data left over, DOES the VDR appliance reclaim the space? I say no, but what he wants to know (and myself included) can someone actually CONFIRM this as fact....

It might REUSE the space, but I don't think it RECLAIMS (shows as free space) anymore..... FF FF FF written to a drive versus 00 00 00 are very different, padding and blanking out the area data and marking it as UNUSED will yield different results for space used. That's what we would like to know. . . . .

[RParker]

I ran a manual integrity check today too, it didn't change the used disk space at all

I see your point, however with backup that space is designated as backup and should not be shared or used for any other purpose. So if you set aside 1TB of space, that space should be considered gone and already spoken for. So to reclaim the space wouldn't matter.

Now I know the VDR will REUSE the space, but you won't see it as FREE space any longer, at least that's my theory. In all the years of dealing with all the stacker, drive space, disk compression, zips and stuff.. I have yet to see one that DOES make this space free again without some monumental process, which in the end simply marks the are as reusable, but it doesn't mark it as unused... So you will get the space back, you just won't be able to tell how much space is occupied and how much can be reused, if that makes sense.

Even SAN drives have to go back over an area marked as unused and do some low level writes to those drives and the space is SLOWLY reclaimed over time... But those systems have their own built in algorithm and proprietary format, so that's not really the same thing.

[admin]

How are you determining that the space is not decreasing? The slab files (where deduplicated data is stored) are sparse files. These files are listed as larger than the actual disk space consumed. Taking data out of a sparse file does not decrease its apparent size.

Fletcher Glenn

[ian4563]

What's most interesting here is that you have somehow managed to get it to work without issue. Have you been running it for more than a week?

[NPBrown]

I'm seeing the same issue here with version 1.2. After receiving "Disk Full" error I marked a load of restore points for deletion and ran a reclaim. Still couldn't backup ("Disk Full"). So I marked all but the most recent restore points for deletion and reclaimed. Checked that I now only have 1 restore point per vm (down from about 10 per VM). Still can't backup.

Switched backups to new, empty destination. Backed up no problem. Clearly the VDR isn't reclaiming space properly.

Cheers, Nick.