Re: VDR Issue with large VMs - Page 2

jarsenea · ‎07-07-2010

Hey all,

I currently have a few webservers that occupy anywhere between 250-500GB of space that need are being backed up with VDR on a nightly basis. The problem however is that after the first backup (which I completely understand will take a while), subsequent, incremental backups also take an exceedingly long amount of time (hours, not minutes). This is case even if only 10-50mb of files have changed from one night to another.

Anyone experience this? VDR 1.2 seems to be working better than any other version so far, but I'm hoping I can speed up these incremental backups because I just won't have enough time in a day to do all my VMs.

Best,

jarsenea · ‎07-08-2010

that are using "network" as the method of backing up instead of HotAdd.

The ESX host that VDR lives on MUST have access to ALL the LUN's it will backup. Not only that, but ALL of the VM's it needs to backup need to be in the SAME datacenter.
Otherwise it will use network backup, not LAN Free backups.

Yep, this is the case.. 2 esx hosts, 1 datacenter, all LUNs accessible via each host.. In fact, I have 2 LUNs specifically for OS/Configuration and 3 LUNs for Data vmdk's, all of them are accessible to each host within the datacenter.

RParker · ‎07-08-2010

I should add that I have two backup stores, each of them are RDMs attached directly to the VDR appliance connected to a backup fibre storage system.

Well we are talking about the VM's to be backed up.

The VDR appliance must be able to "see" the LUN's where the VM's are located. Then it mounts the VM disk directly to the VDR during the backup, when it's done it will commit the snapshot, reconfigure the VDR to remove the disk, and move on to the NEXT VM.

If the LUN's where the VM's are do not show up on the ESX host of the VDR, then it cannot mount these disks, and thus you end up with a Network backup, rather than a SCSI hot add backup.

Doesn't matter how you connect to the target for the backup repository

jarsenea · ‎07-08-2010

Another bit of info..

All the VMs that are defaulting to "Network" are on the same LUN (VMFS_OS_2), yet the actual VMs that are powered on are located on both ESX hosts (thus indicating each ESX host can access the LUN).

RParker · ‎07-08-2010

Yep, this is the case.. 2 esx hosts, 1 datacenter, all LUNs accessible via each host..

Well it should work then. If you gave it the permissions, and it has the LUNs visible, then the VDR should be able to simply mount the disk.

The permissions you setup I assume are the same ones you changed for the ID inside the VDR plug-in? so when you login to the VDR, those credentials should have FULL access to everything (even if it means making it Administrator) just to ensure there are no restrictions, then it should be able to scan the LUN, mount the VM (to the VDR) and that's it.

The VDR of all things, has done this quite well, this hasn't ever been a problem.

jarsenea · ‎07-08-2010

Yep, this is the case.. 2 esx hosts, 1 datacenter, all LUNs accessible via each host..

Well it should work then. If you gave it the permissions, and it has the LUNs visible, then the VDR should be able to simply mount the disk.
The permissions you setup I assume are the same ones you changed for the ID inside the VDR plug-in? so when you login to the VDR, those credentials should have FULL access to everything (even if it means making it Administrator) just to ensure there are no restrictions, then it should be able to scan the LUN, mount the VM (to the VDR) and that's it.
The VDR of all things, has done this quite well, this hasn't ever been a problem.

Yep, I indicated this on page 1.. I also tested the VDR account (logging into vCenter, doing tasks only administrators are able to do such as create datastores, delete datastores, delete vms, create vms, modify vms, add hardware, etc.)

RParker · ‎07-08-2010

All the VMs that are defaulting to "Network" are on the same LUN (VMFS_OS_2)

Your previous log shows VMFS_DATA_1. Is that a local datastore?

The VM's must be on a shared LUN on the SAN not local storage in order to backup using the SCSI method

jarsenea · ‎07-08-2010

All the VMs that are defaulting to "Network" are on the same LUN (VMFS_OS_2)

Your previous log shows VMFS_DATA_1. Is that a local datastore?
The VM's must be on a shared LUN on the SAN not local storage in order to backup using the SCSI method

No local storage in the environment, all FC SAN..

VMFS_DATA_1 is one of the LUNs we use to storage data partitions (vmdks). Our policy is to keep the VM Config/OS partition on a separate LUN than the data partitions.

admin · ‎07-08-2010

vcbAPI logs should have information about why hotadd was not used. Can you confirm the VMFS block size of VMFS_DATA_1 datastore and whether it matches (or sis maller ) that the block size for the VDR appliance?

jarsenea · ‎07-09-2010

Hi azmir,

Thanks for taking some time to help me out with this.. I've looked through the logs, but can't seem to find any error about why Network is being used.

The VDR Appliance's OS vmdk is located on a datastore with a 4MB Blocksize. It's configured with 2 backup stores, each of which are direct access RDMs (so blocksize for the backup stores is irrelevant).

The VMs I'm having an issue with are located on a datastore with 8MB Blocksizes (to support 1-2TB vmdks).

Would this be an issue? Should I reorganize our datastores so that blocksizes are all 8MB (in order to avoid any issues with VDR)?

Best,

RParker · ‎07-09-2010

The VMs I'm having an issue with are located on a datastore with 8MB Blocksizes (to support 1-2TB vmdks).

We have a winner! 8mb block sizes are bigger files, VDR cannot mount files that are too big to fit on a 4MB datastore (no bigger than 1TB). So move the VDR to the 8MB datastore, and I bet it will work then.

Gostev · ‎07-20-2010

Yes, ctk is enabled on all my VMs. I had to do this when I was evaluating Veeam. One of the reasons I didn't go with Veeam is that I'm a little worried about having a single massive file that could be 20-30TB in size. That and I couldn't separate my backup jobs while maintaining deduplication within the datastore.

I am confused with this comment. Veeam does deduplication on per-job basis, so you can easily control the size of your backup files. You can do a single job and single backup file 20-30TB in size. Or you can have 10 separate jobs and so 10 backup files of 3TB in size.

candal02 · ‎07-20-2010

Not that this is a fix to your issue, but I get around that by only backing up the O/S drive using vDR (40GB or less), and use other means to backup the data drives (hundres of GB's). That way if a recovery is needed, I restore the O/S drive quickly using vDR, reconfigure the data drives on the VM, then restore the data drives from the "other means". I realize that not everyone has the resources to do this, and don't see the need for it as vDR should be able to handle this on it's own. So take my comments for what they are worth.

jarsenea · ‎07-20-2010

Yes, ctk is enabled on all my VMs. I had to do this when I was evaluating Veeam. One of the reasons I didn't go with Veeam is that I'm a little worried about having a single massive file that could be 20-30TB in size. That and I couldn't separate my backup jobs while maintaining deduplication within the datastore.

I am confused with this comment. Veeam does deduplication on per-job basis, so you can easily control the size of your backup files. You can do a single job and single backup file 20-30TB in size. Or you can have 10 separate jobs and so 10 backup files of 3TB in size.

That's the thing..

If I have 30-40 VMs, all running windows 2003 or windows 2008, I want the deduplication to be done across the entire backup store (and NOT across backup jobs). VDR does this - I can have 4-5 backup jobs, but all blocks that are similar across jobs that are located on the same backup store get deduplicated (using VDR). This is what I want for maximum space savings.

Gostev · ‎07-21-2010

If I have 30-40 VMs, all running windows 2003 or windows 2008, I want the deduplication to be done across the entire backup store (and NOT across backup jobs). VDR does this - I can have 4-5 backup jobs, but all blocks that are similar across jobs that are located on the same backup store get deduplicated (using VDR). This is what I want for maximum space savings.

I am still confused... VDR dedupe store is limited to 1TB in size (per User Guide). For maximum savings, you do not want any limitations on your dedupe store size, otherwise you are forced to use multiple dedupe stores, which negates the whole idea of deduplication. Essentially, the results here are the same as creating multiple Veeam Backup jobs which each producing 1TB deduped backup files.

With 30-40 VMs, all being Windows 2003 or 2008, the best option is to have a single Veeam job backing this up to a single dedupe store - this would definitely provide the maximum storage savings.

jarsenea · ‎07-21-2010

I am still confused... VDR dedupe store is limited to 1TB in size (per User Guide). For maximum savings, you do not want any limitations on your dedupe store size, otherwise you are forced to use multiple dedupe stores, which negates the whole idea of deduplication. Essentially, the results here are the same as creating multiple Veeam Backup jobs which each producing 1TB deduped backup files.
With 30-40 VMs, all being Windows 2003 or 2008, the best option is to have a single Veeam job backing this up to a single dedupe store - this would definitely provide the maximum storage savings.

I understand where you're coming from. One of my gripes is the 1TB limit. My hope is that in the future, VDR will allow 2TB backup stores (or bigger) and more than 2 stores per VDR appliance.

One aspect that I forgot the mention is FLR (File Level Restore) client for VDR is probably one of the best I've used. Because it works within the actual VM and doesn't require credentials to restore a file from that particular VM, I can give access to various administrators on various servers and let them do their own restores. The FLR process with VDR is absolutely fantastic and I wish all applications of this type had this kind of functionality. PhD's solution is similar (via web interface), but I abhor the vRanger/Veeam method of restoring individual files. VDR FLR is by far the quickest method to restore individual files. The proof for me is in the pudding as I've used it on numerous times to restore files and it's taken me only a few minutes. The fact of the matter is that I will more often than not use FLR instead of doing a complete vm restore (other than catastrophic failures or massive configuration changes). I put a lot of weight on the capabilities of the FLR client.