Hi all,
Thought I'd share my experiences using VDR in a disaster recovery test my company performed a few weeks ago.
Bit of background:
Our DR plan is to recover about 80 servers for our first stage DR set, mostly VMs but a hand full of physicals backed up using Symantec Backup Exec System Recovery (BESR) for image backups. All are recovered into a separate site using vSphere 4.0.
Due to various incompatabilities or other issues we recover using the following methods:
VDR
Stand-alone converter from BESR images
Built-in converter in vCenter client for older versions of BESR
BESR CD booted into a blank VM for images that neither copy of VMware converter can handle, usually slight vagaries in the config files that make Converter not detect the OS or disks.
I'll focus on the VDR restores here, some issues are planning or architecture and some are issues with the product itself and its resiliancy to faults.
One very good positive thing I've got to say is that the backups and the dedupe are great.
Easy to configure, BUT you do need to check them daily as some times they'll wig out and just stop working with no real reason.
Due to various architecture reasons (which are being rectified) we have our layout as such:
Two separate data centres linked by 10Gb dedicated fibre
VDR vApps run in the primary datacentre (PDC) and in the Secondary Datacentre (SDC) the physical backup servers contain CIFS/windows shares to which the VDR vApps connect to and use for backup destinations. (we don't have any production network vSphere hosts at the moment) This setup caused some network contention issues.
Ways to make VDR backups fail:
If for some reason you backup the same server from two vApp separate jobs or VDR appliances you'll get snapshot errors consistantly
The guide says a max of 500GB data stores.. i'd go for 200 to 250 MAX. If something goes wrong (and unfortuanately it does happen often) and it needs to do a full index or integrity check because a backup reported a fault, doing so on a 500 GB datastore TAKES DAYS!!!!!!!!!!!!!!! During this time NO BACKUPS RUN! Keep them small and have lots of vApps rather than larger datastores and fewer vApps. I went from two VDR vApps with two 500GB LUNs on each to 6 VDR vApps running 250 GB LUNs. Backups fail less and the impact of an integrity check rarely means I miss a nights backup.
Keeping the datastores small also means during a DR if it reports that it is corrupt and has to rebuild the index then you'll only be out for a small amount of time rather than 3 days.
Issues we had during our DR test:
Can't restore a VDR image whilst the integrity check is running. Had to cancel it and hope its all good (which it was)
Lock the VDR vApps so DRS doesn't move them around during the DR restores and point all the VMs from that vApp to restore to itself, although stagger the destination datastores if possible to avoide VMDK fragmentation and better FC LUN path utilisation. This results in a far faster restore for all VMs as you're not sucking data in and out of the Host's NICs, all traffic is memory copies.
due to not really thinking about restores (a common practice I've found) and only thinking about backups we ended up putting a lot of our early 1st stage restores on the one datastore server. Our main file servers (3TB of group/home/profiles) along with our VDR backups of Citrix and other key first line restored servers were all stored on the one physical backup server. Thus when we kicked off all those restores at once (well, nearly at once) all the VDR restores started failing with read or write faults. - Solution: spread the load and increase the number of out-bound NICs on the backup servers.
Restoring too many VDRs at once caused many issues including random failures to restore as well as surprisingly different restore times. For example I kicked off a simultaneous restore of 9 Citrix XenApp servers backed up on the same VDR vApp and the same destination datastore (to obtain the best dedupe performance) and some restored in 40 mins (20GB each) and others took SIX hours whilst a few just failed. Retrying those was successful.
Error messages WAAAAAY too vague and quite unhelpful in trying to resolve restore failures in the heat of the moment. 'Read error' or 'write error' being the extent of it. Not too bad when we're doing a DR test, but if it was for real I wouldn't be happy. And it really doesn't give VMware tech support much to go on either!!!! Help yourselves here guys!
No view of the destination datastore sizes and available space. The restore wizard is slow and clunky so when you're flying blind associating VMs to datastores it doesn't feel too good. It means a lot of fluffing around gathering VM sizes, checking and noting down where you'll be restoring each vmdk to and then running the wizard. All that would be cut out if you just had the view of the datastore free space when selecting the server/datastore destinations in the wizard.
Solutions to our issues:
Move the VDR vApps so they run from the SDC
Set the VDR vApps to use local VMDKs rather than CIFS shares
Increase the number of NICs in the teams on the backup servers (teamed NICs do not increase inbound I/O, only outbound. Inbound data traffic must be addressed to the one MAC address whereas outbound traffic is spoofed to appear as though it came from the primary MAC but in fact came from the others)
Recognise which VMs you'll be recovering during which phases of a DR restore and make sure they are all spread across multiple VDR vApps rather than pooled into the one vApp or worse, onto the one Job/datastore. This increases restore throughput.
Restore VMs from the VDR vApp onto its vSphere host rather than sending the data out onto the LAN to other vSphere hosts
Keep your VDR datastores small (150 to 200 GB I reckon) and have more vApps.
Having more VDR datastores increases wasted VMDK overhead (recommendation is to have 50-100% disk overhead from the quantity of data backed up for future change data growth over the life of the backup -- 7 years for most) So, use thin provisioning on the VMDKs or the LUNs at the SAN level
Ultimately this is a free solution and for our use only a stop-gap between using BESR for VM backup, the phasing out of VCB and the VMware Site Recovery Manager implementation.
Happy to answer any questions about our DR and really looking forward to SRM to get rid of a LOT of our manual tasks and hopefully make DR a one day exercise again. 
Overall the DR test was a success, although the ease at which we could kick off many VDR restores at once really bit us in the rear and I've had to break down the restore process into smaller chunks so we don't hit those bottlenecks in the future. The issues we had did definately put a sour taste in the management level's mouth for VDR as a product though and they are still a little skeptical about using it (hey, it's free!)
That said, VDR did do its job in the end and got us through
Cheers,
Steve
