VMWare Data Recovery - Performance Issues

[kcucadmin]

i installed VDR over the weekend. and i'm pretty worried about the performance.

here are a sample of the backup logs. the odd thing is all of these VM's are THIN provisioned. and are no where NEAR as big as the space i have setup for them. most of them are 10-20gbyte total backups. NGTS03 for example is a Terminal Server clean 15gb used space

12/11/2009 6:01:41 PM: Normal backup using NGCSG

12/11/2009 6:02:36 PM: Copying NGTS03

12/11/2009 6:02:53 PM: Performing full back up of disk "[Site1_OS1] NGTS03/NGTS03-000001-flat.vmdk" using "SCSI Hot-Add"

12/11/2009 9:00:08 PM: Task completed successfully

12/11/2009 9:00:08 PM: Completed: 5 files, 80.1 GB

12/11/2009 9:00:08 PM: Performance: 462.3 MB/minute

12/11/2009 9:00:08 PM: Duration: 02:58:25 (00:01:12 idle/loading/preparing)

12/11/2009 8:25:08 PM: Normal backup using NGServers Backup

12/11/2009 8:25:13 PM: Copying NGRTS

12/11/2009 8:25:26 PM: Performing full back up of disk "[Site1_OS1] NGRTS/NGRTS-flat.vmdk" using "Network"

12/11/2009 10:14:19 PM: Task completed successfully

12/11/2009 10:14:19 PM: Completed: 5 files, 80.1 GB

12/11/2009 10:14:19 PM: Performance: 752.3 MB/minute

12/11/2009 10:14:19 PM: Duration: 01:49:10 (00:00:16 idle/loading/preparing)

12/11/2009 8:22:16 PM: Normal backup using NGServers Backup

12/11/2009 8:22:26 PM: Copying NGDATAEXCHANGE

12/11/2009 8:22:41 PM: Performing full back up of disk "[Site1_OS2] NGDATAEXCHANGE/NGDATAEXCHANGE-flat.vmdk" using "Network"

12/11/2009 11:11:38 PM: Task completed successfully

12/11/2009 11:11:38 PM: Completed: 5 files, 80.1 GB

12/11/2009 11:11:38 PM: Performance: 485.0 MB/minute

12/11/2009 11:11:38 PM: Duration: 02:49:18 (00:00:24 idle/loading/preparing)

12/11/2009 11:11:38 PM: Normal backup using KCUC Apps

12/11/2009 11:11:43 PM: Failed to create snapshot for KCUCAPP1, error -3960 ( cannot quiesce virtual machine)

12/11/2009 11:11:43 PM: Task incomplete

12/11/2009 10:14:21 PM: Normal backup using Pulse Servers

12/11/2009 10:14:35 PM: Copying VS-KCUCVM

12/11/2009 10:14:53 PM: Performing full back up of disk "[Site1_DB2] VS-KCUCVM/VS-KCUCVM-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 1:00:51 AM: Task completed successfully

12/12/2009 1:00:51 AM: Completed: 5 files, 75.1 GB

12/12/2009 1:00:51 AM: Performance: 462.8 MB/minute

12/12/2009 1:00:51 AM: Duration: 02:46:28 (00:00:31 idle/loading/preparing)

12/11/2009 6:03:11 PM: Normal backup using Backup Job 1

12/11/2009 6:03:36 PM: Copying NGINTERFACE

12/11/2009 6:04:06 PM: Performing full back up of disk "[Site1_DB1] NGINTERFACE/NGINTERFACE-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 2:12:37 AM: Performing full back up of disk "[Site1_OS1] NGINTERFACE/NGINTERFACE-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 3:50:31 AM: Task completed successfully

12/12/2009 3:50:31 AM: Completed: 7 files, 410.1 GB

12/12/2009 3:50:31 AM: Performance: 716.0 MB/minute

12/12/2009 3:50:31 AM: Duration: 09:47:13 (00:00:55 idle/loading/preparing)

12/11/2009 8:49:32 PM: Normal backup using Pulse Servers

12/11/2009 8:49:52 PM: Copying KCUCDC01

12/11/2009 8:50:08 PM: Performing full back up of disk "[Site1_Pulse1] KCUCDC01/KCUCDC01-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 5:17:59 AM: Task completed successfully

12/12/2009 5:17:59 AM: Completed: 5 files, 135.7 GB

12/12/2009 5:17:59 AM: Performance: 273.5 MB/minute

12/12/2009 5:17:59 AM: Duration: 08:28:22 (00:00:36 idle/loading/preparing)

12/11/2009 11:11:45 PM: Normal backup using KCUC Apps

12/11/2009 11:12:12 PM: Copying KCUCAPP3

12/11/2009 11:12:27 PM: Performing full back up of disk "[Site1_OS1] KCUCAPP3/KCUCAPP3-flat.vmdk" using "Network"

12/12/2009 6:26:20 AM: Task completed successfully

12/12/2009 6:26:20 AM: Completed: 5 files, 70.1 GB

12/12/2009 6:26:20 AM: Performance: 165.2 MB/minute

12/12/2009 6:26:20 AM: Duration: 07:14:30 (00:00:39 idle/loading/preparing)

12/11/2009 6:03:12 PM: Normal backup using Backup Job 1

12/11/2009 6:03:51 PM: Copying NGREPORT

12/11/2009 6:04:25 PM: Performing full back up of disk "[Site1_DB2] NGREPORT/NGREPORT-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 3:56:14 AM: Performing full back up of disk "[Site1_OS2] NGREPORT/NGREPORT-flat.vmdk" using "SCSI Hot-Add"

12/12/2009 7:00:48 AM: Task completed successfully

12/12/2009 7:00:48 AM: Completed: 7 files, 572.1 GB

12/12/2009 7:00:48 AM: Performance: 754.5 MB/minute

12/12/2009 7:00:48 AM: Duration: 12:57:26 (00:01:13 idle/loading/preparing)

[admin]

A few things to note

1) Since this is the first backup you performed, all blocks of data needed to be copied plus deduplicated for the first time. Todays backup should run signficantly faster since VDR will only be transferring changes blocks (incremental backups) and plus expect to do less deduplication. Please post those logs as comparison.

2) Some the VMs were backed up using hot add (should be faster/more efficient) and some were over the network. VDR by default attempts to backup via hot-add and default to network if it cannot - for example, the ESX does not have the hot add license or the datastore cannot be "seen" by the host that VDR is running on. This could be the reason why KCUCAPP3 is the slowest of the lot in the terms of MB/min. But not sure about NGRTS - backed up over network but significantly faster than even other hot -add VMs (like

3) Are you writing to a CIFS share or a VMDK? If CIFS, then ultimate performance is going to limited by the bandwidth available over LAN. Or are you backing up to a remote location?

4) Are you are staggering the various backup jobs - why not run them concurrently (i.e have the same backup window) - VDR can run 8 simultaneously backup jobs

[kcucadmin]

all these HOSTS have either Advanced or Enterprise Plus Licenses. shouldn't be a License Issue.

they all connect to backend storage the same way.

all my TS servers KCUCAPPs and NGTSs use software iscsi initator. so why would some go hot add and some network. very confusing to me.

i'll post the entire log for all my VM's 27 in all. again. 8 esx hosts running update 1. all have NFS and iSCSI data stores. most of my VM's are on NFS data stores only DB's are on iscsi. the VDR Applicance is on iscsi.

I actually have a SR Case open now, i'm just trying to get a feel for others regarding backup times.

i have attached the FULL Log. as you can see some VM's on the same data store do HOT ADD and others do network.

also i should note what spurred the SR Ticket was on sunday the follow up backup, after 3 backups i started to recieve errors, which i belive are at the end of this log. it kicked off an integrity check, which took 4 hours, found 3 corrupt snapshots, instructed me to mark them for deletion and re-run an integrity job. that job has been running since 12:00 only 72% complete now, 6 hours later.

edit: also i have 2 x gige for Storage to a Cellera NX4 for the backend storage, the dedup store are on a 6disk sata raid 5 file system dedicated for backup, no other activity. the VM's are located on the Same NX4 on shared storage pools 15k sas drives, i have 12 x 450gb 15k sas, 18 x 300gb 15k sas. it shouldn't be a disk bottle neck. or lan bottle neck really. when monitoring the network. the Host the VDR appliance is on, the Storage NIC's are only using about 40% of the gig nic. the celllera has a 3 port LACP trunk wasn't maxed either.

the VDR vm was maxed on processor and memory for the entire backup, running 8 tasks. i tried increasing the cpu count to give it more processor recources, but it reverted back to 2 cpu's after a reboot, not sure why.

the VDR dedup store is on a Thick Provisoned 400gbyte virtual disk in the VM, that is on a iscsi Data Store.

not sure if that helps any.

[admin]

What version of VDR are your running?

Also, what is your destination disk?

[admin]

we have seen similar symptoms with VDR 1.0.x but have not seen it reappear with VDR 1.1 (available since Nov).

[kcucadmin]

brand new this weekend, i downloaded latest cd build on website it's 1.1. and i basicly started fresh friday night. this is a new install so i dont mind, scrapping what i got and started over,

i initally started with NFS mounted data store for the dedupe Virtual Disk. but after reading found some suggestiong to thick provision less than 1tb and only provision as much space as you think you will use.

i initaly provisioned 300gbyte, extended to 500gbyte. have 250gbyte free after backups finished.

i updated my post with more host,backend storeage info.

edit: just logged into console of VDR Appliance : VMWare Data Recovery - 1.1.0.7.7 Build 207380

[admin]

Since you have an SR opened, lets work that channel and then post back to the community when we have root caused the various issues. I would address the issues with the damaged restore points and the destination disk (what VDR uses to write data to) first. Worry about the hot-add, etc later since they are more performance related. I have been tracking the community since 1.1 was released and I have not seen any reoccurance of the damaged restore point issue, so it would be interesting as we get updates via the SR.

Support will probably want you to provide more logs so we can see more information about what is going on. See the following KB

[kcucadmin]

i've basicly chucked everything i have and am starting fresh. we'll see if i have any better luck in a pure thick, iscsi deployment.

still if NFS isn't goign to work, you should just say that =D