ascheale
Contributor
Contributor

VCB Performance optimization - how?

Hi all,

my setup looks as follow:

2x HP DL380G5 VI 3.5 VC2.5 -


SAN -


HP EVA4100

1x HP DL360G5 W2k3 X64 R2 SP, 2 FC HBA newest MPIO drivers, BackupExec 11d and VCB Proxy -


SAN -


HP EVA 4100 -


SAN HP MSL4048 LTO3

When I do backup via VCB / Backup Exec I only get about 950MB transfer rate.

Before Virtualization we did SAN Snapshot and backup up this direct via FC to MSL4048.

We get about 3800MB transfer rate.

How can I tune VCB perfomance?

Regards, Alex

0 Kudos
17 Replies
beckhamk
Enthusiast
Enthusiast

are you doing file level backups or full vmdk backups? Also are you running more than one backup job at a time? I know that the backup performance slows down when more than one job is running or when you are doing file level backups with lots of little files.

0 Kudos
ascheale
Contributor
Contributor

There´s is only one backup job at the moment.

We do file-level backup. We tried hard- and software compression without any changes.

Alex

0 Kudos
ascheale
Contributor
Contributor

btw .. which transfer rates do you get?

Is my rate maybe "normal" ?

Alex

0 Kudos
beckhamk
Enthusiast
Enthusiast

I believe there are many factors that influence that rate you are seeing. PRE vm era. The rate would depend on disk speed, raid level and the number of files you are backinh up. We would see systems with small number of files backup faster than ones with large number of files (not size mind you). We see on our 15 disk md1000 when two FUllvm backups are running on the same system BUT to two different md1000's we see a rate around 1600 per job. A single job running 2500-2600. But these are fullvm's not file level. File levels will influence the rate because of all of the time backupexe itself or the agent need to looks for files on the disk, index save and or compress them for backup etc.

0 Kudos
depping
Leadership
Leadership

Can you post your VCB's config.js

Duncan

My virtualisation blog:

0 Kudos
depping
Leadership
Leadership

could you post your config.js?

Duncan

My virtualisation blog:

0 Kudos
ascheale
Contributor
Contributor

Here´s my config.js

Alex

0 Kudos
ascheale
Contributor
Contributor

I think my config.js looks good.

What do you think ?

Alex

0 Kudos
ascheale
Contributor
Contributor

Hi,

I changed my setup a little bit.

My Mount point for the file-level Snapshot on VCB Proxy is now a LUN on the EVA4100.

For Image-Level Snapshot / Backup to MSL I get 5900MB/min which is real fast.

For File-Level Snapshot / Backup to MSL I oly get 1000MB/min which isn´t fast at all.

VMware told me that the disk / partition where the snapshot is mounted on is responsible for backup speed.

I cannot say this is true because of my test with image versus file-level backup.

Please give me a hint.

Alex

0 Kudos
dconvery
Champion
Champion

I have done quite a few VCB installations. The best I can eek out of VCB is about 1GB per minute on a VMDK backup. That is even using tier 1 disk, like a DMX. What you need to do is set up a large server, like a DL580 with six or more fibre channel adapters. Then, you can stagger the jobs to run different stages. Just remeber that VCB is VERY I/O intensive and you can cause a storage bottleneck if you push too hard. Then, the VMs will suffer.

Start a VCB FullVM to copy to FCA#1. on fabric #1. Once complet, copy to tape on FCA#3 on fabric #1. At the same time, start a FullVM on FCA#2 on fabric #2. Then, keep juggling the backups.

Dave

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
0 Kudos
ascheale
Contributor
Contributor

uuh. that´s boring. I thought I will get very more performance.

With remote Agent and Open File Agent like in BackupExec 12 I will get same results by backing up other the network.

Speed was my reason to choose VCB, but now I have to think about it.

Is there any other possibility ?

Alex

0 Kudos
dconvery
Champion
Champion

I think that the ESX servers must throttle the VCB process back a little bit so that the VMs do not suffer from the heavy disk I/O. You can test the difference between the agent-based and SAN based backups to see if it makes a difference. Where you will see a difference is the CPU utilization on the VMs. Also, assuming multiple VMs being backed up, the network I/O could bottleneck unless you have a backup network set up. Most of the installs that I have done were to decrease CPU strain and to take the load off of the network. If the VCB server acts as a media server (backups go directly to tape - not to another backup server), then backups become semi - LAN free. Also, if you look at "enterprise" backup software, like Netbackup or TSM, the client cost is roughly $500-$600 (US), so there is a HUGE cost savings as well if you consider hundreds of VMs. I don't know how much the client cost is for BE.

Dave

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
0 Kudos
ascheale
Contributor
Contributor

I know what you mean but time is my big problem !

Before virtualisation we solved that problem by implementing SAN based snapshots and got very good backup performance.

Now with VCB time is almost the same as before when backing up with agents and my datastore is growing up !

How do you do your backups when you have 4 TB or even more ?

We need full backup, no differential ...

Alex

0 Kudos
dconvery
Champion
Champion

You MAY be able to script something on your EVA using Replication Solutions Manager to snapshot the VMs and then snapshot the LUN on the array... I have never tried it before...

The only other thing you can do to decrease backup time is to use a separate VCB server for each datastore LUN. Your LUNS should have no more than 15-20 VMs each.

Dave

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
0 Kudos
beckhamk
Enthusiast
Enthusiast

We had done alot of testing using backupexec and the plain throught is that file level backups are never going to be super fast even over FC. The more smaller files you have the longer it will take to backup. The bottle neck we found is that with data drives with lots and lots of small files ie 300,000 to 500,000 files there is a lot of running around going on even with vcb. For example a system with about 500,000 files took us 3.5 hours via 1gb lan backup using BE11d, then once that same machine became a vm and then we tested 4gb fc file level backups and the fc speed was faster but the total backup time became 3hours vs 3.5hours. So we now do fullvm backups with are way way faster than filelevel. You can mount the images and do file level restores :).

You might want to consider s CDP option of b ackup exec or r1softs CDP server they do block level backups. We really like veeam backup as its cuts out having to dump the full vm to disk before backing up. and it has data deduplication built in. We are only waiting for email notifications Smiley Happy

dconvery
Champion
Champion

I haven't had hands on with Veeam Backup, but I have heard only good things about it.

Dave Convery, VCDX-DCV #20 ** http://www.tech-tap.com ** http://twitter.com/dconvery ** "Careful. We don't want to learn from this." -Bill Watterson, "Calvin and Hobbes"
0 Kudos
ascheale
Contributor
Contributor

@beckhamk: You made my day. I will go and make Image-Level backups. If I have to restore some files I will mount the restored vmdk´s.

Didn ´t know that this is possible with VCB at all.

I know it from workstation only.

Regards,

Alex

0 Kudos