crazex
Hot Shot
Hot Shot

Backup Speed with VCB and BackupExec

Now that I've managed to get the BEIM for VCB working, I am wondering why is my backup speed so slow?

http://www.vmware.com/community/thread.jspa?threadID=85109&tstart=0

When I process my backup jobs via Backup Exec, I am using the BEIM to make snapshots, and mount them to a SAN LUN on my VCB proxy, but the write to tape is only in the 250-450MB/min range. The tape library is a Dell TL4000 with a native fibre connection. We are using Qlogic 4GB switches for our fabric. When I run an esxRanger backup, whereas, the backup makes a local copy to the volume, and then moves it to tape, I am seeing speeds in the 4000-5000MB/min range. Any idea on why the VCB jobs get written so slowly? All the SAN/Tape/Proxy zoning is working properly. I am currently using a single path, as this is just for testing.

-Jon-

-Jon- VMware Certified Professional
0 Kudos
20 Replies
Jeff1981
Enthusiast
Enthusiast

Hi Jon,

good to see you got the backup running, now let's try to get you out of this problem as well Smiley Happy

If I'm correct, you're are now snapshotting to your SAN LUN and then backing up those snapshot files directly to tape, correct? Whereas for your esxRanger backup, you're first creating a backup file on your LUN and then moving (moving or duplicating?) that file to tape.

The difference is, that when duplicating a backup file to tape, the data is streamed to the tape, which results in better performance (and improved durability for your tape and tape system, since the tape doesn't stop and start in between files) Could you try to create a backup to disk in stead of to tape and report back what the speeds are then?

Further more, here's a link about speed problems in BackupExec (11D):

https://forums.symantec.com/syment/board/message?board.id=115&message.id=897&jump=true#M897

kix1979
Immortal
Immortal

The difference is, that when duplicating a backup

file to tape, the data is streamed to the tape, which

results in better performance (and improved

durability for your tape and tape system, since the

tape doesn't stop and start in between files) Could

you try to create a backup to disk in stead of to

tape and report back what the speeds are then?

Also, keep in mind that all tools that leverage VCB (except esxRanger) write the data to disk in an uncompressed format and actually write it twice to disk, this is the VCB functionality. This adds to the time that it actually takes to write the file vs. one write function that is compressed. What we do with esxRanger is intercept the data stream from VCB and compress it and write to disk in a more structured format.

Kix @ Vizioncore

Message was edited by:

kix1979 for spelling

Thomas H. Bryant III
0 Kudos
crazex
Hot Shot
Hot Shot

That is correct. I am mounting the snapshot to the SAN LUN and then performing a backup to tape. With esxRanger, I am first making a local backup of the VM, and then copying that to tape. Is the speed issue do to the fact that the data is going directly to tape, instead of to the hdd array?

The difference is, that when duplicating a backup

file to tape, the data is streamed to the tape, which

results in better performance (and improved

durability for your tape and tape system, since the

tape doesn't stop and start in between files) Could

you try to create a backup to disk in stead of to

tape and report back what the speeds are then?

I set the job to backup to disk, and it is running even slower, somehwere in the 95-200MB/min range. These speeds are really a draw back, as I don't want to have large jobs running for 20+ hours.

Further more, here's a link about speed problems in

BackupExec (11D):

https://forums.symantec.com/syment/board/message?board

.id=115&message.id=897&jump=true#M897Hi Jon,

good to see you got the backup running, now let's try

to get you out of this problem as well Smiley Happy

If I'm correct, you're are now snapshotting to your

SAN LUN and then backing up those snapshot files

directly to tape, correct? Whereas for your esxRanger

backup, you're first creating a backup file on your

LUN and then moving (moving or duplicating?) that

file to tape.

-Jon- VMware Certified Professional
0 Kudos
Jeff1981
Enthusiast
Enthusiast

That is correct. I am mounting the snapshot to the

SAN LUN and then performing a backup to tape. With

esxRanger, I am first making a local backup of the

VM, and then copying that to tape. Is the speed

issue do to the fact that the data is going directly

to tape, instead of to the hdd array?

Well, it could've been, but you've ruled it out by experiencing even slower results in directly backing up to disk.

Just to make sure: The SAN LUN you're mounting to, is that the same one as where you stored your backup-to-disk file? And is it also the same as where the esxRanger file gets written to?

crazex
Hot Shot
Hot Shot

Well, it could've been, but you've ruled it out by

experiencing even slower results in directly backing

up to disk.

Just to make sure: The SAN LUN you're mounting to, is

that the same one as where you stored your

backup-to-disk file? And is it also the same as where

the esxRanger file gets written to?

The esxRanger files get written to the SAN LUN, but my backup-to-disk is setup as one of the local drives on the server. Do you think it would be faster to mount the snap, backup to the SAN LUN, and then back that file up to disk? I think the slow process will still be backing up the data from the snapshot, whether it is to tape or disk.

-Jon-

-Jon- VMware Certified Professional
0 Kudos
Jeff1981
Enthusiast
Enthusiast

Backup to the SAN LUN should suffice. It's just that I've seen some strange things in backup speeds using BackupExec so I'm just ruling out every possibility here. Also, when you create a backup-to-disk folder on your SAN LUN, put a checkmark at "allocate maximum size for backup file" (properties of the backup-to-disk folder in BackupExec) and set Number of concurrent jobs to "1".

Last weekend, we backed up all our .bkf files to tape, formatted the SAN LUN, recreated the backup-to-disk folders using the above settings and speeds went booming.

0 Kudos
crazex
Hot Shot
Hot Shot

Backup to the SAN LUN should suffice. It's just that

I've seen some strange things in backup speeds using

BackupExec so I'm just ruling out every possibility

here. Also, when you create a backup-to-disk folder

on your SAN LUN, put a checkmark at "allocate maximum

size for backup file" (properties of the

backup-to-disk folder in BackupExec) and set Number

of concurrent jobs to "1".

I tried using the backup-to-disk folder on the SAN LUN, but my speeds are still very slow. Somewhere in ther ange of 100-200MB/min

Last weekend, we backed up all our .bkf files to

tape, formatted the SAN LUN, recreated the

backup-to-disk folders using the above settings and

speeds went booming.

I would prefrer to go directly to tape, as I don't have the space or the time to backup to disk, and them to tape.

Is anyone getting better backup speeds with other sowfware?

-Jon-

-Jon- VMware Certified Professional
0 Kudos
cheeko
Expert
Expert

esXpress does a pretty good job. Very fast.

cheeko not @ esxpress but a fan ... Smiley Happy

0 Kudos
petedr
Virtuoso
Virtuoso

If you are looking for an alternative backup solution I agree with cheeko, we use esXpress and it works great.

www.thevirtualheadline.com www.liquidwarelabs.com
0 Kudos
crazex
Hot Shot
Hot Shot

I'm not looking for an alternative way of backing up .vmdk files. esxRanger works perfectly for that. I want to be able to do file level backups directly to tape. VCB offers this functionality, but the speed is of some concern to me. I'd really hate to backup a 500-800GB file server at 450MB/min...

-Jon-

-Jon- VMware Certified Professional
0 Kudos
kix1979
Immortal
Immortal

Ah, I didn't see the part about the file level mounts. I thought you were just doing image based backup. There are tweaks that some people have done to improve performance of the file level stuff, I don't have it bookmarked, but the big difference is still the tape drive is reading lots of files vs. one file.

Thomas H. Bryant III
0 Kudos
crazex
Hot Shot
Hot Shot

Yeah. I understand that by doing file level backups it will be slower, but I am inclined to believe that it shouldn't be 1/10th, of what I am getting otherwise. I am backing up the esxRanger files at ~4500MB/min, and the file level at a max of 450MB/min. I would expect more along the lines of 1000-1800MB/min for the file level jobs. Am I asking too much for this?

Also, Kix, any idea when the VCB Plugin for esxRanger will support Win 2k3 SP2?

-Jon-

Message was edited by:

crazex

-Jon- VMware Certified Professional
0 Kudos
johnd
Contributor
Contributor

My first try at VCB with Brighstor Arcserve Backup 11.5 SP3

If I do a file level backup after runninc vcbMounter I see a speed of ±1500MB/Min.

Did not try the block level yet.

Message was edited by:

johnd

0 Kudos
crazex
Hot Shot
Hot Shot

Yeah, it definitely seems as though this issue is isolated to Symantec/Veritas Backup Exec, possibly just 11d. I've talked with people using other products, such as NetBackup, ArcServe, CommVault, etc. and they don't seem to have the same problem. I've searched the Symantec Forums, and there doesn't seem to be an answer their either. I am planning to open a support call with Symantec, as soon as I get some time, as I am hoping maybe there is an unreleased hotfix that will take care of this problem.

For now, I'm using esxRanger for my FullVM backups, via VCB, and BE Agents for my file level jobs. I'd really like to eliminate the agent backups, as it will really cut costs, and also leverage the lan-free functionality of the SAN.

-Jon-

-Jon- VMware Certified Professional
0 Kudos
TinternSchools
Contributor
Contributor

Hi Crazex/Jon,

Did you ever log that job with Symantec and get to the bottom of this problem? Needless to say I'm having the same slowness issue with BE and VCB file level backups over our SAN.

Thanks

0 Kudos
SReuter
Contributor
Contributor

Hello,

I just did some tests with BackupExec 12 and file level vcb over san (not FullVM).

My environment is a LTO4-Library and a 2Gbit FC-SAN-Connection.

I backed up 2 drive letters (C:\mnt\machine\letters\c and c:\mnt\machine\letters\d)

C: contains a defragged plain windows 2003 installation and nothing more.

😧 contains 3,5 GB of userdata that I have copied over from a fileserver.

Now the interesting part:

Backup speed for C: -> 850 MB/min

Backup speed for 😧 -> 3500 MB/min (!!!)

I tested this multiple times, so this is reproducable.

I conclude from this the following:

The speed seems to be affected by the structure/linearness the data is layed out on the disk.

Even that C: is fully defragged (using the build in windows defragmenter), the speed is not that great.

BackupExec just backs up linearly the directory structure, but the files on C: are not laid out that way and the internal Window defrag does not sort by dir/file.

😧 on the other hand has a fresh structure, that has been copied over using the normal windows explorer. This structure is completely clean as all dirs and files have been copied over "sorted".

This is the only explanation for me that the backup speeds are so different.

The observation is furthermore proven by the following:

I ran "jkdefrag.exe -a 7 c:" and the backupspeed of C: jumped from 850 MB/min to 2200 MB/min (!!)

The option "-a 7" sorts the defrag by dirs/files.

JKdefrag can be found at http://kessels.com/JkDefrag/index.html

The program is not suitable for very large file servers, so use with caution!

So, if you have any issues with VCB speed, just try to give the problem vm a new disk (e.g. 10 gb) and copy a bigger directory structure to it. Then do your benchmarks.

Hope this sheds some light into the whole topic here.

0 Kudos
TinternSchools
Contributor
Contributor

Hi,

Thanks for your info. Sounds like you have done quite a bit of testing.

I found that our backup speed dramatically increased when I ticked the "Allocate the maximum size for backup-to-disk files" on the properties for the backup-to-disk folder. I made the "Maximum size for backup-to-disk files" 800GB in line with the size our our LTO4 tapes but have since been advised by a Symantec tech that this is too large and not a good idea. (go figure)

I will try the JKdefrag

0 Kudos
crazex
Hot Shot
Hot Shot

Wow, this is an old thread of mine. I pretty much abandoned using BEVCB, as it was just too slow. The problem was that since drives such as C: contain so many small files, it had to issue entirely too many mount/unmount commands, and thus reduced the backup speed tremendously. Now I know this is a problem that you alway run into with backing up many small files, as opposed to a few large files, but I expected it to perform better than it did. The process definitely worked when backing up large files, but with my environment, this method was not viable. I couldn't afford to have open snapshots all day long, as my Datawarehouse servers have too much data changing on a daily basis, and the delta files would grow exponentially. I finally ended up just using agents inside my VMs to do file level and System State backups, and then I use vRanger ProVCB to do my FullVM backups. VCB is a nice idea, but, in my opinion, it just isn't ready for "primetime".

-Jon-

VMware Certified Professional

-Jon- VMware Certified Professional
0 Kudos
khughes
Virtuoso
Virtuoso

If you are looking for an alternative backup solution I agree with cheeko, we use esXpress and it works great.

Who would've thunk it. Smiley Happy

- Kyle

-- Kyle "RParker wrote: I guess I was wrong, everything CAN be virtualized "
0 Kudos