2 Replies Latest reply on Apr 18, 2010 12:52 PM by mitchellm3

    Our Arcserve, VCB and VDDK Backup Experience...and a question

    mitchellm3 Hot Shot

      Our company has had virtualization in the datacenter for quite a long time but when we started implementing it, we decided it would be best to continue to treat the VMs like they were physical when it came to backups.  This was to avoid confusion for our operations group as well as avoid introducing a new backup product.  At the time we ran Brightstor 11.x on the AIX platform so most plug-ins didn't work anyways.

       

      Due to CA's roadmap, we went to Arcserve 12.5 on Windows and we also decided to look into using VCB.  The main reason was because we wanted to shorten the backups of our 3 file servers which are 1+ terabytes in size and also we thought that having the raw backups of VMs would make restores of VMs easier as well as simplify servers that only require monthly backups.  The short of it was that we thought it would be good to utilize our SAN to speed up backups.

       

      Our first attempt was to test the VCB (vcb 1.5 u1) backups on our 3.5 farm which is where our new fileserver VMs were located.  They are all 1TB or bigger but only have about 400G of test data on them.  We started with just raw mode and it backed up just over 400G in decent time because it squeezed out the white space.  The next try...and our goal...was to back them up in raw mode with file level detail.  This failed because our mount location was too small.  Apparently if the VM is 1TB, you need more than 1TB of mount space to back up the VM.

       

      We ended up giving the VCB server a big enough mount location and then kicked off a backup.  This backup was literally taking days.  The problem was that the VCB was mounting the VM which was taking forever.  I believe the problem has something to do with the 2GB/minute limitations of the windows 2003 cli.

       

      This prompted another call with CA and they thought if we were using vSphere we could use the VDDK method which would keep us from having to mount the VM so everything would speed up.  Since that is where we are moving, we migrated the file servers over to our vSphere 4.0 farm, rezoned the vcb server and then installed that toolkit which allows us to use VDDK.  Well, once this was done, all our backup tests failed.  Using VCB from the cli worked but backups from Arcserve using VDDK failed.  We could not mount the VMs.

       

      On our next call with CA they tested it out and saw that VCB could mount the VMs but not VDDK.  So, and here is the kicker, they said the problem was that we use VMs with multiple drives with the same name on different datastores.  I couldn't believe this was the problem because that used to be a vcb limitation with 3.0x but surely not VDDK and vSphere 4.0.  Well we took one of the file servers down, renamed the 4 other vmdk files and voila!  The darn thing backed up.  Here are the stats.

       

      Total Session(s)............. 1

      Total Directories............ 0

      Total File(s)................ 9

      Total Skip(s)................ 0

      Total Size (Disk)............ 1,157,133.75 MB

      Total Size (Media)........... 1,158,263.81 MB

      Elapsed Time................. 2h 57m 3s

      Average Throughput........... 6,541.99 MB/min (3 times faster than over our backup network)

       

      So, after this long post I actually have a question.  Does anyone else using VDDK backups for VMs with multiple .vmdks on different datastores have to go through all this and start renaming .vmdk files?  This can be resolved going forward by adjusting our server build guides but we have over 200 existing VMs with 2 disks on different datastores that have the same name.  Boy I wish vmware allowed you to pick the name of the .vmdk file you create like they did though the MUI back in the day.

        • 1. Re: Our Arcserve, VCB and VDDK Backup Experience...and a question
          Frank.Hornscheidt Lurker

          We use VDDK with Arcserve too, but our VDDK seams to load the full VMs via network instead of the san-connection.

          If you have used V4 from the beginning there should be no problems, because VSphere names new hdd-files as their vm-name.

          You should use scripting to rename each of your 200 machines hdds. That is an affordable investion of time - and it is easier to guess the vm from the hdd-file-name afterwards.

          • 2. Re: Our Arcserve, VCB and VDDK Backup Experience...and a question
            mitchellm3 Hot Shot

            Well as of right now we have been renaming them when we upgrade the VMtools and virtual hardware.  The big limitation of this manual approach...other than the time doing so, is that if you storage vmotion a virtual disk from one datastore to another, the .vmdk will be renamed back to the origional name.  This means that you can't storage vmotion these servers without having to rename the files again.  This of course means downtime to rename the disks before the next backup...which we usually run nightly.

             

            I did open a call with VMware to see if they were working on a fix so we don't have to rename the .vmdks.  They said there is no fix because it works just like they designed it too.  They said that when you take a snapshot each .vmdk file has its own unique name.  They said the problem lies in the way that CA is making calls through the vddk...so it's their problem and not VMwares.  I haven't gotten our backup guys to open a ticket with CA for this yet.  They have too many issues open with them already concerning other vddk/vcb backup issues.