Experience with Avamar?

[jasper9890]

Does anyone have any experience with Avamar? I know the client can run inside the VM's just fine like it was a physical machine, but do any of you additionally use it in any way to backup full machine snapshots?

[epping]

i have done some testing with Avamar but i did not hook it in to VCB, that would the functionality that u mention, will do some more testing soon with the data vault product.

[arisaperstein]

Yes. I use Avamar as an agent inside the COS. We run a perl script to snapshot all the vmdk's registered to the host. Then we use a CLI provided by Avamar to create datasets on the fly to backup all the files except the snapshot related files. I cannot go into exact detail because I am under NDA with Avamar and we did a lot of customization, but the solution does indeed work. If you choose to use Avamar with VCB, you need to get clarification on how they count "data protected"; Avamar is licensed based on "data protected". If Avamar sees the same file multiple times, it counts it as the same file, but if the same file changes date and time stamp and size, it may consider the file NOT the same and mess up the calculations of "data protected". It's kinda complicated. But it works really well.

For those not familiar with Avamar, it is a date de-duplication backup software product. It "finds" block commonality and only backs up changed data. In a traditional backup, if I had 10 20GB identical files, the backup software would backup 200GB. Avamar would backup 20GB since all the other data is common to all the files. It then keeps a database of all the pointers in order to restore any of the 10 files. Additionally, let's say you have a 20GB file and you make 1MB of changes. Netbackup/CommVault/Legato, etc. would see that this file changed and backup the entire 20GB file again. Avamar would only backup the 1MB that changed and reference the original 1999MB. Because of the nature of VMDK's, we get 98% commonality. So, I have 350 20GB vmdks which would normally require 7TB of backup space, now only consume 27GB of backup space. Oh, and not just backup space, but strain on the network. I am only passing 27GB across the wire, not 7TB

[jasper9890]

Interesting stuff, thanks.. Can you speak a little bit to the advantages of doing it that way versus using a client in each file system? How many ESX hosts do you have the client running on? How many VM's total?

[arisaperstein]

We are also running the Avamar client inside the guest VM's as well for file level backup. We found that the Avamar client causes FAR less strain on the guest session (CPU/RAM/Disk/Network) than a product like Netbackup (which we run in the rest of the environment). Firstly, the de-dup allows for much quicker backups (obviously). Secondly, we did a comparison to overall performance of the guest and the host during a netbackup and during an Avamar backup. Avamar "behaved" far better. The windows service is designed to run as a low priority thread so if another process needs CPU, Avamar gives it up. On another note, we can restore an entire file system of 10GB of data in 6 minutes. We were able to restore a VMDK of 12GB in under 10 minutes.

In short, we run it both in the guest, and in the COS and we have been very successful.

[jasper9890]

Interesting, the professional services guys i'm working with are saying that's out of the ordinary and against their best pracitices. They say to only do it in one of the three: 1) within the vm, 2) on the esx server or 3) with VCB (though i don't see the difference between 2 and 3 so far.)

If you remember, could you private message me the name(s) of the guys you worked with?

[arisaperstein]

I cant imagine why it would be "against their best practices". Each of the two types of backups serve different purposes (at least for us). Running the agent within the guest allows for restoration of individual files. Running the agent in the COS and backing up vmdk's allows for the restoration of the entire machin back to a point-in-time. We take VMDK backups for the same reason we take images every week of allour windows (physical servers) -- worm mitigation.

[jasper9890]

How difficult is the restore process? Can you restore directly from avamar to an ESX server if it was originally backed up via the proxy?

Also another doc i'm reading about the pre/post scripts, at least in the example, shows you have to create a new policy for each VM. Can you do it by folder or group in any way?

[epping]

this is a great thread, very useful

i can see an advantage for having an agent in the COS and agent in the VM and an Agent in the VCB proxy, here is my reasoning.

OK i want to do backups using VCB as it offloads the effort from the ESX host, VCB is great for backups but rubbish for restores,

So i need the agent in the COS because i want to do a direct restore to the host, not via the VCB proxy and then to the host...too manual

Ok so now for the file level restore, i think i am right saying that even if the backup was done via VCB a file level restore is possible, u have to mount the .vmdk file on the proxy and then you can go in and get the file u want, however if there is an agent in the VM does it let you do a redirected restore?

also on the de-duplication, when you do a backup from the COS i guess it sees this as .vmdk and the software will see this as new and completly different data than if the agent is within the VM, does this mean that infact you require twice as much storge for the backups (ignore the saving from de-dup for now) also is this the issue u mentioned about lisencing as it is worked out on protected data (and in effect it is the same data just in a different format).

thanks for all the info

what backend are you using, Data Store, virtual Edition or rackservers with local disk??

are you using the Avamar front end or are you using Legato??

thanks

[epping]

"Can you restore directly from avamar to an ESX server if it was originally backed up via the proxy?

"

U can do this but only if there is also a agent in the COS as well (i states this in there documentation), remember the restore will also be going across the network not the fibre (i think).

One more point anyone know what they are planing to do when 3i comes out, there is no COS anymore!!!

[arisaperstein]

If you were to use Avamar in conjunction with VCB, you would have to follow the same procedures for restoration as you would with any backup software. Avamr will allow you to restore the data to any destination you wish. Our Avamar infrastructure uses several DL320S each with 10TB of local storage. the Avamar infrastructure is set up as RAIN (redundant array of independent nodes) so there are at least 3 nodes.

[jasper9890]

Does your NDA prevent you from sharing your pre/post scripts? Can you speak to how that came about if so, would it be something my group could get involved with too?

The ones i've been given to work with are extremely primiative.

[arisaperstein]

I found out that while our NDA wouldnt prevent me from sharing the pre/post scripts, interal rules do; sorry. But here is the gist of the scripts which are written in PERL and use the VMware Perl SDK and the Avamar CLI command sets:

1. Get a list of all registered VM's on a host

2. Check for available disk space on the datastore -- to make sure it has room for snapshots

3. Remove any current snapshots

4. Snapshot all registered sessions

5. Take the list on VM's and create the dataset to be backed up via Avamar

6. Create an inclusion list on *.vmdk *.vmx, *.vmxf,

7. Create an exclusion list of *.vmsn, -000001.vmdk, *.vswp, *.log

8. Start the backup job

9. A loop checks the status of the backup job from Avamar

10. When the status is finished, run a post-script that unsnaps all the sessions

[jasper9890]

Very cool.. So if i understand it right, that is done then on the ESX server not VCB proxy, or can you do that even on the VCB proxy? My dablings in the SDK has only been superfical trying to get some extra monitoring enabled....

[arisaperstein]

We do not use the VCB proxy. We found that it has way too many deficiencies.VMware admits that you cant really run more than 8 concurrent backups. I did the math and 125 sessions would take the proxy 64 hours to backup em all up.

We run everything from the COS.

[jasper9890]

When you do a restore, after the machine is restored from Avamar, can you just register the machine's .VMX thru the gui or do you have to do the vcbRestore command? If so, What's an example of a working vcbRestore command? I have the data restored back from Avamar but i can't seem to either convert it back or start it up in the format it wants.

[Schorschi]

Same here. We gave up on all VM snapshot methods, esxRanger, esxMigrator/Replicator, DoubleTake for ESX, etc. Anything that runs in the COS has been kicked to the curb, anything that uses the VMware API for backup just does not get the job done at our scale, specifically not stating it, but it big and global. We are moving to LUN snap/cloning/long-haul replication methods as fast as we can. Of course, doing this is just one step to moving to SRM, leveraging Storage VMotion, etc., leveraging virtualization in true Global Datacenter architecture, inter-site, not just intra-site.

One thing more, VCB has proven to be horrible no matter what we tried, it works, but does not scale. On VCB server for every couple of ESX hosts, just was plain crazy. For our smaller sites will expect to use AVAMAR transporting back to core sites replicated data. Once data is replicated, we use a dedicated ESX host to act as a bridge to link into NetBackup, since right now, NetBackup can not natively access VMFS, the NetBackup IP client is a bottleneck. This is a tactical solution, because we expect to replace all our existing EMC SAN infrastructure with something that is a Virtualized Storage architecture, no one product has jumped out at us as yet, but NetApp, EMC, etc. all have ideas moving from paper or better in development/promotion to product offering. But all the options in concept work logically like IBRIX or similar concepts... between images, thin-disking, and smarter LUNing, this time next year we should be in good shape.

[lueckma]

This is a really interesting thread for me because we will get our Avamar store with 12TB RAW capacity in the next two weeks.

When I read about heavy usage of backup agents installed under the COS. But now I will go this way.

I hope we will get the pre/post scripts from EMC to create a good backup solution.

Actually we save our VMs with a NetBackup guest agent and every week we create a VMDK backup with vRanger.

Everyone who can share experiences with Avamar should post it here, I think there are not many information about practical usage of Avamar and VMware backups.

I hope I can share my experiences soon.

Marc

[matuscak]

We're been using Avamar for about two years and with VMware for about a year. All in all, the product is great (EMC, on the other hand...) We're running it inside the VMs at the moment so no backup of the VMDKs. That is something that I'd like to get going here, but we've not been able to get the time to focus on it. The load on the hosts with Avamar running inside the VMs is quite reasonable. The first backup you run will take quite a while since it has to chunk, hash and store pretty much everything. After that it pretty much flies through the backups.