Symantec GRT does not work with VSphere

[gbh66]

Hi All,

It’s been one of those weeks.... so forgive me if seem a little tired and cross.

We upgraded to vSphere a little over 2 weeks ago, and the upgrade went very well, except in one area. Backups.

Symantec GRT in Backup Exec in 2010 does not work with vSphere. When I say doesn't work, it backs up, but it will not restore individual files.

This is the support document that describes the problem.[http://seer.entsupport.symantec.com/docs/351420.htm|http://seer.entsupport.symantec.com/docs/351420.htm]

If you read the support document carefully you will see that;

A restore of the full vm is possible. (Lucky otherwise we'd really be nuked)

It will work if you backup directly from an ESX host

Symantec are aware of the issue but have no concrete plans to fix it. (Ok)

We should all talk to a Symantec salesperson if we want to get it fixed. (Awesome)

But

They offer 2 work arounds

1) Don’t use it... not technically a work around.

2) Back up directly to each of your ESX hosts. Now I haven’t tested what issues that may create for backup and restore jobs, if for example I don’t want to back up or restore an entire host, but I'm guessing if you motion anything and don’t put it back where it's "supposed" to be you're going to have issues.

I have many things I want to say now, a tirade about releasing incomplete products, another tirade about promoting heavily features that don't work, a smaller tirade about the IT industry and its current risk of becoming the used car salesmen of the 21st century... but I'll keep that for another forum and another day.

I will however say this, in general the product does what I'd hoped it would do, backups straight to tape without wasting terabytes of valuable storage for staging, a single process to back up everything, being able to back up an Exchange server’s virtual machine alone and restore individual mail items, fantastic. Except that it doesnt work with vSphere.

The fact that Symantec even need to question whether this is a significant business issue means that they are completely out of touch with the requirements of VMware vSphere users, or dodging the fact that they have released an incomplete product, this issue could not have been missed in testing!!! Only in our industry can you release a product that you say does something, and then nothing happens to you when it doesn't, it makes me very sad, angry, frustrated, annoyed etc

Now I know you test everything, but in case you forgot, or haven't yet tried to restore an individual file, I write this hopefully to save you from thinking your backups are fine, they are not.

Now if you’ll excuse me, I’m off to do research on re-skilling for another industry

[RobertPRAG]

Hi there,

I've just tested a GRT restore using Backup Exec 2010 and vSphere 4 Update 1 and it worked absolutely fine!

I have a Windows XP machine as a VM - I deleted two directories. Both contained Dell drivers. I then set my backup job to restore these two folder from my B2D folder and the job took about 10 seconds to run and the directories appeared on the VM.

I'm not sure what you mean when you say it will work when you backup direct from an ESX host - I'm using the VMware agent connecting to vCenter and backing up two ESX hosts. It works fine for me.

I cannot test the Application agent GRT's as I don't have Exchange within a VM, but the file GRT works fine and that seems to be what you were trying to do according to your original post.

Thanks

Rob

[blue23]

GRT does work with vsphere. In your GRT options, on the backup job if you are not backing up an exchange or sql database on that job you have to disable it, if not you will get that error. You can leave the Active Directory option enabled under the GRT options.

[gbh66]

Hi Rob,

thanks for the feedback. Encouraged by your comments (and the fact the pamphlets on choko farming are yet to arrive ) I decided to go back and do some more testing.

Here is what I discovered.

1) File level restore (GRT) DOES work with vSphere backups. The error that I saw (did I mention I was tired) was actually due to the fact that I had failed to untick the exchange item I had used in the first test.

2) Individual mail items restore fails with the error previously mentioned.

The mail error seems to be fairly low level, as it simply does not restore the volume, or the mail store, to retrieve the data from.

We are backing up on a FC san directly off to tape, as this no longer requires disk storage which is great. But the restores still require a disk based staging area, and I noticed that it wasn't filling up during the restore.

If you perform a file level restore, it writes the backed up VMFS volume to the staging area you specify,then mounts it and restores the selected files.

If you do a restore of an email from a "normal" file based backup using GRT, the Exchange store and log files get written to the staging area, then the item gets restored.

If you do the same with vSphere backup , nothing gets written to the staging area, then it complains that it cant mount the disk,

V-79-57344-38721 - Failed to mount one or more virtual disk images:

and fails.

I can't test the issue against SQL, so sorry to any SQL users, but would love to hear from any SQL or other Exchange users (we are using Exchange 2007) if anyone has had positive experiences.

I am a little embarrased at having thrown this up in such a hurry, but on the strength of Symantec's own advisory, and the no (useful) help I have received from their support department so far, I really felt I had to say something. What concerned me most is that the error only showed up during the restore process, and I was worried that people may have partially tested with for example restoring a virtual machine, and assuming everything else was going to be fine.

Oh in terms of the direct to ESX comment, that was a Symantec comment, apparently you can add each ESX server directly using the root authentication and back machines up that way... I haven't tested it, and honestly don't plan to.

Now if I could only figure out a way to change the title of the post..

thanks for your help.

Graham

[RobertPRAG]

Hi Graham,

I was going to attempt to build a Windows Server 2003 VM with Exchange and test it today with Exchange GRT, but unfortunately other things have cropped up so doubt I will get chance.

However, I noticed on reading the Backup Exec 2010 Administrators Guide that it says this under Exchange GRT within the VMware Agent:

Mailboxes, individual messages, calendar items, tasks, journal entries, and public folder data (disk-backups only)

I'm just wondering if this only works if you use a B2D folder instead of restoring direct from a tape - the bit at the end mentioning disk-backups only made me think this - it could just apply to the public folder data as its a little ambigious.

Just a thought anyway and worth a try. If I do get chance - maybe next week, I will try a Exchange GRT backup and restore.

Thanks

Rob

[scottziehr]

I came across this post doing some research, waiting for Symantec support to call me back...

I am having issues performing granular restore of Exchange 2003. The error in the log led me to this happy kb:

http://seer.entsupport.symantec.com/docs/351373.htm

So I can't really tell you if grt works on Exchange installed on the C: drive but I can tell you it doesn't work with Exchange installed on anything but the c: drive. (Maybe symantec will call me back and fix me right up but I'm afraid they're just going to confirm that I can't do it.)

I've also had some issues with AVVI and SQL and have yet to perform a granular restore successfully. I can't say at this point that this is a Symantec issue or user error. And I had a domain controller lock up in the middle of an AVVI backup last night.

gbh66, have you made any progress with your issues? I can't say grt doesn't work with vsphere, but I will say it doesn't seem to work as advertised.

[RobertPRAG]

I've just dragged up this old thread as I ran Live Update on my Backup Exec 2010 servers and it downloaded a few hot fixes. One of the hotfixes has the following information listed:

http://seer.entsupport.symantec.com/docs/354451.htm

Issues fixed (among many others)

- An individual email/mailbox/database level restore from an Application Granular Recovery Technology (GRT) enabled backup set of an Exchange Virtual Machine created using the Agent for Virtual VMware Infrastructure (AVVI) fails if the Exchange application or its databases is running in a non-default location. http://support.veritas.com/docs/351373

It looks as if they have fixed it. I cannot test as I don't have Exchange running within VMware, but just thought I would let you know.

Thanks

Rob

[scottziehr]

Thanks Rob. I have been working with Symantec Support for about 3 weeks now and still have no resolution. We applied that hot fix last week and it failed to resolve the issue. As it stands, even on a file server, with AD, SQL and Exchange GRT turned off, we cannot restore individual files to anything but the C: drive. Support hasn't been much help. Mostly, they've collected logs and told me they're waiting for Engineering to get back to them. It's been pretty frustrating.

Is anybody else having/not having issues with GRT through AVVI?

[RobertPRAG]

Hi Scott,

I've just tested a file restore to one of my virtualised servers that has a C and D drive. I deleted two files from the D drive (about 1MB in size) and then carried out a restore using Backup Exec 2010. The restore worked perfectly and the two files appeared back on the virtualised server.

A little bit of background - my backups are taken to B2D folders and then duplicated to tape, so all of my restores have been from B2D folders. I have GRT enabled on the backups for file system restores, but I have AD, SQL and Exchange GRT disabled as we don't have any of them virtualised.

I can definately say that file level GRT works from a VM backup with no problems for me here!

Thanks

Rob

[gbh66]

Hi Scott, Rob

Scott, firstly let me apologise for not getting back to you, your post came in whilst I was on leave and somehow got overlooked (may have had something to do with the 7 million unread emails).

I am not experiencing your problem Scott, (the file one, the problem with Symantec support, that I can relate to) we are able to restore individual files to drives other than the C drive, it was one of the tests that I performed on the Exchange server, but we are using exch 2007 on a win 2008 box. It's been so long that I just tested an indivivual file restore on one of our 2003 servers to make sure and it worked ok.

The only thing I have seen that sounds similar was in early tests we could restore the C drive files, but not other drives, this turned out to be my own fault, as I hadn't properly specified the staging area, and it was defaulting to the c drive which ran out of space on the larger drives. You don't pick it up unless your actually watching it, as the job fails, and then the c drive goes empty again. It's in restore settings - Vmware - path to tmporarily restore files, location needs to have more free space than the drive you are restoring from. Probably not your issue.

Rob thanks for the heads up on the fix, that sounds like it should (if it works) do the trick, as we do run exchange databases in a non default location (the default is the c drive), anyone who follows microsofts or vmwares recommendations does. As it was only the exchange server we had issues with, I have rolled back to doing just the exchange server as a normal client based backup using GRT, and both my files and mail items restore Ok. I had tried your suggestion of a disk based backup but the results were identical, files ok, exchange items not.

I'll apply the update, and see if that fixes the problem, and get back to you shortly.

Thanks again for the feedback guys, it's been the only useful info I've had. We have been onto a product specialist", but they have gone awfully quiet, not responding to emails etc, shhhh, if we're very quiet maybe they'll just go away, you gotta love this industry!

regards

Graham

[RobertPRAG]

Hi Graham,

That would be great if you could apply the updates and then let me know if it fixes the Exchange side of things. Purely from an interested stand point!

I've always kept the default Exchange installation on the C drive and then placed mail stores and logs on other drives. I'm putting Exchange 2010 on R2 in VMware soon, so that should be a good test of Backup Exec 2010!

Thanks

Rob

[scottziehr]

Graham,

I'd be interested to know what error you are receiving when you try grt on exchange. Pre-hotfix 354451, my error was "a communications failure has occurred with an exchange store resource". Since applying the update, I now get "unable to attach to a resource". It errors out within seconds of submitting the job. (Looks like the file level restore is working to other drives. Thanks for the refresher on the temp folder Graham!)

Thanks,

Scott

[The_Bublitz]

I've recently migrated to both Vsphere 4.0, and BE 2010. While using AVVI plugin, I am simply trying to get an image level backup, with GRT file level at the same time.

At first I was getting "Failed to mount one or more virtual disk images:"

I found Symantec Document

which seems to tell me I can't use Virtual Center. Anyone find a way around this? I tried backing up directly to the host, and it succeeded. This is of course a horrible workaround, since our VM's are often migrated between hosts. Has Symantec seriously left us without vCenter integration?

Thanks -

IntegIS

[RobertPRAG]

Hi there,

Sorry for the late reply but I've been dealing with a few other things!

I am using Backup Exec 2010 connected to vCenter to do my backups - image level with file GRT and it works absolutely fine! I've seen the Symantec knowledge base article, but I've never experienced that problem.

My advice would be to create a domain account that has basic domain user priviledges (this will be used for your VM backups). Add this account to the local Administrators group on your vCenter server. Also add this account to the vSphere vCenter at the top level and grant it the Administrator role.

Add this account to BackupExec and make sure it is specified for your backups via vCenter - don't test the account as it will error (by design as there are no agents installed within ESX/i to test).

Try your backup - what happens? If everything is okay, then it will be to do with permissions and you can either leave it as above or narrow it down for your own setup.

Thanks

Rob

[gbh66]

Hi Scott, Rob,

sorry things have been a bit hectic. I havent finished testing, but wanted to update you while you still cared

Now to the real question, does the patch fix the problem, yes, yes it does. I have been able to do a successful GRT restore to exchange 2007. The restore behaves as I would expect, it creates folders for the disk lock files, then virtual mount folders appear MP00 under which the drives appear, then a folder containing log files for processing the Exchange database. It even sends the user a nice email that the files have been restored.

HOWEVER, I have also seen the reappearance of the unable to copy disk file as it is corrupted message occuring randomly, you know, as the disks corrupt and uncorrupt themselves, http://seer.entsupport.symantec.com/docs/337860.htm, this was fixed in a very early patch, but seems to have returned... Another patch I suppose will make that go away again,

Was it all worth it, the amount of time saved in the backup doing it via vcenter and FC as opposed to directly over the network, is about 30%. It will make an entire machine restore quicker, but I get the feeling individual restores will be fairly line ball.

Scott original error message, same as yours.

Error : e000ff12 - A communications failure has occurred with an Exchange Store resource.

One curious thing I did note was that in spite of having set a GRT restore folder, as I had backed up to a disk folder, it created all the folders and files there, didnt create a thing in the GRT folder, but presume the GRT setting will have an effect when restoring from tape, still some things to test when I feel like doing someone elses software for them

Graham

[gbh66]

Hey IntegIS,

Yeah its a great little post from Symantect isn't it, sent me clear around the twist when I saw it too... Firstly it is now incorrect, (it probably was at the time of writing), secondly it shows a distinct lack of concern not to keep this stuff updated. (I sent them some hate mail asking them to update)

I agree that connecting straight to the servers would make the process useless the moment you VMotioned anything.

If you apply all the patches, you can run a GRT type backup, so run liveupdate from the tools menu in BackupExec.

You need to make sure that your authorisation to VMware VSphere is correct, as Rob mentioned, particularly if you are not AD integrated, you need to have the server agent on each virtual machine, you need to have a Backup Exec license for any other VSS aware things (Exchange, SQL, AD etc) if you want that part of the process to work, it will do what it is licensed to do, but will throw an error on the machine if it detects anything it is not licensed to do, even though the process completes for the stuff you are licensed to do.

The other thing is to make sure that your GRT folder is set to a spot large enough to cope with your largest machine, but thats more a restore thing.

I still feel it is buggy, but I'm sure by 2011...

[scottziehr]

Glad to hear things are functioning as expected for you all. We are still experiencing issues with AVVI and have our own KB aritcle written from our logs!

http://seer.entsupport.symantec.com/docs/357257.htm

So for now, we're going back to the network-based backups. The speed isn't a huge issue. But I do want an image-level backup to be able to restore the server from, so that's a bit of a pain.

It's been frustrating, Symantec doesn't seem too interested in troubleshooting this issue, they seem content to look at logs and say "Gee, haven't seen this one before, we'll send it up to engineering" and that'll be the last I hear from them until I contact them. Just venting a little, I'm done now. If I make anymore progress, I'll post back here.

Scott