VMware Cloud Community
sgadsby
Contributor
Contributor
Jump to solution

VCB 1.1 + BackupExec 12: Directory X was not found, or could not be accessed

Hi,

I have VCB configured and can backup both File and FullVM successfully with BackupExec 12, however I am having issues with 90GB File backup. I have disabled Verify and am getting weird errors in the BackupExec log and failed jobs.

2 Questions:

1) I get file corrupt warnings for some database files. Since the volume has been snapshotted, how can the files still be open? I get "X is a corrupt file. This file cannot verify." I thought this happened because a file was updated between backup and verify, but surely it doesn't change on a snapshotted volume?

2) I get "Directory X was not found, or could not be accessed. None of the files or subdirectories contained within will be backed up." If I mount the server manually with either pre-backup.bat or browse-start.bat I can successfully access those directory and sub-file; there does not seem to be an access problem. This happens after about 30mins of successful backing up. Does anyone know if a VCB mount has a time limit?

Are NTFS permissions applied to the VCB mount? Does my backup user need to have NTFS permissions to files on the volume that is mounted? Or does the SYSTEM user need permissions (since SYSTEM mounts the volume)?

Thx,

Simon.

--

I need to read "Xen and the art of VMware sales" Smiley Happy

-- I need to read "Xen and the art of VMware sales" 🙂
Tags (4)
Reply
0 Kudos
1 Solution

Accepted Solutions
dinny
Expert
Expert
Jump to solution

Sorry,

My last post was a little incorrect.

For file level backups the snapshot is kept mounted - (as opposed to being copied locally first, as in a full vm backup, and then backed up to tape)

So from your logs it most likely is an issue with VCB itself - most likely an intermittent issue with access to the SAN disks via VCB?

I have intermittently had similar errors with both the vmount2 service and with blklist errors.

There are a couple of posts (if you search the forums) about increasing the blocklist timeout (or something similar)

I did once ask vmware support about doing that - but didn't really get a definitive answer...

Personally if I was you, I'd raise a call with Support.

I would also try to make sure you never run vm full, and vm file backups at the same time, on the same VCB server - as apparently that is not supported...

I also try to only ever have one vcb backup job executing at a time - as more than one seems to increase the likelihood of scsi reservation or blklist errors - again these always seem to be far more prevalent with file level backups, and again the larger the VM the more likely problems seem to be.

Unfortunately my personal opinion is that VMware still need to do quite a bit more work on the file level backup side of VCB...

Dinny

View solution in original post

Reply
0 Kudos
9 Replies
dinny
Expert
Expert
Jump to solution

Hiya,

I presume you are running vmware tools on the (windows) VM in question?

If so are you using the pre-freeze-script.bat and post-thaw-script.bat scripts to stop any running DBs prior to the snapshot being taken - and to re-start them after the snapshot is taken?

If not - then the vmware tools sync driver could well be causing problems to any running DBs.

Definitely best to use the pre-freeze and post-thaw scripts to stop and DBs prior to VCB snapshots.

If you don't your DB backups may well not be consistent - and the actual DB on the VM could be adversely affected.

Dinny

Reply
0 Kudos
sgadsby
Contributor
Contributor
Jump to solution

Yes, thanks Dinny, VM tools is running but I have not configured the pre/post-thaw scripts. I'm not actually too concerned with the DB consistency of the backup at this stage, it's more the backup access failures that I am concerned about.

I actually got one successful backup (140GB), but the rest have failed at closer to 100GB. So it seems the problem is certainly not to do with real file access...

I am backing up to disk, which is doing about 2500MB/min. I am wondering whether the VCB driver might have some performance limitations, and that bottleneck might be causing the issues? I don't know how I could slow the backup down to test...

Also my VCB mount point is D:\VCB, and I am backing up to a separate directory on the same volume, D:\BACKUPS. I wonder whether the result might be different if the OS thinks the source is on a different volume? Maybe? Long shot but I might try it.

Is anyone else doing VCB file backups to disk?

--

I need to read "Xen and the art of VMware sales" Smiley Happy

-- I need to read "Xen and the art of VMware sales" 🙂
Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Hiya,

May be worth re-doing the test with the DB stopped - or using the pre-freeze scripts - just to rule out the DB(s) in your first question.

I take it there are no issues with full vm backups of the same VM?

(I've always found file level backups to be far flakier than full VM backups...)

As I understand it the file level backup is in two stages:

a) mounting the snapshot - then b) taking the backup.

So once VCB has mounted it - it is then down to your backupexec software to back it up - so presuming that your failures are during the backup stage - it should not really be related to the VCB driver anyway?

Unless perhaps the snapshot is suddenly unmounted (or otherwise corrupted in some way)?

Again if one backup worked - that would also seem to rule out file permission issues too...

Any clues in the pre or post logs in c:\windows\temp?

Dinny

Reply
0 Kudos
sgadsby
Contributor
Contributor
Jump to solution

This is a file server, I do not want to do FullVM backups of it, so I haven't tested that. I did a FullVM of another machine and it seemed to work fine.

Immediately before Backup Exec fails, the following two log files under C:\WINDOWS\Temp report errors:

  • vmware-vlun-2.log

  • vmware-vmount.log

Sanitised excerpts of each are attached. In these files you can see that the mount completed successfully at 23:17:55, however there were a whole stack of "BlockList" errors from 23:54:58, and then VmountDisk reported Error: 2. The BE job failed at 23:56:06.

I hope this means something to someone... I am running x64 Windows in case that is relevant.

Simon.

--

I need to read "Xen and the art of VMware sales" Smiley Happy

-- I need to read "Xen and the art of VMware sales" 🙂
Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Sorry,

My last post was a little incorrect.

For file level backups the snapshot is kept mounted - (as opposed to being copied locally first, as in a full vm backup, and then backed up to tape)

So from your logs it most likely is an issue with VCB itself - most likely an intermittent issue with access to the SAN disks via VCB?

I have intermittently had similar errors with both the vmount2 service and with blklist errors.

There are a couple of posts (if you search the forums) about increasing the blocklist timeout (or something similar)

I did once ask vmware support about doing that - but didn't really get a definitive answer...

Personally if I was you, I'd raise a call with Support.

I would also try to make sure you never run vm full, and vm file backups at the same time, on the same VCB server - as apparently that is not supported...

I also try to only ever have one vcb backup job executing at a time - as more than one seems to increase the likelihood of scsi reservation or blklist errors - again these always seem to be far more prevalent with file level backups, and again the larger the VM the more likely problems seem to be.

Unfortunately my personal opinion is that VMware still need to do quite a bit more work on the file level backup side of VCB...

Dinny

Reply
0 Kudos
sgadsby
Contributor
Contributor
Jump to solution

Well I came across this post about increasing the blocklist timeout:

After restarting hostd however esx core dumped and I lost a whole bunch of machines Smiley Sad I should have been abit more careful!

So I commented out the path node leaving just the leasetimeoutsecs, and after that everything started ok and I got 5 successful backups in a row.

I'll keep an eye on it but it looks like the default leasetimeout (whatever it is) is not adequate for the hardware I am running (IBM).

Thanks for all your input Dinny. I think you must be right re VMware needing to polish this product a bit more.

--

I need to read "Xen and the art of VMware sales" Smiley Happy

-- I need to read "Xen and the art of VMware sales" 🙂
Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

Quick question. Are you using multipath software on your vcb proxy server? It is supposed to be supported, but. . . . Try disabling paths other than one per LUN, and see if you get better results.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
dinny
Expert
Expert
Jump to solution

Pleasure,

Glad you got them to work in the end.

(By the way - please remember to award points by marking answers as correct or helpful)

Dinny

sgadsby
Contributor
Contributor
Jump to solution

Thanks KjB, only one FC path available to VCB server so I doubt multipathing issue in this case.

God bless, Simon.

--

I need to read "Xen and the art of VMware sales" Smiley Happy

-- I need to read "Xen and the art of VMware sales" 🙂
Reply
0 Kudos