VMware Cloud Community
jarsenea
Contributor
Contributor

Potential VDR+CIFS Workaround

Hey everyone,

Like many, I too have been experiencing some rather unpleasantries while using VDR (as some of you may have read in my posts on this board). However, I recently discovered a workaround while using CIFS shares that has gotten my backups (and integrity checks) to not only work, but work consistently and quickly for the past 3 weeks. While I am happy that my backups have been working, I'm hoping I don't jinx myself by sharing this workaround that has worked in my environment.

A little backstory on this... After having redeployed the VDR appliance for the Nth time, I was issue my first test backup on a new repository while logged into a root session when, almost serendipidously I discovered that the appliance was outputting various CIFS errors when executing any kind of job on any of my test CIFS shares. After a little research, I discovered that you can essentially throttle the SMB protocol by disabling Opportunitistic locking. Essentially, opportunistic locking (for those who don't know) caches the files on the client side in order to improve performance. I figure it was worth a shot to disable (which normally results in lower performance) the OpLocks on the linux filesystem and see what the end result was. After all, I haven't had any success for more than 3-4 days using VDR up to this point (since May might I add), so really, there was nothing to lose.

To my astonishment, not only did the errors stop appearing, but the backups and integrity checks were occuring on schedule and without issue. To this day (3 weeks later), I've ramped up the VMs that get backed up in my nightly job and have consistently gone through the backup process.

So how did I disable OpLocks?

You will need to login to the VDR appliance as root and issue the following command:

echo 0 > /proc/fs/cifs/OpLockEnabled

This will disable Opportunistic Locking (until the system is rebooted). You can add this to an init script if it works for you and you want to make sure it gets disabled every time you reboot the appliance. Issuing a :

cat /proc/fs/cifs/OpLockEnabled

will confirm whether or not it is enabled or disabled (0 = disabled, 1 = enabled). Remember that when you reboot the appliance, it will default to 1.

As I've said, YMMV, but it has worked for me and I'm not up to 15 VMs being backed up every night without issue. I'm keeping my fingers crossed and this by no means excuses VMware for the current state of VDR, but I'm just happy I've getting some backups done. I'd be interested to see if I'm just having dumb luck or this could actually help some people.

PS: When I did this I started with a fresh repo. I can't speak to the effect of using an existing repository.

0 Kudos
8 Replies
JRoch
Contributor
Contributor

That seems to have fixed the problems. Backup speed is great even on intial back, no lockups for several days now. Thanks for discovering what seems to be the solution.

0 Kudos
mrchrisvm
Contributor
Contributor

Awesome digging my friend.

This helped improve the stability of my VDR deployment.

Thanks!

0 Kudos
mhanbury
Contributor
Contributor

Thanks for this. I've implemented the change and so far so good one week of successful backups with VDR and CIFS shares! With the ops lock on mainly backup failures and file access errors 3902 using VDR 1.0.2.

0 Kudos
mbraak
Contributor
Contributor

Tried it also bu after three weeks i got the same problems.

Integrity check fails/hangs. Smiley Sad

0 Kudos
AndreTheGiant
Immortal
Immortal

Interesting workaround... thanks!

PS: another way to save the configuration is add to /etc/sysctl.conf the line:

fs.cifs.OplockEnabled = 0

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos
riaanb
Contributor
Contributor

Sigh me too (CIFS lockups) I have yet to try this workaround.

I see you mention that Opportunistic Locking is done to improve performance on the client - should one not also disable this on the VM's being backed up?

How is the VDR the client in this case? Oh, might just have answered my own question - it is the client of the server that hosts the CIFS share - is that right?

--

riaan

0 Kudos
admin
Immortal
Immortal

JRArseneau - thanks for the tip. In VDR 1.1 (just release tonight), we went ahead and disable OpLocks by default.

0 Kudos
Bers
Contributor
Contributor

In vdr 1.1 i'm experience a problem with connection CIFS shares with domain credential auth..

With local credentials all - OK.

0 Kudos