WalkingFridge
Contributor
Contributor

Some VDR errors

Hi guys,

I'm having some problems with VDR.

There I explain my situation:

16 x vm to backup

4 x VDR

8 x 500 GB disks mounted as Network Share

Each VDR backups 4 vm and has 2 disks.

There are vm that we want to backup every day and there are vm that we want to backup 1 time a week.

First of all, there seems a good configuration?

The problem I'm having about continuos errors of Integrity Check and some backup that fails.

The error codes I get are:

1-> 3942 (delete Snapshot failed)

2-> 2241 (destination index invalid / damaged)

3-> a popup window "The integrity check failed because there was a problem with the destination"

Errors 3942 and 2241 looking at the community and other sites, I found that I had to find *.lck files in the VDR (find / -name *.lck) and delete what was found. (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101806...)

Or after stopped the 'datarecovery" service (service datarecovery stop), delete file
/<...>/VMwareDataRecovery/BackupStore.cat

/<...>/VMwareDataRecovery/BackupStopr.cat.bak

and restart the service ((http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101590...)

I've tried to do a manual snapshot of a vm in error and is working properly. If I try to delete the manual snapshot, I get the error "Unable to acces file <unspecified filename> since is locked".

In conclusion, errors spawns again and I have at least 1 error for each VDR, that means 1 vm in error on each VDR.

Right now, from Browse Datastore of a vm in error, inside its folder, I have found some folders called

<VMname>-Snapshot99.vmem.lck

If I try to delete one of these folders I get the message

"Cannot delete file [P2000] <vmname>/<vmname>-Snapshot99.vmem.lck" but in the end the folder is gone....

Please help me...

p.s.: If you need more details, just ask

I'm using VDR 1.2.1

0 Kudos
11 Replies
WalkingFridge
Contributor
Contributor

One thing to add: if i delete the snapshot99 folder I get the error that it can't be deleted but if disappears. But, if I click the refresh button, the folder is still there

0 Kudos
durakovicduro83
Enthusiast
Enthusiast

Have you try with Intergity check ?

Cheers,

Denis

To err is human - and to blame it on a computer is even more so
0 Kudos
WalkingFridge
Contributor
Contributor

The detailed situation (integrity check included) is this:

VDR_1:

   Disk_1: Integrity check OK

   Disk_2: Integrity check OK

   Backup VM_1: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_2: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_3: OK

   Backup VM_4: OK

VDR_2:

   Disk_3: Integrity check OK

   Disk_4: Integrity check OK

   Backup VM_5: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_6: OK

   Backup VM_7: OK

   Backup VM_8: OK

VDR_3:

   Disk_5: Recatalog OK, Integrity check can't start (error window: "The integrity check failed because there was a problem with the Destination")

   Disk_6: Integrity check OK

   Backup VM_9: can't start because I've set Disk_5 as destination...and it's unavailable

   Backup VM_10: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_11: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_12: can't start because I've set Disk_5 as destination...and it's unavailable

VDR_4:

   Disk_7: Integrity check OK

   Disk_8: Trouble reading from destination volume, error -2241 (Destination index invalid/damaged)

   Backup VM_13: OK

   Backup VM_14: Trouble reading files, error -3942 (delete snapshot failed)

   Backup VM_15: Can't access Backup Set <Disk_8>, error -2261 (can't use Backup Set until integrity check secceeds)

   Backup VM_16: Can't access Backup Set <Disk_8>, error -2261 (can't use Backup Set until integrity check secceeds)

Wow....now that I see the situation written here....it looks really(!) seriuos.
I've tryed to format a disk (Disk_8) to restart backups from a cleaned situation....but I can't because all these disks are network volume (warning windows: The format operation is not required for a Network Volume)

0 Kudos
phykell
Enthusiast
Enthusiast

The first thing I'd say is that for 16 VMs, four VDR appliances seems a lot where each appliance can have two destinations and more than one backup job. What's the size of the VMs and is there any chance that the VDRs are causing some contention issues which may be causing the snapshot problems?

Not that I think you're doing so but it's worth stating - I don't think it's a good idea for VDR appliances to share destinations for obvious reasons.

I'd say a network share (CIFS) is the least reliable of the possible types of VDR destination.

Do you know if it's possible to publish NFS volumes instead of network shares (CIFS) on your NAS (my assumption)? Another advantage of an NFS volume would be that you could have a 1TB destination instead of sticking to the network share limit of 500GB.

If that was my infrastructure, and assuming I could publish NFS volumes, I'd probably have two VDRs, each with one 1TB destination. The VMs could be spread over the two VDRs and one of them could even have two backup jobs, one of which backed up certain VMs weekly. Depending on my needs, I may just settle for the one VDR with one or two destinations and two backup jobs, a daily one and a weekly one.

Regarding a fix for your issue, I'd suggest looking at swapping to NFS if at all possible and re-creating your destinations unfortunately.

0 Kudos
WalkingFridge
Contributor
Contributor

I'm back Smiley Happy

Yesterday I've called the vmware support and in the vm of the VDR (in VMs and Template, not in the VDR plugin), clicking Edit Settings we have found that the vm has hot added a disk for every failed backup of the VMs. I've removed disks (the support guy tell me to remove only, not remove and delete). Then I've started a manual snapshot of a VM and then I've deleted all snapshots.
Snapshot removed successfully Smiley Wink
Next phase was to start a backup with VDR. Select a job, backup now, all sources....waiting....crossing fingers....DONE
But I remember that I've some disks in error (destination index invalid/damaged)
This is the detailed situation now:
VDR_1:
   Disk_1: Integrity check OK
   Disk_2: Integrity check OK
   Backup VM_1: OK
   Backup VM_2: OK
   Backup VM_3: OK
   Backup VM_4: OK
VDR_2:
   Disk_3: Integrity check OK
   Disk_4: Integrity check OK
   Backup VM_5: OK
   Backup VM_6: OK
   Backup VM_7: OK
   Backup VM_8: OK
VDR_3:
   Disk_5: Integrity check failed, error window: This destination is not available for a backup job. Please ensure that the destination has been formatted and has suffient free space
   Disk_6: Integrity check OK
   Backup VM_9: error -2246 (wrong destination index found)
   Backup VM_10: OK
   Backup VM_11: OK
   Backup VM_12: error -2246 (wrong destination index found)
VDR_4:
   Disk_7: Integrity check OK
   Disk_8: Trouble reading from destination volume, error -2241 (Destination index invalid/damaged)
   Backup VM_13: OK
   Backup VM_14: OK
   Backup VM_15: Can't access Backup Set <Disk_8>, error -2261 (can't use Backup Set until integrity check secceeds)
   Backup VM_16: Can't access Backup Set <Disk_8>, error -2261 (can't use Backup Set until integrity check secceeds)

OK, now it's better then before....but, obviously, my client wants every VMs backupped!
Any suggestion?

0 Kudos
phykell
Enthusiast
Enthusiast

Well it's certainly looking more positive! Smiley Happy

Can you confirm whether NFS is a possiblity? I know I've said it before, but if at all possible, replace your network shares with NFS volumes. I just don't think you'll get those faulting destinations to work properly again and even if you did, would you really trust that a restore of a VM would be fine?

0 Kudos
WalkingFridge
Contributor
Contributor

Now I'm going to show you the infrastructure, bacause I have note made it.

Everithing start from a host (HP DL370 G6) with vmware ESXi 4.

Two vm on this host:

1 vm (ubuntu) as syslog

1 vm OpenFiler 2.3

This host has a datastore (HP D2600) connected via SAS.

If I log into the web administration GUI of OpenFiler -> Quota Tab -> Select Volume    I can choose between:

nfs (group: backup_6T / volume: nsf)

DataRecoveryTarget _1 (group: sixtera / volume: 500gb_1)

DataRecoveryTarget _2 (group: sixtera / volume: 500gb_2)

...

DataRecoveryTarget _3 (group: sixtera / volume: 500gb_3)

so, from here, I would say that are NFS volumes.

If I log into a VDR and look at /etc/mtab, I can see two CIFS volumes.

For exemple:

//<ipaddress/sixtera.500gb_1 /<ipaddress>/sixtera.500gb_1 cifs rw,mand 0 0

Therefore, I'm not understanding if they are nsf or cifs. But, since the VDR says cifs, i would say that they are all cifs volumes.

You say that I can use these backup disks as NFS? How can I do?

0 Kudos
phykell
Enthusiast
Enthusiast

I've not used OpenFiler before but I've just a quick Google and there seems to be quite a few decent articles on how to deploy an NFS Openfiler datastore for use with vSphere. As an example though, I'm using a fairly old HP NAS running Windows Storage Server 2003. All I had to do was to enable NFS and specify a UNIX user name mapping to copies of the passwd and group files (taken from the ESXi host). That way, when ESXi connects to my datastore, it does so as a UNIX user which is then mapped to a Windows account which has access rights to the various NFS volumes.

Incidentally there are some great reasons to go with NFS other than just as a VDR backup destination - you can use it to run VMs, vMotion works fine as does Storage vMotion. I use it to store VM templates as well as the usual ISOs, etc. 

Assuming you can get Openfiler to publish an NFS share and you can sort the user name mapping, it's then just a matter of adding the NFS volumes as datastores on your ESXi host. You can then create a virtual disk on the NFS volume with a 1 TB size (double the CIFS limit) and then you can add it to your VDR appliance. On seeing it appear in the destinations, you can select it and format it as a VDR destination. Apparently, you can also access the NFS share directly by adding it to the file systems table "/etc/fstab" in the appliance - I'm not sure if that's a supported configuration though! 

To see if your destinations are NFS volumes, just look at the list of destinations on your VDR appliance - if it's "Network Share" (or something like that) it's CIFS; for example, my NFS volume shows as "Local Volume".

I might get a moment to try Openfiler this weekend so I'll let you know if I find anything useful Smiley Happy

slaclair
VMware Employee
VMware Employee

phykell wrote:

I've not used OpenFiler before but I've just a quick Google and there seems to be quite a few decent articles on how to deploy an NFS Openfiler datastore for use with vSphere. As an example though, I'm using a fairly old HP NAS running Windows Storage Server 2003. All I had to do was to enable NFS and specify a UNIX user name mapping to copies of the passwd and group files (taken from the ESXi host). That way, when ESXi connects to my datastore, it does so as a UNIX user which is then mapped to a Windows account which has access rights to the various NFS volumes.

Incidentally there are some great reasons to go with NFS other than just as a VDR backup destination - you can use it to run VMs, vMotion works fine as does Storage vMotion. I use it to store VM templates as well as the usual ISOs, etc. 

Assuming you can get Openfiler to publish an NFS share and you can sort the user name mapping, it's then just a matter of adding the NFS volumes as datastores on your ESXi host. You can then create a virtual disk on the NFS volume with a 1 TB size (double the CIFS limit) and then you can add it to your VDR appliance. On seeing it appear in the destinations, you can select it and format it as a VDR destination. Apparently, you can also access the NFS share directly by adding it to the file systems table "/etc/fstab" in the appliance - I'm not sure if that's a supported configuration though! 

To see if your destinations are NFS volumes, just look at the list of destinations on your VDR appliance - if it's "Network Share" (or something like that) it's CIFS; for example, my NFS volume shows as "Local Volume".

I might get a moment to try Openfiler this weekend so I'll let you know if I find anything useful Smiley Happy

Make note NFS is only supported as a duplication store when it is a traditional NFS datastore in ESX and the VMDK is attached to the vDR appliance.

VCAP5-DCD/DCA/CIA, VCA4-DT
WalkingFridge
Contributor
Contributor

Yes... are network share...

VDR appliance -> configuration -> Destination -> Type: Network share

0 Kudos
phykell
Enthusiast
Enthusiast

As suggested, I'd say you should move to NFS if at all possible (OpenFiler *does* support NFS). In the meantime, it's worth at least restarting the appliances which you still have issues with. When the VDR restarts the re-catalog job should run which will hopefully synchronise your restore points with the contents of the deduplication store; on completion of that task, the integrity check should run. You'll probably need to stop the integrity check, delete any damaged restore points and manually start the integrity check again. At this point though, it's worth confirming the procedure with VMware support if you have a case open with them.

0 Kudos