VMware Cloud Community
fgl
Enthusiast
Enthusiast

VDR error -105

Hello,

Has anyone else encounter or know how to resolve a error -105? The exact error message is "Trouble writing to destination volume, error -105 (unexpected end of data)". This is only occuring on 1 of 25 VMs, all of the other 24 VMs are backing up just fine and the one that is giving the error has been backing up fine for the last 2 months on a daily basis. I've tried to do a manual backup of the VM and it goes through and create the snapshot and actually seems to be doing a backup until it gets to about 40% before it errors out. I can manually create a snapshot outside of VDR just fine and the system is also up and running just fine. The storage destination for the VDR appliance has plenty of space (using 170GB of 500GB) and like I said all of the other 24 VMs are backing up fine. Any suggestion would be appreciated.

Thanks in advance.

0 Kudos
16 Replies
USER27895
Contributor
Contributor

I have the exact same issue, getting the same error for one VM ( for the most important:( ) although I have enough space at the destination and the snapshot of that VM is created just fine.

Yet another error, although I am using 1.1 version. It feels like I payed for the VDR license in vain it is still in alpha state...

0 Kudos
fgl
Enthusiast
Enthusiast

Update: The only way for me to resolve the problem VM was to completely delete all VDR backups for that VM by marking it for deletion and then running the integrity check again which took about 9 hours and then manually started a new backup for the VM. This of course means all of my previous backups for that VM are now all gone and god forbid if I need to now restore from anything earlier than yesterday. It seems to me that there was some kind of data corruption with the backup data for just that VM but a integrity check did not reveal any errors.

0 Kudos
admin
Immortal
Immortal

Can you share information about the VM?

a. HW version (4 or 7)

b. Guest OS? If Windows, which one and if VMware Tools is updated?

c. How many virtual disks on this and how big are they in GB? Are the virtual disk size a nice round integer (25 GB as opposed to 25.00247645GB)?

d. Anything special about the VMFS datastore that the virtual disks reside on (i.e do other VMs that backup OK share the same datstore)

0 Kudos
fgl
Enthusiast
Enthusiast

a. HW version (4 or 7) - On hardware version 4. Reason it's not on version 7 is because we haven't found a window to bring the server down.

b. Guest OS? If Windows, which one and if VMware Tools is updated? - Windows 2003 std 32-bit, vmtools is the most current under vSphere 4 update 1.

c. How many virtual disks on this and how big are they in GB? Are the virtual disk size a nice round integer (25 GB as opposed to 25.00247645GB)? - 1 virtual disk of size 20GB.

d. Anything special about the VMFS datastore that the virtual disks reside on (i.e do other VMs that backup OK share the same datstore) - Nothing special just a standard installation of 2003 server with 1 medium usage application.

0 Kudos
bemobile
Contributor
Contributor

I have the same issue. In fact, it's not the only issue i've had. I had to erase all my backups a few days back as well.

The specs:

- All vm's were created using vsphere client and are version 7

- We are talking 2 ESX hosts, SAN and a seperate backup server (that also hosts the vcenter server/vsphere client)

- we're running vsphere with the advanced license.

I've been using the product for about 10 days now, after I had read the entire administrators manual.

What I did was setup a single cifs share on our backup server and configured vmdr to use that. Also note that at first it failed to create the share because my password was too long? (i'm thinking the password must have been about 20 chars). in any case, it should prompt a message that it failed for that reason.

Then i created a single backup job, that included 7 VM's. Some are Centos 5.4, others are Windows Server 2003 (both 32 and 64 bit enterprise/Standard versions).

Most VM's have a single 15 GB C:\ drive, and optionally a additional data drive of usually 15/20 GB (always round numbers). I also have a database server running with a 200 GB data drive and a 20 GB c:\ drive.

Problem 1:

Integrity check : 8 execution errors.

Destination "/.../vmdr/" will be locked until the restore point with errors are deleted and integrity check succeeds.

I then marked the failed restore points for delete and ran integrity check again, without success, as it seemed to hang. (I've let it run overnight, but it never got passed 70% or so if i remember correctly. So i logged in and stopped the datarecovery service (which didnt respond). Then I killed it, deleted all the backups and went on with my new way of storing data.

I created, 7 cifs shares on the backup machine (for each VM one share), but all on the same server. This would allow for smaller indexes and problem seperation if it occurred. I also have 7 different backup schedules.

All went fine for 2 days. 2 restore points were created, but then last night I'm getting the same error as the OP. On one virtual machine. We'll name the VM 'tic'.

Tic has currently 2 restore points, while all others have 3. In the first restore point, there is only a single disk c:\ with 15 GB. I then added an extra data disk of 20 GB with thin provisioning on a different lun. This seemed to work fine in the second restore point. it shows the added disk of 20 GB. I thought all fine.

However, last night (where it would have created the 3rd restore point), it failed with the following error:

- Tic : Task incomplete

Job tic incomplete

trouble writing to destination volume, error -105 (unexpected end of data)

Nothing new was written to the newly added volume, perhaps its looking for changes, but not finding anything. The destination is still mounted properly and can be viewed from the date recovery console.

Any news on 1.2 releasing anytime soon? :).

0 Kudos
emachabert
Contributor
Contributor

Hi everyone,

I have the same issue.

I am backuping many VMs without any issue but one is failling since July 01st.

I am running vsphere 4.0 update 01, VDR 1.1. The VM is running Centos 5.4 x64 (it's a Zimbra mail server), Virtual HW v7, 2 disk (0S: 27GB, DATA: 750GB) both thin provisioned. All is running on two HP SANs (SAS disks, FC backend).

From February to June 30 all was working great. July 1st the daily backup failed with error -107 (out of application memory).

I restarted the VDR with 4GB of RAM and it began showing error -105. I found this thread and tried those things:

- 1 : delete all snapshots, run integrity check and then backup. It worked the first time (the backup took 5 hours). the day after error 105 was back again.

- 2 : unmounted the destination, formated, remounted. It worked the first time (the backup took 5 hours). the day after error 105 was back again.

What is wrong ? Can anyone from VMware tell us what is wrong ? How can I come back to normal situation ? I suppose the problem is on the VM side since others are backuping fine everyday.

Upgrading is not an option for the moment, the farm is running dozens of critical systems.

Here is the log:

17/07/2010 14:54:10: Normal backup using ZIMBRA DAILY

17/07/2010 14:54:15: Copying LNX-ZIMBRA

17/07/2010 14:54:32: Performing incremental back up of disk "[HEB-SAS-02-V02]LNX-ZIMBRA/LNX-ZIMBRA-flat.vmdk" using "SCSI Hot-Add"

17/07/2010 14:54:41: Performing incremental back up of disk "[HEB-SAS-02-V02]LNX-ZIMBRA/LNX-ZIMBRA_1-flat.vmdk" using "SCSI Hot-Add"

17/07/2010 14:54:41: Trouble writing to destination volume, error -105 ( unexpected end of data)

17/07/2010 14:55:06: Task incomplete

17/07/2010 14:55:06: Remaining: 3 files, 750,1 GB

17/07/2010 14:55:06: Completed: 6 files, 27,1 GB

$[//]7/17/2010 2:55:06 PM: Performance: 48790.6 MB/minute$[//37/%D %.1T: Performance: %.1~F MB/minute]$[/D/30/2010-07-17T14:55:06.000Z02:00/]$[/T/30/2010-07-17T14:55:06.000Z02:00/]$[/F/7/48790.6/]

17/07/2010 14:55:06: Duration: 00:00:56 (00:00:21 idle/loading/preparing)

Many thanks !!

0 Kudos
fgl
Enthusiast
Enthusiast

emachabert,

I don't have an answer for you but I can provide you some insight of what I've seen and experienced.

1st I don't use thin provisioning on any of my VMs as I have had problems with it in the past so I only do thick now and my VMs are much happier not just with VDR but overall.

2nd deleting all snapshots had worked for me under VDR 1.1 so I can't explain why it's not working for you.

3rd I know you said upgrading is not an option for you but ever since I upgraded to vSphere 4.0 update 2 and VDR 1.2 my VDR backups have been running for 6 weeks straight now without any problems so far (knock on wood) but only time will tell.

I think pre VDR 1.2 versions has problems with thin provisioned disk and a whole of other problems but VDR 1.2 is a lot better at least in my installation. I'm not saying VDR 1.2 is perfect but it's a major improvement over previous versions.

Good luck.

0 Kudos
emachabert
Contributor
Contributor

Many thanks for the update.

I never had any problem with thinprovising whith hundreds of machines, I don"t think the problem comes from thinprovisionning.

I think I have to plan an upgrade on that cluster, with vmotion it will be non disruptive.

thanks.

0 Kudos
o_casas
Contributor
Contributor

I know this is an old thread but I have same problem with a more recent environment: VDR 2.0 and vSphere 4.1 update 1.

About 10 VMs are being backed up right but one (Linux Centos 5). It always gives same error on the log:

Trouble writing to destination, error -105 ( unexpected end of data) 

This is what I have tried:

  • Backup with VM shut down
  • Created new share as Destination Store
  • Move VM to another ESXi host
  • Reboot ESXi host
  • File system check of guest Linux (only one disk)
  • Backup of a VM clone.
  • Remove any inconsistent restore points
  • Integrity checks of Destination Store

None of the above has given any positive result or change of behaviour, always error -105.

I think I am only left facing a .vmdk corrupted file.

Anybody know how can I check it? Or if there is another source of the problem? Troubleshooting hints?

Thank you, guys.

Oriol.

0 Kudos
BDETELLIN
Contributor
Contributor

I have the same problem with one VM in vSphere 5 and DR 2.0.

Any idea ?

0 Kudos
o_casas
Contributor
Contributor

I ended up submitting a Support Request to VMware Tech Support. They asked me for VDR and host logs, and a OVA template of the VM.

No answer yet. I have a Basic Support contract and guess this goes slooooowly.

However, if I move the vmdk disk of the VM from shared storage to a host local disk, then no problem with VDR backup.

It is not a good solution though,  because it impedes vMotion of the VM.

I'll post the final solution if I ever get one.

Cheers!

0 Kudos
BDETELLIN
Contributor
Contributor

Thanks for your answer.

I will contact Vmware support too.

0 Kudos
cpa01
Contributor
Contributor

Hello,

Today we solved 2 errors -105 by removing and recreating swap partitions inside the VMs (Ubuntu 10.04 and CentOS 5.7)  - and perhaps by adjusting the first partition on cylinders boundaries with GParted for Ubuntu.

Hope this helps.

0 Kudos
o_casas
Contributor
Contributor

Hi, all.

Finally solved the problem thanks to VMware technical support that pointed to KB article (2013450):

http://kb.vmware.com/kb/2013450

VMware Data Recovery (VDR) does not back up swap files normally. To work around this issue, configure VMware Data Recovery to back up the swap file by completing these steps:

Open a console or SSH to the VDR Appliance.
Log in using your credentials. The default credentials are:
Username: root
Password: vmw@re
Stop the data recovery service with this command:
# service datarecovery stop
Open the /var/vmware/datarecovery/datarecovery.ini file in a plain text editor and ensure it contains these values:
[Options]
SetVCBLogging=6
SetVolumesLogging=6
SetRAPILogging=6
BackupUnusedData=1
Note: You can create the file if it does not exist. For further/related information, see Editing files on an ESX host using vi or nano (1020302)
Start the data recovery service with this command:
# service datarecovery start
0 Kudos
cpa01
Contributor
Contributor

Hi,

VMware pointed us to that KB too but I thought it'd be a bit dumb to waste space to back up swap files or partitions. Also the KB states that this error should happen when there is a dedicated VMDK for swap, which was not the case for us - and we have many similar VMs without a problem. Indeed removing and recreating the swap partitions was the simplest way to solve this for us.

0 Kudos
LatinSuD
Enthusiast
Enthusiast

Recreating swap worked for me too. In particular:

VM is a Debian GNU/Linux 6.0.

Swap used to live in sda5, an extended partition. So I unmounted swap, with fdisk i removed the swap and extended partition, added a swap partition, rebooted, mkswap, and adjusted fstab with the new UUID and rebooted again.

Swap is now in primary partition sda2 and VDR works fine.

0 Kudos