montelukast
Contributor
Contributor

vSphere Data Protection stucks at 92% on Windows 2008 R2 VMs

I'm having issues with backing up Windows 2008 R2 VM using data protection... it stucks at 92 % and lasts like forever, unless I cancel the job. Does anyone have solution to this already? I've seen a lot from the community...

0 Kudos
14 Replies
snekkalapudi
VMware Employee
VMware Employee

If your backup is still running, try this..

1.Get the backup job ID of the running job (If you have VI client connected, See recent tasks at the bottom and the one under details column of the VDP backup job is the Job ID associated with the backup)

2.SSH to the VDP appliance and run the command tail -f /usr/local/avamarclient/var-proxy*/<Backup JOB ID>*vmimage?.log

By this atleast you will know if you backup is still running..if you see something like this..

==> /usr/local/avamarclient/var-proxy-1/Sanity-Feb-04-Sanity-Feb-04-1360209600062-a6e80a4cd7b59d767a97a2626da0a950503f215f-3016-vmimagew.log <==

2013-02-07 00:19:24 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : numIOs = 1150000 numMergedIOs = 0 numSplitIOs = 0
2013-02-07 00:25:48 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : numIOs = 1200000 numMergedIOs = 0 numSplitIOs = 0
2013-02-07 00:32:15 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : numIOs = 1250000 numMergedIOs = 0 numSplitIOs = 0
2013-02-07 00:38:42 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : numIOs = 1300000 numMergedIOs = 0 numSplitIOs = 0

If the backup is completed, you should see something like this..

==> /usr/local/avamarclient/var-proxy-8/Sanity-Feb-04-Sanity-Feb-04-1360209600062-ee4eb28536c3f37cb0b5bde293816852cd0fb559-3016-vmimagew.log <==

2013-02-06 20:29:12 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLoadVMCb

2013-02-06 20:29:12 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLogout

2013-02-06 20:29:12 avvcbimage Info <0000>: VixDiskLib vMotion reservation successfully released

--------------------------------------------------------------------------------------------------------

----- END avvcbimage log 2013-02-06 20:29:15 PST  (1 warning, 0 errors, 0 fatal errors)

--------------------------------------------------------------------------------------------------------

-Suresh
montelukast
Contributor
Contributor

Hi Suresh,

Thanks for a great suggestion, I'll surely try this... however... in case I found out that it is still running... odd thing is we left it overnight and its still running until morning... and its a newly installed OS... os usage is less than 20GB... we did a successful backup on Win7 and Win2003 in less than 20 minutes, only on Windows 2008 R2 that we're having issues, we even encountered error at first, which was resolved by editing the disk.UUID as per vmware KB and now the only problem is that it halts at 92%. Datastore for the VDP has 3TB available.

0 Kudos
digitalj
Contributor
Contributor

My environment was almost all Windows 2008R2.  For some reason we had to increase the memory available to VDP in order for it to complete.  At 6GB it completed 23 VMs in 14 hrs, at 8GB  it completes 33 VMs at 4 hrs.

montelukast
Contributor
Contributor

Hi Guys,

Here's the error on the VDP appliance while stuck at 92%:

Error.png

Hope this helps for our guys to look into...

0 Kudos
pizzingrilli
Contributor
Contributor

Hello

Same problem here. Any updates on this?

Regards

Sandro

0 Kudos
GSparks
Enthusiast
Enthusiast

From digging in to the error you supplied, it looks like this might be a known issue with the VMware VDDK 5.1.  In the release notes for VDDK 5.1, they have the following entry:

Whenever vCenter Server does not provide an SSL thumbprint for the target host, the VDDK library might core dump in the function VixDiskLib_Open(). This could happen (but does not always happen, depending on memory content) when vCenter Server 5.1 manages virtual disk on ESXi 5.0 and older hosts. A fix has been identified, and will be available in the VDDK 5.1 first update release.

It appears that the code base that VDP is based off of exposes this issue.  If you are using VC 5.1 and ESXi 5.0 the suggested workaround is to upgrade to 5.1.

Does the VC / ESXi configuration apply to your situation? 

-Greg Sparks
0 Kudos
pizzingrilli
Contributor
Contributor

No, we are using VC 5.1 with ESX 5.1.

I', going to open a call.

Regards

Sandro

0 Kudos
zafeirop
Contributor
Contributor

Same problem here.

Many Win2008R2 VMs seem to be stuck in 92 %, for hours, and sometimes, for days.

Job usually is completed byitself.

0 Kudos
fibercool
Contributor
Contributor

Hi,

i'm experiencing the same issue, backups hangs at 92% and never finish.

Is there any fix/solution? we can't finish any backup.

Thanks

0 Kudos
HAThomas
Contributor
Contributor

VDP has a known issue with windows 2008 R2 machines. The new version of VDP (5.1.10.32) has this issue resolved.

The workaround for the issue in older VDP is as follows:

1.Go to edit Settings of your WIN 2008 R2 VM

2.In Options tab go to Advanced--> General.

3.Select configuration parameters and in that set the value of Disk.EnableUUID parameter to false.

4.Remove the VM from inventory and add it back. Restart your back up Job.

0 Kudos
OscarDavey
Hot Shot
Hot Shot

Hello,

Try to check disk, defrag of Windows 2008 Server and then to make backup.

Hope will help

Your Oscar

0 Kudos
szilagyic
Hot Shot
Hot Shot

Having this exact problem on 5.5 and have been troubleshooting for weeks.  I too increased the memory to 8 GB and problem completely solved!  Thank you for posting this.

0 Kudos
ankurgoyal01
Enthusiast
Enthusiast

I was facing the same problem and thought that there is an issue with the Appliance which was deployed.

Hence I tried re-deploying the VDP Appliance from Scratch and started the Job.

Even after that the Job was getting stuck at 92%.

As a workaround i upgraded the RAM to 16 GB and rebooted the appliance. Initiated the Job and it worked perfectly fine.

Regards

Ankur

0 Kudos
SavkoorSuhas
Expert
Expert

Do not go by the GUI progress. It is totally incorrect.

Use:

# watch -d mccli activity show --active

Will be more accurate.

# Suhas

If you found this or any other answer useful please consider the use of the Helpful or Correct buttons to award points. Don't Backup. Go Forward! Rubrik Peek into my Website: http://www.virtuallypeculiar.com
0 Kudos