Highlighted
Enthusiast
Enthusiast

VDP 6.1.3 / vSphere 6.5 - 92% hang on backup jobs, not the issue in the release notes

Jump to solution

Hey guys,

With a fresh VDP 6.1.3 install on top of vSphere 6.5, most backup jobs hang at 92%.  This is not the issue documented in the release notes, and the backups almost never complete. I DID NOT have this issue with VDP 6.1 under vSphere 6.0.

I have tried backing up photon OS machines, centos 6.5 machines, centos 6.6+ machines, and centos 7 machines.  So far the only ones to successfully complete are some of those old centos 6.5 ones.  I'm not sure if the O/S version is a red herring here, or if it's related to some other pattern I just can't identify at the time.

I have tried backing up images with the quiesce option both on and off, and it doesn't seem to make a difference in the result.  The backups hang regardless and are unsuccessful.

I should mention the storage was created along with the appliance and is on VSAN, if that makes any difference, but - this absolutely all worked with VDP 6.1 and vSphere 6.0.

I looked in a failed job's vmimage log in /usr/local/avamarclient/var/ and noticed this at the end:

2017-01-17T04:02:15.693+04:00 avvcbimage Info <9679>: Virtual disk connection to '[vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1.vmdk' has been closed
2017-01-17T04:02:15.693+04:00 avvcbimage Info <19718>: Found Disk from Device Key: '2001'.
2017-01-17T04:02:15.693+04:00 avvcbimage Info <9674>: virtual disk [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk will be connected readonly
2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_OpenEx: Open a disk.

2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk.

2017-01-17T04:02:16.008+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: Request RandomAccessRO diskKey = 2001, readOnly = 1, openSnapshot = 0.

2017-01-17T04:02:16.094+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.

2017-01-17T04:02:16.101+04:00 avvcbimage FATAL <5889>: Fatal signal 11 in pid 15065
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  FATAL ERROR: <0001> uapp::handlefatal: Fatal signal 11
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006b93b1
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba127
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba44b
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba53e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 0000000000628810
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c90850
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5050193f
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c277
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c75e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b142
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b678
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecab73
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecb123
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004284bc
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004291f9
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004993c4
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 000000000070ac4c
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c88806
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e558cd9bd
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  ERROR: <0001> uapp::handlefatal: aborting program pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage FATAL <5890>: handlefatal: Aborting program with code 176, pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.

2017-01-17T12:00:07.999+04:00 avvcbimage Info <40660>: vcbimage_progress::terminate

the log ends there.

This looks serious to me, but it's hard to figure out where to go from here.

Does anyone have any advice as to how to proceed?

Thanks for any assistan

1 Solution

Accepted Solutions
Highlighted

Ah I have seen this issue.

There is VDDK 6.5 issues for VSAN / vVOL on SAN mode transport.

To fix this:

1. SSH into VDP and go to /usr/local/avamarclient/var

2. Edit avvcbimageAll.cmd using vi

3. Add this parameter: (There are two '-')

--transport=nbd

4. Save the file

5. Restart avagent

service avagent-vmware restart


This will run backups on NBD protocol which will fix your issue.



Suhas

If you found this or any other answer useful please consider the use of the Helpful or Correct buttons to award points. Don't Backup. Go Forward! Rubrik Peek into my Website: http://www.virtuallypeculiar.com

View solution in original post

5 Replies
Highlighted

Ah I have seen this issue.

There is VDDK 6.5 issues for VSAN / vVOL on SAN mode transport.

To fix this:

1. SSH into VDP and go to /usr/local/avamarclient/var

2. Edit avvcbimageAll.cmd using vi

3. Add this parameter: (There are two '-')

--transport=nbd

4. Save the file

5. Restart avagent

service avagent-vmware restart


This will run backups on NBD protocol which will fix your issue.



Suhas

If you found this or any other answer useful please consider the use of the Helpful or Correct buttons to award points. Don't Backup. Go Forward! Rubrik Peek into my Website: http://www.virtuallypeculiar.com

View solution in original post

Highlighted
Enthusiast
Enthusiast

You, my friend, are a life saver.  This worked!

Did the default protocol change from 6.0 to 6.1.3?

Anyway my backups are working again and this is a huge concern lifted.  Would love to see this as a KB!

Thanks again!

Highlighted

I don't believe there is a public facing KB yet. There are issues with VDDK 6.5 release with VSAN. Probably might be fixed sometime down the line.

Glad the workaround helped.

Suhas

If you found this or any other answer useful please consider the use of the Helpful or Correct buttons to award points. Don't Backup. Go Forward! Rubrik Peek into my Website: http://www.virtuallypeculiar.com
0 Kudos
Highlighted
Contributor
Contributor

Would this explain why I can't seem to back up any VMs with more than one attached VMDK residing on VSAN?

I found that the backups work fine if all VMs with multiple VMDKs attached are on an NFS datastore, but not when they're on VSAN.

0 Kudos
Highlighted

I believe this would be the case for any number of VMDK if the VM is on vSAN.

Suhas

If you found this or any other answer useful please consider the use of the Helpful or Correct buttons to award points. Don't Backup. Go Forward! Rubrik Peek into my Website: http://www.virtuallypeculiar.com
0 Kudos