5 Replies Latest reply on Jan 23, 2017 12:00 AM by SavkoorSuhas

    VDP 6.1.3 / vSphere 6.5 - 92% hang on backup jobs, not the issue in the release notes

    eai Novice

      Hey guys,


      With a fresh VDP 6.1.3 install on top of vSphere 6.5, most backup jobs hang at 92%.  This is not the issue documented in the release notes, and the backups almost never complete. I DID NOT have this issue with VDP 6.1 under vSphere 6.0.


      I have tried backing up photon OS machines, centos 6.5 machines, centos 6.6+ machines, and centos 7 machines.  So far the only ones to successfully complete are some of those old centos 6.5 ones.  I'm not sure if the O/S version is a red herring here, or if it's related to some other pattern I just can't identify at the time.


      I have tried backing up images with the quiesce option both on and off, and it doesn't seem to make a difference in the result.  The backups hang regardless and are unsuccessful.


      I should mention the storage was created along with the appliance and is on VSAN, if that makes any difference, but - this absolutely all worked with VDP 6.1 and vSphere 6.0.


      I looked in a failed job's vmimage log in /usr/local/avamarclient/var/ and noticed this at the end:

      2017-01-17T04:02:15.693+04:00 avvcbimage Info <9679>: Virtual disk connection to '[vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1.vmdk' has been closed
      2017-01-17T04:02:15.693+04:00 avvcbimage Info <19718>: Found Disk from Device Key: '2001'.
      2017-01-17T04:02:15.693+04:00 avvcbimage Info <9674>: virtual disk [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk will be connected readonly
      2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_OpenEx: Open a disk.

      2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk.

      2017-01-17T04:02:16.008+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: Request RandomAccessRO diskKey = 2001, readOnly = 1, openSnapshot = 0.

      2017-01-17T04:02:16.094+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.

      2017-01-17T04:02:16.101+04:00 avvcbimage FATAL <5889>: Fatal signal 11 in pid 15065
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  FATAL ERROR: <0001> uapp::handlefatal: Fatal signal 11
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006b93b1
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba127
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba44b
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba53e
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 0000000000628810
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c90850
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5050193f
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c277
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c75e
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b142
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b678
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecab73
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecb123
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004284bc
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004291f9
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004993c4
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 000000000070ac4c
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c88806
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e558cd9bd
      2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  ERROR: <0001> uapp::handlefatal: aborting program pid=15065, sig=11
      2017-01-17T04:02:16.103+04:00 avvcbimage FATAL <5890>: handlefatal: Aborting program with code 176, pid=15065, sig=11
      2017-01-17T04:02:16.103+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.

      2017-01-17T12:00:07.999+04:00 avvcbimage Info <40660>: vcbimage_progress::terminate


      the log ends there.


      This looks serious to me, but it's hard to figure out where to go from here.


      Does anyone have any advice as to how to proceed?


      Thanks for any assistan