I have not had any luck in the past year trying to get VDP backup to work reliably.
can anyone tell me what I might be doing wrong ?
You're not doing anything wrong. VDP is (was) not a very good product, primarily owing to the fact it's based on Avamar. At this point, though, salvage your losses and pick another data protection product which is both more robust and has a lifespan, because VDP is dead. A community favorite here is Veeam and it seems to be the defacto standard in protection of vSphere and Hyper-V workloads. There are of course others, however.
If you had at least some logs or any information we may be able to help. I've ran VDP for years with no problems, but I do know it does have it's limitations on large data sets. I think the most I've been able to backup on VDP was about 8TB on a stable basis. Above that, we had to look into dedicated spindles or DataDomain to handle the backup data.
The most common scenario is always lack of memory on vdp. Or the disks dedicated for VDP aren't fast enough. You can usually see this by monitoring the swap usage on the linux sub system. If the swaps are high, Gsan wont like it and will crash.
Hope this helps.