VMware Cloud Community
dschoenbeck
Contributor
Contributor

Veeam backup taking 9 hours on a 2tb backup then disconnecting for 20 minutes at the end

Im using Veeam 8.0 to backup one of my File server VMs    windows server 2008r2   Its about a 2 TB VM and it takes about 9 hours just for an incremental that is only processing about 13 gb of data.    At the end of the backup,  Vsphere tells me its removing snapshot.   it gets to about 99% and then the VM disconnects from the system and it takes about 20 minutes for it to come back online.   I have talked to Veeam and they say its normal for it to do what its doing but not at this length of time and theytold me to go to VMware.    VMware tells me to try the boards for help.

anyone have any Idea why its taking this long and disconnecting.

thanks

Dave

Reply
0 Kudos
9 Replies
ldelloca
Virtuoso
Virtuoso

Hi Dave,

some more informations are needed to tell if it's normal or not. What's the storage and the network of the environment? How is Veeam deployed? What's the backup storage? And in the backup report, what is the average speed, transferred data, and the bottleneck stats?

Luca

Luca Dell'Oca | vExpert 2011-2012-2013-2014-2015-2016-2017, VCAP-DCD, CISSP #58353 | http://www.virtualtothecore.com | @dellock6 | http://www.linkedin.com/in/lucadelloca | If you find this post useful, please consider awarding points for "Correct" or "Helpful"
Reply
0 Kudos
dschoenbeck
Contributor
Contributor

Here is the log from the last backup.

1/3/2016 10:00:21 PM :: Job started at 1/3/2016 10:00:18 PM

1/3/2016 10:00:22 PM :: Building VM list

1/3/2016 10:01:31 PM :: VM size: 2.1 TB

1/3/2016 10:01:31 PM :: Changed block tracking is enabled

1/3/2016 10:01:43 PM :: Preparing next VM for processing

1/3/2016 10:01:48 PM :: All VMs have been queued for processing

1/3/2016 10:01:49 PM :: Processing GODC2

1/4/2016 7:35:01 AM :: Queued for processing at 1/4/2016 7:35:01 AM

1/4/2016 7:35:01 AM :: Required backup infrastructure resources have been assigned

1/4/2016 7:35:07 AM :: Full backup file merge completed successfully

1/4/2016 7:54:16 AM :: Load: Source 95% > Proxy 56% > Network 5% > Target 1%

1/4/2016 7:54:17 AM :: Primary bottleneck: Source

1/4/2016 7:54:18 AM :: Job finished at 1/4/2016 7:54:16 AM

it says processing speed for 8mbs.      

for main storage I am on Dell Equallogic arrays connected with Iscsi   and for backup, it is running a plain generic storage array that runs windows server on in on its own contained system. so the backup storage is directly connected to the backup OS.

Reply
0 Kudos
ldelloca
Virtuoso
Virtuoso

95% bottleneck on the source clearly means the production storage cannot read data faster than it is doing right now.

And this also explains why the snapshot commit takes so long and the VM hungs during this phase.

If you select the VM on the left side of the report, what are the details of the VM processing? If the veeam server is not configured for san mode, you are probably using network mode...

Luca

Luca Dell'Oca | vExpert 2011-2012-2013-2014-2015-2016-2017, VCAP-DCD, CISSP #58353 | http://www.virtualtothecore.com | @dellock6 | http://www.linkedin.com/in/lucadelloca | If you find this post useful, please consider awarding points for "Correct" or "Helpful"
Reply
0 Kudos
ldelloca
Virtuoso
Virtuoso

I just noticed now, the job starts at 10pm but this specific VM is only processed at the end:

1/4/2016 7:35:01 AM :: Queued for processing at 1/4/2016 7:35:01 AM

1/4/2016 7:35:01 AM :: Required backup infrastructure resources have been assigned

1/4/2016 7:35:07 AM :: Full backup file merge completed successfully

1/4/2016 7:54:16 AM :: Load: Source 95% > Proxy 56% > Network 5% > Target 1%

1/4/2016 7:54:17 AM :: Primary bottleneck: Source

1/4/2016 7:54:18 AM :: Job finished at 1/4/2016 7:54:16 AM

And it just took 21 minutes, not 9 hours!!!!

The job has other VMs before this one that are processed?

Luca Dell'Oca | vExpert 2011-2012-2013-2014-2015-2016-2017, VCAP-DCD, CISSP #58353 | http://www.virtualtothecore.com | @dellock6 | http://www.linkedin.com/in/lucadelloca | If you find this post useful, please consider awarding points for "Correct" or "Helpful"
Reply
0 Kudos
cyberfed2727
Enthusiast
Enthusiast

Based on what you shared my bet is on your network, either the iSCSI traffic or the entire network. 8MB/s is absolutely horrible.

Check to make sure Veeam hasn't left any stale snapshots on the VM. I've seen that cause issues before similar to what you described. Also ensure there is enough free disk space on the datastore to support the snapshot/Veeam operations.

Reply
0 Kudos
dschoenbeck
Contributor
Contributor

I looked at the snapshot manager for the VM and its lists a "VEEAM backup temporary snapshot"   I can not consolidate it. and if I click delete it says " please do not delete it is in use by veeam backup"    All my backup jobs are finished.   

could this be the issue?

Reply
0 Kudos
cyberfed2727
Enthusiast
Enthusiast

It's possible yes. If the Veeam job has completed then you are OK to delete the snapshot.

This used to happen to us quite a bit on older Veeam installations where it wouldn't cleanup its snapshot after finishing the job for whatever reason.

We wrote a PowerCLI script to check for these snapshots after Veeam completed and it would email us if they existed so we could keep an eye on it. With later versions of Veeam this happened a lot less. Delete the stale snapshot it cant hurt anything.

Reply
0 Kudos
dschoenbeck
Contributor
Contributor

Well I clicked delete and its been about 20 minutes and its at 1% removed only.   basically the same thing it did on the backup.   I will give it time tonight and hope for the best.

Reply
0 Kudos
cyberfed2727
Enthusiast
Enthusiast

If its a super old snapshot that has grown in size over time it will take a long time to remove the snapshot. Remember it has to consolidate all those changes back to the VMDK. Give it time.

Cheers mate.

Reply
0 Kudos