VMware Cloud Community
jward256
Contributor
Contributor

vCenter 7 Backup Fail

I have been battling an issue with backups for a couple weeks now. Backups run, some files get uploaded to the ftp server, then the backup fails. Backups had been running fine and suddenly stopped. I have not been able to identify a change that coincides with the failures.

Backups had been running to an sftp server. Yesterday I installed the 7.0.2 update to vCenter, dumped the original backup folder, and changed the backup method to ftp to simplify things. What appears to be the key line in the backup.log file is:

[Proc::UpdateExceptionStatus:Proc.py:383] ERROR: Full backup not allowed during VM snapshot. 

Looking at VCDB.py it would appear that the script thinks that there is an active snapshot on vCenter. I have confirmed that there is not a snapshot. 

Ideas?

 

9 Replies
CLS78
Contributor
Contributor

The same issue is happening in 1 of my environments since 25 March 2021.  I'm using NFS protocol for the VAMI scheduled backup.  The vCSA version is 7.0.1.0300 build number 17491101 (update 1d).

The error message in the backup.log is:

"Error: Encounter error during backup VCDB....File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py, line 1937, BackupVCDB raise Exception('Full backup not allowed during VM snapshot')"

I have confirmed there is no snapshot either.  

However, I have another vSphere environment with same version/build number and same VAMI scheduled backup settings that is running with no issues.

Ideas?

Reply
0 Kudos
CLS78
Contributor
Contributor

Here was the fix action for the issue:

Please check whether the snapshot marker file exists, and if so, rename it. This can sometimes be left on the system in case of backup failure.

# mv /etc/vmware/backupMarker.txt /etc/vmware/backupMarker.txt-BAK

The file was empty on my vCSA so I removed it.

 

jward256
Contributor
Contributor

Thank you so much, that did it!

Reply
0 Kudos
GuiMendonca
Contributor
Contributor

You are a life saver! Thanks a lot!

Reply
0 Kudos
anyMOTION
Contributor
Contributor

IT WORKED!!

THANK YOU ❤️

Reply
0 Kudos
Sitz
Contributor
Contributor

This works for me too!

Thank you so much.

Reply
0 Kudos
Ricardo42
Contributor
Contributor

I'm using 7.0.3.01700 and my backuplog was showing:

[MountedFSStorageIOLib::_run_timed_cmd:MountedFSStorageIOLib.py:119] ERROR: Cmd ['/usr/bin/mount', '-t', 'nfs',...

 

Resolution required a change to /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py

 

use command: cat -n /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py | grep -i 'ErrorCodes.timeout'

 

if cat command returns "200 if e.errorcode == ErrorCodes.timeout:" line 200 needs to be change to read "e.errcode == ErrCodes.timeout"

 

then reboot vCenter 

Reply
0 Kudos
aurora-chase
Contributor
Contributor

Hello,
Even though you have confirmed that there are no active snapshots, sometimes vCenter's status may not update immediately. You can verify the snapshot status through the ESXi command line or use a PowerCLI script to ensure that nothing is missing.
If the backup script checks the VCDB database to confirm the snapshot status, it may need to check whether there are any remaining records in the database indicating a non-existent snapshot.
When dealing with such issues, it is crucial to choose a stable and reliable backup solution. Vinchin Backup & Recovery is designed to help in situations like these, providing detailed logs and reports that can help you diagnose the problem faster. If you're considering alternatives to your backup solution, Vinchin may be an option worth considering.

Reply
0 Kudos
BarryGrowler
Enthusiast
Enthusiast

When facing vCenter backup failures due to a supposed active VM snapshot, despite none existing, consider addressing remnant backup marker files. Removing or renaming the backupMarker.txt often resolves the issue by clearing the system's mistaken snapshot status, allowing backups to proceed. Additionally, for NFS backup errors, modifying the MountedFSStorageIOLib.py script to correct timeout error handling has proven effective. Verifying snapshot statuses via ESXi command line or PowerCLI can also help identify discrepancies. If these steps don't resolve the issue, exploring dedicated backup solutions, designed for VM environments, might provide a more stable and insightful backup process.

Reply
0 Kudos