gbbhat
Contributor
Contributor

vCenter Appliance 6.5 File based backups fails with ERROR: Error at process LotusBackup

I have shell script which executes the file based native backup via cron job on a daily basis.

It gets triggered by making the POST call to the endpoint /rest/appliance/recovery/backup/job.

Backup shipping is done via SCP protocol.

Log::

The /var/log/vmware/applmgmt/backup.log shows the following trace

2019-10-17 02:00:38,291 [ConfigFilesBackup:PID-38685] INFO: tarCmd = tar -cz -C / --ignore-failed-read -T "/var/log/vmware/applmgmt/cfg_incl_pq_ggq.lst" -X "/var/log/vmware/applmgmt/cfg_excl_eB6DWP.lst" --warning="no-file-ignored"

2019-10-17 02:00:38,340 [LotusBackup:PID-38686] INFO: stdout: VdcBackupDB: Setting vmdir state to VMDIRD_READ_ONLY

VdcBackupDB: Backing up: /storage/db/vmware-vmdir//data.mdb

VdcBackupDB: Backing up: /storage/db/vmware-vmdir//lock.mdb

VdcBackupDB: Setting vmdir state to (3)

2019-10-17 02:00:38,340 [LotusBackup:PID-38686] INFO: BackupLotus: Dispatching files ['lock.mdb', 'data.mdb', 'partnerlist']

2019-10-17 02:00:38,341 [LotusBackup:PID-38686] INFO: tarCmd = tar -cz -C /tmp/backup_lotus/ --ignore-failed-read --warning="no-file-ignored" "lock.mdb" "data.mdb" "partnerlist"

2019-10-17 02:00:40,595 [LotusBackup:PID-38686] ERROR: Failed to backup Lotus DB

2019-10-17 02:00:40,595 [LotusBackup:PID-38686] ERROR: Failed to backup Lotus DB

Underlying process status. rc: 255

stdout:

stderr:

Traceback (most recent call last):

  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/Lotus.py", line 93, in BackupLotus

    raise BackupRestoreError(err_msg, status)

BackupRestoreError: Failed to backup Lotus DB

Underlying process status. rc: 255

stdout:

stderr:

2019-10-17 02:00:40,696 [LotusBackup:PID-38686] INFO: Successfully completed Lotus cleanup.

2019-10-17 02:00:41,113 [MainProcess:PID-38117] ERROR: Error at process LotusBackup; rc:255.

2019-10-17 02:00:41,113 [MainProcess:PID-38117] ERROR: stderr:Failed to backup Lotus DB

2019-10-17 02:00:41,113 [MainProcess:PID-38117] INFO: Following error message isn't localized:

  stderr:Failed to backup Lotus DB

2019-10-17 02:00:41,114 [MainProcess:PID-38117] ERROR: BackupManager encountered an exception: Hit exception inside process LotusBackup:

2019-10-17 02:00:41,115 [MainProcess:PID-38117] INFO: Cleaning up the failed backup job.

2019-10-17 02:00:41,116 [MainProcess:PID-38117] INFO: Cleaning up all running backup child processes.

2019-10-17 02:00:41,181 [MainProcess:PID-38117] INFO: Cleaning up all sizeFiles.

2019-10-17 02:00:41,181 [MainProcess:PID-38117] INFO: Cleaning up ConfigFiles.

2019-10-17 02:00:41,181 [MainProcess:PID-38117] INFO: Cleaning up Lotus.

2019-10-17 02:00:41,181 [MainProcess:PID-38117] INFO: Successfully completed Lotus cleanup.

Solution I found is simply executing the script again to fire the backup, and it works.

Please help me understand,

1. What is Lotus DB and what information it holds w.r.t vCenter Appliance.

2. How can debug further to find the root cause for this issue.?


Thanks.

0 Kudos
11 Replies
Vijay2027
Expert
Expert

Is it a windows or linux server you are backing up to?

It basically holdsyour sso information.

0 Kudos
Vijay2027
Expert
Expert

And what is the size of data.mdb file?

/storage/db/vmware-vmdir

0 Kudos
gbbhat
Contributor
Contributor

It's a Linux based appliance.

0 Kudos
gbbhat
Contributor
Contributor

I don't have that path,

under /storage/db,

I have

journal

lost+found

vmware-imagebuilder

vmware-vmafd

vpostgres

0 Kudos
Vijay2027
Expert
Expert

External PSC?

0 Kudos
gbbhat
Contributor
Contributor

yes. this is external PSC.

0 Kudos
gbbhat
Contributor
Contributor

also, 3 more vCenters use the same PSC.

0 Kudos
Vijay2027
Expert
Expert

Can you check the size of the data.mdb file from psc ?

/storage/db/vmware-vmdir

0 Kudos
gbbhat
Contributor
Contributor

16K    lost+found

172K    vmware-vmafd

1.9G    vmware-vmdir

0 Kudos
gbbhat
Contributor
Contributor

data.mdb -- 1.8GB

0 Kudos
Vijay2027
Expert
Expert

Check your DM pls..

0 Kudos