vCenter Server Appliance Backup Process

[insearchof]

vCenter VCSA 6.7

I have an FTP site where I want to place the backup files

I setup a schedule for backup of the VCSA

Backup job failed to complete 

Where can I find out why this failed?

Thank you

Tom

[berndweyand]

ssh to the vcsa  - in the folder /var/log/vmware/applmgmt you see backup.log, backupscheduler.log, backupschedulercron.log.

also see /var/log/vmware/messages

[scott28tt]

Moderator: Thread moved to the vCenter Server area.

[insearchof]

Bewe

Thanks for the info.

The backup.log is a very large file   what can I search for in that log?

Not I Linux person but I did a cat of the log and it was very long too long to scan thru.

Thank you,

[berndweyand]

do you know "grep" for filtering ? search for error, fail etc. or start a new backup job then you have only a short time span to search

[insearchof]

I have some notes on GREP 

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-13T10:31:03.443 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T10:31:03.444 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:03.445 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T10:31:03.858 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:03.860 [ComponentScriptsBackup:PID-23321] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: Unable to perform backup: [Errno 32] Broken pipeException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>

2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: BrokenPipeError: [Errno 32] Broken pipe

2020-05-13T10:31:03.890 [ComponentScriptsBackup:PID-23321] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 1.

2020-05-13T10:31:03.892 [ComponentScriptsBackup:PID-23321] ERROR: Error during component vum backup

2020-05-13T10:31:04.259 [MainProcess:PID-23044] ERROR: Error at process ComponentScriptsBackup; rc:1.

2020-05-13T10:31:04.260 [MainProcess:PID-23044] ERROR: stderr:Error during component vum backup

2020-05-13T10:31:04.261 [MainProcess:PID-23044] ERROR: BackupManager encountered an exception: Hit exception inside process ComponentScriptsBackup:

2020-05-13T10:31:04.318 [LotusBackup:PID-23327] ERROR: rc: 2, stderr: /usr/bin/tar: -: Wrote only 4096 of 10240 bytes

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 2, but expected exit codes are [0].

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

2020-05-13T10:31:04.323 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

2020-05-13T10:31:04.667 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T10:31:05.738 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T10:31:05.746 [MainProcess:PID-23044] ERROR: Failed to perform on-exit cleanup

2020-05-13T14:30:55.541 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T14:30:55.543 [StatsMonitorDBBackup:PID-62440] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: Component rbd backup dispatch failed.

2020-05-13T14:30:55.793 [ComponentScriptsBackup:PID-62442] ERROR: Failed to dispatch component rbd backup

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-13T14:30:56.512 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T14:30:57.442 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T14:30:57.451 [MainProcess:PID-62063] ERROR: Failed to perform on-exit cleanup

Looks like it had a problem cleaning up the data in the folder should I clear the folder and try again?

Thoughts.

Thank you

Tom

[berndweyand]

no there are to many errors before cleaning up - the main error will sit on the beginning of the job.

better rename the backup.log and start a new backup - then you have a small backup.log to read

[insearchof]

renamed the backup.log file and restarted the backup job and it is just as large as before.

I got this on the console

Structure com.vmware.appliance.recovery.backup.job.details.info has a union with a field not required for this case = end_time

On the FTP Server in the folder I see only two files    config_files.tar.gz  and lotus_backup.tar.gz

On the Appliance the backup

It is just sitting there no % moving

Any ideas?

Thanks

Tom

[insearchof]

It finally finished and it failed again

this time not so many errors

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-13T17:27:05.280 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T17:27:05.282 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T17:27:05.283 [StatsMonitorDBBackup:PID-12644] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T17:27:05.475 [ComponentScriptsBackup:PID-12645] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Component rbd backup dispatch failed.

2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Failed to dispatch component rbd backup

2020-05-13T17:27:05.772 [MainProcess:PID-12343] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-13T17:27:05.995 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T17:27:06.934 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T17:27:06.943 [MainProcess:PID-12343] ERROR: Failed to perform on-exit cleanup

Any ideas?

Thank you

Tom

[berndweyand]

maybe this can help ?

https://www.techcrumble.net/2018/06/vcsa-6-7-backup-failed/

VCSA 6.7 Appliance Native Backup Failing at BackupLotus

or you can post the backup.log here ? it should not contain any confidential information. but please do not post the content here - better attach as .txt or .zip

[insearchof]

Bewe

I was thinking about the permissions issue on my SFTP server I checked and the user account I am using has full control of the folder in use.  It does place several folders and two file there so it can not be a permissions problem.

I would be glad to post the backup log file but I am unable to copy it off the VCSA appliance.

I have WINSCP on my windows 10 computer   but unable to scp to the appliance.

Any ideas on how I can get the file off?

Thank you

Tom

[berndweyand]

here is described how to scp to the appliance : VMware Knowledge Base

but do a test:

- ssh to the appliance

- from there open a ftp session to your target

- upload the backup.log to the ftp

the side effect is that you test with this upload that the permissions to the ftp are correct

[insearchof]

That worked see attached log

[berndweyand]

i will check the log tomorrow.

could be an firewall problem because vcsa uses passive ftp. what os is your ftp-server ?

[insearchof]

My FTP server is Windows 2012 R2  Using IIS  for FTP

The firewall in Windows Firewall I will look into the firewall setting too.

Thanks

Tom

[berndweyand]

please doublecheck your permissions on ftp: open a ftp session from vcsa and create, modify and delete files and folders.

also for a test disable the firewall on your ftp

the log reports trouble with statsmonitordbbackup and componentsscriptbackup, also with modifying files on the backup target

[insearchof]

bewe

Ok

1  How can I open an FTP session on the VCSA from the command prompt I type shell    then type FTP and get unknown command

Command> shell

Shell access is granted to root

root [ ~ ]# ftp

bash: ftp: command not found

root [ ~ ]#

.

I then disabled the firewall on the FTP Server and ran the job the same thing happened job failed only copied the two files and stops.

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-14T12:23:37.991 [StatsMonitorDBBackup:PID-37802] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-14T12:23:37.992 [StatsMonitorDBBackup:PID-37802] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-14T12:23:37.993 [StatsMonitorDBBackup:PID-37802] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-14T12:23:37.993 [StatsMonitorDBBackup:PID-37802] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-14T12:23:37.994 [StatsMonitorDBBackup:PID-37802] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-14T12:23:38.307 [ComponentScriptsBackup:PID-37807] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-14T12:23:38.307 [ComponentScriptsBackup:PID-37807] ERROR: Component rbd backup dispatch failed.

2020-05-14T12:23:38.308 [ComponentScriptsBackup:PID-37807] ERROR: Failed to dispatch component rbd backup

2020-05-14T12:23:38.691 [MainProcess:PID-37647] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-14T12:23:38.691 [MainProcess:PID-37647] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-14T12:23:38.692 [MainProcess:PID-37647] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-14T12:23:38.876 [MainProcess:PID-37647] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-14T12:23:39.748 [MainProcess:PID-37647] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-14T12:23:39.763 [MainProcess:PID-37647] ERROR: Failed to perform on-exit cleanup

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]#

Any ideas?

Thank you

Tom

[berndweyand]

ah sorry - vcsa has no ftp client. but you can test from windows if your account can create, delete files and folders

because of the errors i will look further

[insearchof]

Using the account in filzilla  FTP app I was unable to connect So I chose another account and I was able to connect to the FTP folder    created files folders deleted files and folders no problem

I tried that account on the backup job but I get the same results

I have the firewall still off too

[berndweyand]

hi,

do you have another status for backup ?

with deactivated firewall and proper permissions to ftp it should run