1 2 Previous Next 29 Replies Latest reply on May 20, 2020 11:03 AM by insearchof

    vCenter Server Appliance Backup Process

    insearchof Hot Shot

      vCenter VCSA 6.7

       

      I have an FTP site where I want to place the backup files

       

      I setup a schedule for backup of the VCSA

       

       

      Backup job failed to complete 

       

      Where can I find out why this failed?

       

      Thank you

       

      Tom

        • 1. Re: vCenter Server Appliance Backup Process
          bewe Expert

          ssh to the vcsa  - in the folder /var/log/vmware/applmgmt you see backup.log, backupscheduler.log, backupschedulercron.log.

          also see /var/log/vmware/messages

          • 2. Re: vCenter Server Appliance Backup Process
            scott28tt Champion
            Community WarriorsUser ModeratorsVMware Employees

            Moderator: Thread moved to the vCenter Server area.

            • 3. Re: vCenter Server Appliance Backup Process
              insearchof Hot Shot

              Bewe

               

              Thanks for the info.

               

              The backup.log is a very large file   what can I search for in that log?

               

              Not I Linux person but I did a cat of the log and it was very long too long to scan thru.

               

               

              Thank you,

              • 4. Re: vCenter Server Appliance Backup Process
                bewe Expert

                do you know "grep" for filtering ? search for error, fail etc. or start a new backup job then you have only a short time span to search

                • 5. Re: vCenter Server Appliance Backup Process
                  insearchof Hot Shot

                  I have some notes on GREP 

                   

                  root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

                  2020-05-13T10:31:03.443 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is -13, but expected exit codes are [0].

                  2020-05-13T10:31:03.444 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 1, stderr: Traceback (most recent call last):

                  2020-05-13T10:31:03.445 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is 1, but expected exit codes are [0].

                  2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                  2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

                  2020-05-13T10:31:03.858 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is -13, but expected exit codes are [0].

                  2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: rc: 1, stderr: Traceback (most recent call last):

                  2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is 1, but expected exit codes are [0].

                  2020-05-13T10:31:03.860 [ComponentScriptsBackup:PID-23321] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                  2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: Unable to perform backup: [Errno 32] Broken pipeException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>

                  2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: BrokenPipeError: [Errno 32] Broken pipe

                  2020-05-13T10:31:03.890 [ComponentScriptsBackup:PID-23321] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 1.

                  2020-05-13T10:31:03.892 [ComponentScriptsBackup:PID-23321] ERROR: Error during component vum backup

                  2020-05-13T10:31:04.259 [MainProcess:PID-23044] ERROR: Error at process ComponentScriptsBackup; rc:1.

                  2020-05-13T10:31:04.260 [MainProcess:PID-23044] ERROR: stderr:Error during component vum backup

                  2020-05-13T10:31:04.261 [MainProcess:PID-23044] ERROR: BackupManager encountered an exception: Hit exception inside process ComponentScriptsBackup:

                  2020-05-13T10:31:04.318 [LotusBackup:PID-23327] ERROR: rc: 2, stderr: /usr/bin/tar: -: Wrote only 4096 of 10240 bytes

                  2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 2, but expected exit codes are [0].

                  2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: rc: 1, stderr: Traceback (most recent call last):

                  2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 1, but expected exit codes are [0].

                  2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                  2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

                  2020-05-13T10:31:04.323 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

                  2020-05-13T10:31:04.667 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                  2020-05-13T10:31:05.738 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                  2020-05-13T10:31:05.746 [MainProcess:PID-23044] ERROR: Failed to perform on-exit cleanup

                  2020-05-13T14:30:55.541 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is -13, but expected exit codes are [0].

                  2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 1, stderr: Traceback (most recent call last):

                  2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is 1, but expected exit codes are [0].

                  2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                  2020-05-13T14:30:55.543 [StatsMonitorDBBackup:PID-62440] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

                  2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                  2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: Component rbd backup dispatch failed.

                  2020-05-13T14:30:55.793 [ComponentScriptsBackup:PID-62442] ERROR: Failed to dispatch component rbd backup

                  2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

                  2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

                  2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

                  2020-05-13T14:30:56.512 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                  2020-05-13T14:30:57.442 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                  2020-05-13T14:30:57.451 [MainProcess:PID-62063] ERROR: Failed to perform on-exit cleanup

                   

                  Looks like it had a problem cleaning up the data in the folder should I clear the folder and try again?

                   

                  Thoughts.

                   

                  Thank you

                   

                  Tom

                  • 6. Re: vCenter Server Appliance Backup Process
                    bewe Expert

                    no there are to many errors before cleaning up - the main error will sit on the beginning of the job.

                    better rename the backup.log and start a new backup - then you have a small backup.log to read

                    • 7. Re: vCenter Server Appliance Backup Process
                      insearchof Hot Shot

                      renamed the backup.log file and restarted the backup job and it is just as large as before.

                       

                      I got this on the console

                       

                       

                       

                      Structure com.vmware.appliance.recovery.backup.job.details.info has a union with a field not required for this case = end_time

                       

                      On the FTP Server in the folder I see only two files    config_files.tar.gz  and lotus_backup.tar.gz

                      On the Appliance the backup

                      It is just sitting there no % moving

                       

                      Any ideas?

                       

                      Thanks

                       

                      Tom

                      • 8. Re: vCenter Server Appliance Backup Process
                        insearchof Hot Shot

                        It finally finished and it failed again

                         

                         

                        this time not so many errors

                         

                        root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

                        2020-05-13T17:27:05.280 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is -13, but expected exit codes are [0].

                        2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 1, stderr: Traceback (most recent call last):

                        2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is 1, but expected exit codes are [0].

                        2020-05-13T17:27:05.282 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                        2020-05-13T17:27:05.283 [StatsMonitorDBBackup:PID-12644] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

                        2020-05-13T17:27:05.475 [ComponentScriptsBackup:PID-12645] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

                        2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Component rbd backup dispatch failed.

                        2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Failed to dispatch component rbd backup

                        2020-05-13T17:27:05.772 [MainProcess:PID-12343] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

                        2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

                        2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

                        2020-05-13T17:27:05.995 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                        2020-05-13T17:27:06.934 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

                        2020-05-13T17:27:06.943 [MainProcess:PID-12343] ERROR: Failed to perform on-exit cleanup

                         

                         

                        Any ideas?

                         

                        Thank you

                         

                        Tom

                        • 9. Re: vCenter Server Appliance Backup Process
                          bewe Expert

                          maybe this can help ?

                           

                          https://www.techcrumble.net/2018/06/vcsa-6-7-backup-failed/

                          VCSA 6.7 Appliance Native Backup Failing at BackupLotus

                           

                          or you can post the backup.log here ? it should not contain any confidential information. but please do not post the content here - better attach as .txt or .zip

                          • 10. Re: vCenter Server Appliance Backup Process
                            insearchof Hot Shot

                            Bewe

                             

                            I was thinking about the permissions issue on my SFTP server I checked and the user account I am using has full control of the folder in use.  It does place several folders and two file there so it can not be a permissions problem.

                             

                            I would be glad to post the backup log file but I am unable to copy it off the VCSA appliance.

                             

                            I have WINSCP on my windows 10 computer   but unable to scp to the appliance.

                             

                            Any ideas on how I can get the file off?

                             

                             

                            Thank you

                             

                            Tom

                            • 11. Re: vCenter Server Appliance Backup Process
                              bewe Expert

                              here is described how to scp to the appliance : VMware Knowledge Base

                               

                              but do a test:

                              - ssh to the appliance

                              - from there open a ftp session to your target

                              - upload the backup.log to the ftp

                               

                              the side effect is that you test with this upload that the permissions to the ftp are correct

                              • 12. Re: vCenter Server Appliance Backup Process
                                insearchof Hot Shot

                                That worked see attached log

                                • 13. Re: vCenter Server Appliance Backup Process
                                  bewe Expert

                                  i will check the log tomorrow.

                                  could be an firewall problem because vcsa uses passive ftp. what os is your ftp-server ?

                                  • 14. Re: vCenter Server Appliance Backup Process
                                    insearchof Hot Shot

                                    My FTP server is Windows 2012 R2  Using IIS  for FTP

                                     

                                    The firewall in Windows Firewall I will look into the firewall setting too.

                                     

                                    Thanks

                                     

                                    Tom

                                    1 2 Previous Next