File Based Backup 7.0.2 Build 17920168 Fails to run

[manadrain]

I recently upgraded our VCenter Applaince from verison 7.0.1D to version 7.0.2 Build 17920168.  The upgrade took several tries because of failed upgrades and I had to open another case with VMWare Support.  We did get the upgrade to complete but the VCenter Update Manager Service would not start after a successful upgrade.  Also the Upgrade had to be done with a mounted IOS and CLI.  To fix Lifecycle Manger run the process to reset the database.  This got the service to start and I just had to pull updates down again.

However I had a nightly task that ran the built in file based backup to an SMB share and this has worked successfully since we upgraded from a late version of 6.7 to 7.0 Update 1.  If I recall the file based backups to SMB shares was fixed in a later 6.7 build and 7.0 builds for SMB V2.  The problem I am having is that my backup starts.  I can see that the folder get created and it backs up about 7 or 8 tables and then after about 2 minutes the backup gets and error and removes the directory.  

I have tried to reconfigure the backup.  I have a case open with VMware and it has been escalated to a higher level support.  Has anyone else experienced problems like this with VCenter Appliance 7.0.2X.

[Ajay1988]

Will suggest to use  SCP/FTP/HTTP. 

What is the SR?

[manadrain]

I misstated something in the original post this is an SMB V3 share.  It is running on Server 2019.  However I did get in contact with VMWare Support and so we still don't have a solution.  It was suggested that the issue may be with using SMB and try to use another protocol.  I put together a quick ftps server and I still have the same results.  The backup starts, creates the folder and starts backing up different files/tables/directories.  The backup fails and just deletes the directory and files that was backed up. 

Again this didn't happen when we was running VCenter Server Appliance 7.0.1D.  You can see in our backup history that these where successful running this version and then the backups failed when the VCenter was running version 7.0.2.  I did have issues with the upgrade from 7.0.1D to version 7.0.2.  We had to run the upgrade with downloading the FP ISO and mounting it.  Then run the upgrade from the CLI.  The upgrade said it was successful but could not start all services.  The service that would not start was Update Manager.  This is where the tech had me run the process to clear the Update Manger/Lifecycle Manger database.  Once we did this the services would start.  Prior to this I had tried to run the upgrade from the GUI management for 7.0.1D to 7.0.2 and it would stage, install, and give an error and if I recall it was a PostInstall Hook Error.  If I rebooted the VCenter Server the Update Manager Service would not start.  I guess this is similar to what we had with running the upgrade form the CLI.

I don't know if any of these issues are related but I have been having lots of problems upgrading VCenter 7.X in general.

[Ajay1988]

I put together a quick ftps server and I still have the same results.  

I would be interested in the above results. Also share the backup logs from /var/log/vmware/applmgmt  after attempting the vami backup.

In general  if vc has issues ; vami backup will fail.

[manadrain]

I ran a manual backup to my FTPS site this morning and had the same results.  Below is a coy of some of the information in the backup.log file that appears to show when it is failing.  Most of this log is filled with the VCDB backup WAL start not received yet.  The part that I find interesting it this line:

2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::get_vc_version:telemetry.py:312] INFO: vCenter version is 7.0U1d

My VCenter Appliance is running version 7.0.2.00100 Build 17920168.  I was running 7.0U1D prior to this but the upgrade was not directly successful.  I had to open a case with VMWare and we started with the 7.0.1D build that was restored from a file based backup, Ran the update to 7.0.2.00100 from the ISO FP and the CLI.  The tech from VMware helped me with the Upgrade and could see that Update Manger Service would not start.  If I remember we ran the steps that are in KB2147284 for Resetting VMware Update Manger Database and this allowed the Update Manger/Lifecycle Manger service to start.  I rebooted the VCenter server and everything appeared to start correctly and the tech  I worked with said everything looked good with the upgrade.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2021-06-08T09:45:01.490 [20210601-235905-17920168] [VCDB-WAL-Backup:PID-57854] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.632 [20210608-094245-17920168] [VCDB-WAL-Backup:PID-9299] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.799 [20210607-181938-17920168] [VCDB-WAL-Backup:PID-3918] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.809 [20210606-235904-17920168] [VCDB-WAL-Backup:PID-4378] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.811 [20210528-145959-17920168] [VCDB-WAL-Backup:PID-36785] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.811 [20210602-235903-17920168] [VCDB-WAL-Backup:PID-57580] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.812 [20210604-235904-17920168] [VCDB-WAL-Backup:PID-15721] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.812 [20210607-162249-17920168] [VCDB-WAL-Backup:PID-56123] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.814 [20210530-235903-17920168] [VCDB-WAL-Backup:PID-4332] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.814 [20210605-235904-17920168] [VCDB-WAL-Backup:PID-10125] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.814 [20210603-160947-17920168] [VCDB-WAL-Backup:PID-45539] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.814 [20210531-235904-17920168] [VCDB-WAL-Backup:PID-62921] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.815 [20210607-235904-17920168] [VCDB-WAL-Backup:PID-42314] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.815 [20210603-235904-17920168] [VCDB-WAL-Backup:PID-20279] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.821 [20210528-235904-17920168] [VCDB-WAL-Backup:PID-11527] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:01.825 [20210528-154454-17920168] [VCDB-WAL-Backup:PID-23491] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.490 [20210529-235903-17920168] [VCDB-WAL-Backup:PID-13038] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.491 [20210601-235905-17920168] [VCDB-WAL-Backup:PID-57854] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.633 [20210608-094245-17920168] [VCDB-WAL-Backup:PID-9299] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.782 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:372] ERROR: Failed to clean up backup child processes.
Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 616, in main
backupObj.DoBackup()
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 301, in DoBackup
self.LaunchBackupProcesses()
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 269, in LaunchBackupProcesses
self.ExecBackupsInParallel()
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 239, in ExecBackupsInParallel
taskId=self.args.id, operation='BACKUP')
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 202, in LaunchMultipleProcesses
timeout, logger)
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 178, in VerifyProcStatusAndGetArchive
(procRecord.process.name, procRecord.status.excMsg))
Exception: Hit exception inside process LotusBackup:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 251, in CleanupChildProcesses
proc.wait(timeout=30)
File "/usr/lib/python3.7/site-packages/psutil/__init__.py", line 1262, in wait
return self._proc.wait(timeout)
File "/usr/lib/python3.7/site-packages/psutil/_pslinux.py", line 1459, in wrapper
return fun(self, *args, **kwargs)
File "/usr/lib/python3.7/site-packages/psutil/_pslinux.py", line 1637, in wait
return _psposix.wait_pid(self.pid, timeout, self._name)
File "/usr/lib/python3.7/site-packages/psutil/_psposix.py", line 104, in wait_pid
delay = check_timeout(delay)
File "/usr/lib/python3.7/site-packages/psutil/_psposix.py", line 66, in check_timeout
raise TimeoutExpired(timeout, pid=pid, name=proc_name)
psutil._exceptions.TimeoutExpired: psutil.TimeoutExpired timeout after 30 seconds (pid=9278)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 369, in Cleanup
CleanupChildProcesses()
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 259, in CleanupChildProcesses
raise Exception("%s" % str(e))
Exception: psutil.TimeoutExpired timeout after 30 seconds (pid=9278)
2021-06-08T09:45:02.784 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:375] INFO: Cleaning up all sizeFiles.
2021-06-08T09:45:02.784 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up ConfigFiles.
2021-06-08T09:45:02.784 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up ComponentScripts.
2021-06-08T09:45:02.784 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up VCDB.
2021-06-08T09:45:02.784 [20210608-094245-17920168] [MainProcess:PID-8303] [VCDB::BackupVCDBCleanup:VCDB.py:2102] INFO: Finishing full database backup
2021-06-08T09:45:02.790 [20210608-094245-17920168] [MainProcess:PID-8303] [VCDB::_cleanup_full_database_backup:VCDB.py:1356] INFO: Failed to terminate backup job with PID 9300
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [VCDB::BackupVCDBCleanup:VCDB.py:2105] INFO: Successfully cleaned up for VCDB backup.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up Lotus.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [Lotus::BackupLotusCleanup:Lotus.py:127] INFO: Successfully completed Lotus cleanup.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up StatsMonitorDB.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:383] INFO: Cleaning up VTSDB.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [VTSDB::BackupVTSDBCleanup:VTSDB.py:237] INFO: Successfully cleaned up for VTSDB backup.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::Cleanup:BackupManager.py:392] INFO: Cleaning up failed backup files.
2021-06-08T09:45:02.791 [20210608-094245-17920168] [VCDBBackup:PID-9278] [VCDB::BackupVCDB:VCDB.py:2060] ERROR: Encounter error during backup VCDB.
Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1996, in BackupVCDB
br_state.isFastBackupRequired())
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 570, in _start_pg_backup
"backupfast" : 'true' if backup_fast else 'false'})
psycopg2.OperationalError: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

2021-06-08T09:45:02.792 [20210608-094245-17920168] [VCDBBackup:PID-9278] [Proc::UpdateExceptionStatus:Proc.py:383] ERROR: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::check_telemetry_enabled:telemetry.py:285] INFO: Check Feature Switch PG_TELEMETRY state
2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::check_telemetry_enabled:telemetry.py:290] INFO: Check the telemetry conf file
2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::check_telemetry_enabled:telemetry.py:294] INFO: Check configuration parameter vcenter.telemetry
2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::check_telemetry_enabled:telemetry.py:298] INFO: Telemetry for Postgres is enabled
2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::get_vc_version:telemetry.py:306] INFO: Get vCenter version from /etc/vmware/.buildInfo
2021-06-08T09:45:02.793 [20210608-094245-17920168] [VCDBBackup:PID-9278] [telemetry::get_vc_version:telemetry.py:312] INFO: vCenter version is 7.0U1d
2021-06-08T09:45:02.794 [20210608-094245-17920168] [VCDBBackup:PID-9278] [VCDB::BackupVCDB:VCDB.py:2083] INFO: Telemetry command is ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Full Backup', '--data3', '7.0U1d', '--data2', 'Success']
2021-06-08T09:45:02.794 [20210608-094245-17920168] [VCDBBackup:PID-9278] [Proc::RunCmdForOutput:Proc.py:537] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Full Backup', '--data3', '7.0U1d', '--data2', 'Success'].
2021-06-08T09:45:02.800 [20210607-181938-17920168] [VCDB-WAL-Backup:PID-3918] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.811 [20210606-235904-17920168] [VCDB-WAL-Backup:PID-4378] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.813 [20210602-235903-17920168] [VCDB-WAL-Backup:PID-57580] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.813 [20210604-235904-17920168] [VCDB-WAL-Backup:PID-15721] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.813 [20210528-145959-17920168] [VCDB-WAL-Backup:PID-36785] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.814 [20210607-162249-17920168] [VCDB-WAL-Backup:PID-56123] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.815 [20210530-235903-17920168] [VCDB-WAL-Backup:PID-4332] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.815 [20210605-235904-17920168] [VCDB-WAL-Backup:PID-10125] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.815 [20210603-160947-17920168] [VCDB-WAL-Backup:PID-45539] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.816 [20210531-235904-17920168] [VCDB-WAL-Backup:PID-62921] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.817 [20210607-235904-17920168] [VCDB-WAL-Backup:PID-42314] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.817 [20210603-235904-17920168] [VCDB-WAL-Backup:PID-20279] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.822 [20210528-235904-17920168] [VCDB-WAL-Backup:PID-11527] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:02.827 [20210528-154454-17920168] [VCDB-WAL-Backup:PID-23491] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.14 [20210608-094245-17920168] [VCDBBackup:PID-9278] [VCDB::BackupVCDB:VCDB.py:2085] INFO: Success to push telemetry data for backup
2021-06-08T09:45:03.14 [20210608-094245-17920168] [VCDBBackup:PID-9278] [VCDB::BackupVCDBCleanup:VCDB.py:2102] INFO: Finishing full database backup
2021-06-08T09:45:03.18 [20210608-094245-17920168] [VCDBBackup:PID-9278] [VCDB::BackupVCDBCleanup:VCDB.py:2105] INFO: Successfully cleaned up for VCDB backup.
2021-06-08T09:45:03.319 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::HandleBackupCleanup:BackupManager.py:440] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.
2021-06-08T09:45:03.491 [20210529-235903-17920168] [VCDB-WAL-Backup:PID-13038] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.491 [20210601-235905-17920168] [VCDB-WAL-Backup:PID-57854] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.532 [20210608-094245-17920168] [MainProcess:PID-8303] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:275] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
2021-06-08T09:45:03.536 [20210608-094245-17920168] [MainProcess:PID-8303] [BackupManager::main:BackupManager.py:648] INFO: Backup job failed.
2021-06-08T09:45:03.634 [20210608-094245-17920168] [VCDB-WAL-Backup:PID-9299] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.802 [20210607-181938-17920168] [VCDB-WAL-Backup:PID-3918] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.811 [20210606-235904-17920168] [VCDB-WAL-Backup:PID-4378] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.814 [20210604-235904-17920168] [VCDB-WAL-Backup:PID-15721] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.814 [20210602-235903-17920168] [VCDB-WAL-Backup:PID-57580] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.814 [20210528-145959-17920168] [VCDB-WAL-Backup:PID-36785] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.815 [20210607-162249-17920168] [VCDB-WAL-Backup:PID-56123] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.815 [20210530-235903-17920168] [VCDB-WAL-Backup:PID-4332] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.817 [20210605-235904-17920168] [VCDB-WAL-Backup:PID-10125] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.817 [20210603-160947-17920168] [VCDB-WAL-Backup:PID-45539] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.817 [20210531-235904-17920168] [VCDB-WAL-Backup:PID-62921] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.818 [20210607-235904-17920168] [VCDB-WAL-Backup:PID-42314] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.818 [20210603-235904-17920168] [VCDB-WAL-Backup:PID-20279] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.824 [20210528-235904-17920168] [VCDB-WAL-Backup:PID-11527] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:03.828 [20210528-154454-17920168] [VCDB-WAL-Backup:PID-23491] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.491 [20210529-235903-17920168] [VCDB-WAL-Backup:PID-13038] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.493 [20210601-235905-17920168] [VCDB-WAL-Backup:PID-57854] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.635 [20210608-094245-17920168] [VCDB-WAL-Backup:PID-9299] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.803 [20210607-181938-17920168] [VCDB-WAL-Backup:PID-3918] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.813 [20210606-235904-17920168] [VCDB-WAL-Backup:PID-4378] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.815 [20210602-235903-17920168] [VCDB-WAL-Backup:PID-57580] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.815 [20210528-145959-17920168] [VCDB-WAL-Backup:PID-36785] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.815 [20210604-235904-17920168] [VCDB-WAL-Backup:PID-15721] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.815 [20210607-162249-17920168] [VCDB-WAL-Backup:PID-56123] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.817 [20210530-235903-17920168] [VCDB-WAL-Backup:PID-4332] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.818 [20210531-235904-17920168] [VCDB-WAL-Backup:PID-62921] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.818 [20210603-160947-17920168] [VCDB-WAL-Backup:PID-45539] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.818 [20210605-235904-17920168] [VCDB-WAL-Backup:PID-10125] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.819 [20210607-235904-17920168] [VCDB-WAL-Backup:PID-42314] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.819 [20210603-235904-17920168] [VCDB-WAL-Backup:PID-20279] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.825 [20210528-235904-17920168] [VCDB-WAL-Backup:PID-11527] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:04.829 [20210528-154454-17920168] [VCDB-WAL-Backup:PID-23491] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.
2021-06-08T09:45:05.493 [20210529-235903-17920168] [VCDB-WAL-Backup:PID-13038] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.

[manadrain]

Just to add some additional information because I think my issue is still related with the upgrade not completing as expected.  If I view the file /etc/vmware/.buildInfo most things in this file are referencing my old build which was 7.0.1 Update D.  However everything I see looking at VCenter states that I am running 7.0.2.00100.  I decided to setup the VMware Skyline Health Diagnostic VM today to see if I got any additional information back from running this against my VCenter.  I did get an WARNING that referenced KB 76024 "Unexpected content in /etc/issue file" - VCSA patching succedded but failed to update the VAMI build.  I looked at this file on my VCenter Server and it is showing that I am running 7.0.2.00100. 

Again I don't know if these things are related with my backups failing because it thinks it has a mismatch in versions.  I have updated my case that I have open and waiting for a response back from VMWare Support.

[Ajay1988]

that could be related. What is the SR number with Support?

[manadrain]

21223273805

[Ajay1988 · Accepted solution]

You have an issue with /etc/issue file . Also for sure the buildInfo contains wrong info. Something was not right before the last patch.

Please try the below . 

a. Take cold snapshots.
b. Fix the /etc/issue .

Example:
1
2 VMware vCenter Server 7.0.2.00200
3
4 Type: vCenter Server with an embedded Platform Services Controller
5

c. Patch to 7.0 U2b - 17958471 via CLI and check  if backup run's fine . 

[manadrain]

The issue appears to have been resolved. I took another snapshot prior to making any change.   I updated my /etc/issue file on VCenter.  It didn't have the line about the Platform Service Controller and we are running an Embedded Service Controller.  I ran the upgrade for VCenter 7.0.2 build 17958471 from a mounted ISO and CLI.  The update appears to have ran without any problems.  All services appear to have started correctly and my scheduled backup to my SMB share ran last night successfully.  The backup is a lot smaller in size but from what I see this would be because we reset the VMware Update Manger Database according to the VMWare tech to get our upgrade from 7.0.1D to 7.02 to install.  The file vum.gz which I believe is the Update Manager database was over 10GB in size and is now about 560MB.

I also ran another Health Check from the Skyline Health Diagnostics and I am no longer getting a warning about "Build Number does not get updated after applying patch to vCenter Server Appliance or patching fails KB 76024 No fix Available".

I appreciate your help with this issue Ajay1988.

[Ajay1988]

Glad that it worked.