SMcT
Enthusiast
Enthusiast

Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state.

Jump to solution

Hi All

My VCSA backups have started failing recently with the following error in the VAMI - Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state. 

I'm not seeing any function problems with the operation of my vcsa and can backup successfully when I unselect the DB checks.  I haven't found anything compelling in the log files either. 

Can anyone direct me to a specific log file or provide any thoughts?  I'm either not searching for the right thing on Google or there isn't much out there on this...

VCSA 7.0.2 - Build 17694817

(Its a lab so no support)

Thanks 

0 Kudos
1 Solution

Accepted Solutions
SMcT
Enthusiast
Enthusiast

Inadvertently fixed this by restoring the vcsa from file backup for another issue.  The backup I restored was taken when the issue existed.  I'm assuming the recreation of the appliance removed an issue, that wasn't something present in the backup.

View solution in original post

0 Kudos
10 Replies
Ajay1988
VMware Employee
VMware Employee

Use the script from  https://kb.vmware.com/s/article/53062   and check for any issues. 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
SMcT
Enthusiast
Enthusiast

Hi Ajay

thanks for the imformation.  I ran the script and got the following output.

root@vm-vcsa-01 [ / ]# /scripts/vpgsql-consistency-check.sh -a -b -f /tmp/ -v
amcheck extension is not installed
amcheck extension is not installed

05-10-2021T10:48:15 Starting the catalog diagnostic
05-10-2021T10:48:16 Successfully finished the catalog diagnostic

Taking a backup of the VCDB to: /tmp/

05-10-2021T10:48:16 Starting backup of the VCDB database.
05-10-2021T10:48:23 Successfully finished backup of the VCDB database.

05-10-2021T10:48:23 Starting backup of the AFD database.
05-10-2021T10:48:24 Successfully finished backup of the AFD database.

05-10-2021T10:48:24 Starting PSQL Instance Vacuum
vCenter is currently running.
05-10-2021T10:48:46 Finished PSQL Instance Vacuum

05-10-2021T10:48:46 Starting Consistency Check
05-10-2021T10:48:54 Finished Consistency Check.

Consistency Check PASSED. The database has been found to be consistent. No further action required.

However I am still getting the error when backing up the DB via the VAMI.  I have attached the log file which mentions 3 circular foreign-key constraints on the following 3 tables - vpx_task, vpx_entity, pm_config_commits.

Thanks 

Tags (1)
0 Kudos
Ajay1988
VMware Employee
VMware Employee

Attach the  backup log from  /var/log/vmware/applmgmt

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
0 Kudos
SMcT
Enthusiast
Enthusiast
 
0 Kudos
Ajay1988
VMware Employee
VMware Employee

2021-05-10T09:50:35.541 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::getDbHealthStatus:BackupManager.py:480] INFO: getDbHealthStatus returns status as UNHEALTHY
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::HealthCheckStatus:BackupManager.py:493] ERROR: HealthCheckStatus Overall status: Failed. Check /var/log/vmware/applmgmt/applmgmt.log for more details.
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::main:BackupManager.py:625] ERROR: BackupManager encountered an exception: Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.

 

Please grab the   /var/log/vmware/applmgmt/applmgmt.log and   /var/log/vmware/cloudvm/fileintegrity.log  . Also   /etc/vmware/appliance/fileintegrity_config.json

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
0 Kudos
SMcT
Enthusiast
Enthusiast

Thanks for taking the time to assist 

0 Kudos
Ajay1988
VMware Employee
VMware Employee

fileintegrity.log

2021-05-10T13:00:01.675Z ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-10T13:00:01.675Z ERROR: Exception occurred during check functionality. Exception: Failed verify config file
2021-05-10T13:00:01.675Z ERROR: Exception in IntegrityChecker, [Errno 2] No such file or directory: '/var/vmware/applmgmt/fileintegrity/integrity_check.sqlite_bkp'

 

Also the  applmgmt.log has rolled over .. Need the log which covers the below timestamp.

2021-05-06T12:49:38.443 [20210506-124823-17694817] [MainProcess:PID-43465] [integrity_checker::log_and_print:integrity_checker.py:75] ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-06T12:49:38.444 [20210506-124823-17694817] [MainProcess:PID-43465] [BackupManager::CheckFileIntegrity:BackupManager.py:420] ERROR: Fileintegrity [FILE_INTEGRITY_CHECK] Overall status: Failed. Check /var/log/vmware/cloudvm/fileintegrity.log for more details.

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
0 Kudos
SMcT
Enthusiast
Enthusiast
 
0 Kudos
SMcT
Enthusiast
Enthusiast

Here is the snippet from the time stamp you mentioned.

 

2021-05-06T12:49:37.227 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:37.228 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:37.229 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:37.230 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:38.264 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:38.264 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:38.264 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:38.265 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:39.290 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:39.291 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:39.292 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:39.293 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

0 Kudos
SMcT
Enthusiast
Enthusiast

Inadvertently fixed this by restoring the vcsa from file backup for another issue.  The backup I restored was taken when the issue existed.  I'm assuming the recreation of the appliance removed an issue, that wasn't something present in the backup.

View solution in original post

0 Kudos