VMware Cloud Community
SMcT
Enthusiast
Enthusiast
Jump to solution

Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state.

Hi All

My VCSA backups have started failing recently with the following error in the VAMI - Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state. 

I'm not seeing any function problems with the operation of my vcsa and can backup successfully when I unselect the DB checks.  I haven't found anything compelling in the log files either. 

Can anyone direct me to a specific log file or provide any thoughts?  I'm either not searching for the right thing on Google or there isn't much out there on this...

VCSA 7.0.2 - Build 17694817

(Its a lab so no support)

Thanks 

Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
1 Solution

Accepted Solutions
SMcT
Enthusiast
Enthusiast
Jump to solution

Inadvertently fixed this by restoring the vcsa from file backup for another issue.  The backup I restored was taken when the issue existed.  I'm assuming the recreation of the appliance removed an issue, that wasn't something present in the backup.

Blog: stephanmctighe.com Twitter: @vStephanMcTighe

View solution in original post

Reply
0 Kudos
17 Replies
Ajay1988
Expert
Expert
Jump to solution

Use the script from  https://kb.vmware.com/s/article/53062   and check for any issues. 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
SMcT
Enthusiast
Enthusiast
Jump to solution

Hi Ajay

thanks for the imformation.  I ran the script and got the following output.

root@vm-vcsa-01 [ / ]# /scripts/vpgsql-consistency-check.sh -a -b -f /tmp/ -v
amcheck extension is not installed
amcheck extension is not installed

05-10-2021T10:48:15 Starting the catalog diagnostic
05-10-2021T10:48:16 Successfully finished the catalog diagnostic

Taking a backup of the VCDB to: /tmp/

05-10-2021T10:48:16 Starting backup of the VCDB database.
05-10-2021T10:48:23 Successfully finished backup of the VCDB database.

05-10-2021T10:48:23 Starting backup of the AFD database.
05-10-2021T10:48:24 Successfully finished backup of the AFD database.

05-10-2021T10:48:24 Starting PSQL Instance Vacuum
vCenter is currently running.
05-10-2021T10:48:46 Finished PSQL Instance Vacuum

05-10-2021T10:48:46 Starting Consistency Check
05-10-2021T10:48:54 Finished Consistency Check.

Consistency Check PASSED. The database has been found to be consistent. No further action required.

However I am still getting the error when backing up the DB via the VAMI.  I have attached the log file which mentions 3 circular foreign-key constraints on the following 3 tables - vpx_task, vpx_entity, pm_config_commits.

Thanks 

Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Tags (1)
Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Attach the  backup log from  /var/log/vmware/applmgmt

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
SMcT
Enthusiast
Enthusiast
Jump to solution

 
Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

2021-05-10T09:50:35.541 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::getDbHealthStatus:BackupManager.py:480] INFO: getDbHealthStatus returns status as UNHEALTHY
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::HealthCheckStatus:BackupManager.py:493] ERROR: HealthCheckStatus Overall status: Failed. Check /var/log/vmware/applmgmt/applmgmt.log for more details.
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::main:BackupManager.py:625] ERROR: BackupManager encountered an exception: Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.

 

Please grab the   /var/log/vmware/applmgmt/applmgmt.log and   /var/log/vmware/cloudvm/fileintegrity.log  . Also   /etc/vmware/appliance/fileintegrity_config.json

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
SMcT
Enthusiast
Enthusiast
Jump to solution

Thanks for taking the time to assist 

Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

fileintegrity.log

2021-05-10T13:00:01.675Z ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-10T13:00:01.675Z ERROR: Exception occurred during check functionality. Exception: Failed verify config file
2021-05-10T13:00:01.675Z ERROR: Exception in IntegrityChecker, [Errno 2] No such file or directory: '/var/vmware/applmgmt/fileintegrity/integrity_check.sqlite_bkp'

 

Also the  applmgmt.log has rolled over .. Need the log which covers the below timestamp.

2021-05-06T12:49:38.443 [20210506-124823-17694817] [MainProcess:PID-43465] [integrity_checker::log_and_print:integrity_checker.py:75] ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-06T12:49:38.444 [20210506-124823-17694817] [MainProcess:PID-43465] [BackupManager::CheckFileIntegrity:BackupManager.py:420] ERROR: Fileintegrity [FILE_INTEGRITY_CHECK] Overall status: Failed. Check /var/log/vmware/cloudvm/fileintegrity.log for more details.

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
SMcT
Enthusiast
Enthusiast
Jump to solution

 
Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
SMcT
Enthusiast
Enthusiast
Jump to solution

Here is the snippet from the time stamp you mentioned.

 

2021-05-06T12:49:37.227 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:37.228 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:37.229 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:37.230 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:38.264 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:38.264 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:38.264 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:38.265 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:39.290 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:39.291 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:39.292 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:39.293 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
SMcT
Enthusiast
Enthusiast
Jump to solution

Inadvertently fixed this by restoring the vcsa from file backup for another issue.  The backup I restored was taken when the issue existed.  I'm assuming the recreation of the appliance removed an issue, that wasn't something present in the backup.

Blog: stephanmctighe.com Twitter: @vStephanMcTighe
Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Hello. I encountered the same issue in my lab and noticed it was caused due to proxy. I have blogged about it here >> https://virtual-power.in/f/30-vami-backup-fails-on-vcsa-configured-with-proxy

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
BenLor00
Contributor
Contributor
Jump to solution

Hi SMcT,

Today I encountered the same error on two distinct vCenters 7.0U3a and I solved it following the workaround described in the article 86174

Ciao!

--
Lorenzo

Reply
0 Kudos
CLS78
Contributor
Contributor
Jump to solution

The same error occurred in 3 out 5 vSphere environments upgrading to vCSA 7.0 Update 3a.   The fix action is to run the following command via shell on the vCSA:  /usr/bin/dbcc -fbss embedded

This command refreshes the latest db schema on the vCSA.  The reason why this occurred is because there has been change in DB schema which is not refreshed post patching.

Reference URL:  https://vinfrastructure.it/2021/10/file-level-backup-issue-with-vcenter-7-0-3/

 

 

CLS78
Contributor
Contributor
Jump to solution

However, running the command only I see on 3 on my environment only resolves it for a day.  I wonder if anybody else is encountering the same issue?

Reply
0 Kudos
BenLor00
Contributor
Contributor
Jump to solution

Hi CLS78,

Yes, after a couple of days the issue returns. We just have to wait for the new patch, in the hope that the problem will be definitively resolved.

Lorenzo

 

Reply
0 Kudos
CLS78
Contributor
Contributor
Jump to solution

Lorenzo,

Thanks for the quick response.  I had the same idea,

 

Christian

Reply
0 Kudos
marsherian
Enthusiast
Enthusiast
Jump to solution

The script referenced in the original KB did not work for me, but this KB did:

https://kb.vmware.com/s/article/86084

Reply
0 Kudos