Postgres Database Corrupted?

[DanMc85]

After a storage power issue, vCenter now has a corrupted database.  

I am looking for any guidance if this is a recoverable issue or not.  Any help or database commands I should run would be appreciated. 

Thanks!

vCenter shows

Could not connect to one or more vCenter Server systems:https://FQDN:443/sdk

and You have no privileges to view this object or it does not exist.

vpxd.log:

cat /var/log/vmware/vpxd/vpxd.log |grep "error vpxd"
2022-02-12T15:51:36.849-05:00 error vpxd[06243] [Originator@6876 sub=vpxdVdb] Database version will not be set in vpxd.cfg
2022-02-12T15:51:36.926-05:00 error vpxd[06243] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(Could not find file : /etc/motd)
2022-02-12T15:51:36.936-05:00 error vpxd[06243] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: ads.checkInterval
2022-02-12T15:51:36.936-05:00 error vpxd[06243] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: ads.checkIntervalEnabled
2022-02-12T15:51:36.940-05:00 error vpxd[06243] [Originator@6876 sub=MoOptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: Fault cause: vmodl.fault.InvalidArgument
Binary file (standard input) matches

2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] Execute result code: -1
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed: SELECT ESX_AGENT_VM_STATE,LOCAL_FILE_NAME,AGENT_CNX_STATE,AGENT_ID,IS_TEMPLATE,DISABLED_SNAPSHOT,MEM_SIZE_MB,NUM_VCPU,POWER_STATE,RESOURCE_GROUP_ID,HOST_ID,DATACENTER_ID,ID,FILE_NAME FROM VPX_VM
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] Execution elapsed time: 16 ms
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] Statement diagnostic data from driver is XX000:0:1:ERROR: MultiXactId 2877285 has not been created yet -- apparent wraparound;
--> Error while executing the query
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX000:1
2022-02-12T15:59:58.838-05:00 error vpxd[06141] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: MultiXactId 2877285 has not been created yet -- apparent wraparound;
--> Error while executing the query" is returned when executing SQL statement "SELECT ESX_AGENT_VM_STATE,LOCAL_FILE_NAME,AGENT_CNX_STATE,AGENT_ID,IS_TEMPLATE,DISABLED_SNAPSHOT,MEM_SIZE_MB,NUM_VCPU,POWER_STATE,RESOURCE_GROUP_ID,HOST_ID,DATACENTER_ID,ID,FILE_NAME FROM VPX_VM"
2022-02-12T15:59:58.839-05:00 error vpxd[06141] [Originator@6876 sub=Main] Init failed. VdbError: Error[VdbODBCError] (-1) "ODBC error: (XX000) - ERROR: MultiXactId 2877285 has not been created yet -- apparent wraparound;
--> Error while executing the query" is returned when executing SQL statement "SELECT ESX_AGENT_VM_STATE,LOCAL_FILE_NAME,AGENT_CNX_STATE,AGENT_ID,IS_TEMPLATE,DISABLED_SNAPSHOT,MEM_SIZE_MB,NUM_VCPU,POWER_STATE,RESOURCE_GROUP_ID,HOST_ID,DATACENTER_ID,ID,FILE_NAME FROM VPX_VM"
--> Backtrace:
--> Backtrace:
--> Backtrace[0] 00007fff617c09b0 rip=000055db23e6cfcc rbx=00007f0c4804d920 rbp=00007fff617c0ad8 r12=00007fff617c09b8 r13=00007fff617c0ae8 r14=00007fff617c0af0 r15=00000000ffffffff
--> Backtrace[1] 00007fff617c09f0 rip=000055db23e6d8db rbx=00007f0c4804d920 rbp=00007fff617c0ad8 r12=00007fff617c0ad8 r13=00007fff617c0b08 r14=000055db26bbea00 r15=00000000ffffffff
--> Backtrace[2] 00007fff617c0a40 rip=000055db23e6d9d5 rbx=0000000000000000 rbp=00007f0c48090f40 r12=00007fff617c0ad8 r13=00007fff617c0b08 r14=000055db26bbea00 r15=00000000ffffffff
--> Backtrace[3] 00007fff617c0a60 rip=000055db23e77316 rbx=0000000000000000 rbp=00007f0c48090f30 r12=00007f0c4804d920 r13=00007fff617c0b10 r14=000055db26bbea00 r15=00000000ffffffff
--> Backtrace[4] 00007fff617c0bc0 rip=000055db23e6b832 rbx=000000000000000e rbp=00007fff617c0d38 r12=00007fff617c0be8 r13=00007fff617c0bd8 r14=0000000000000000 r15=00007f0c48090f30
--> Backtrace[5] 00007fff617c0c30 rip=000055db230747ef rbx=000055db26b9b620 rbp=00007fff617c0df0 r12=00007f0c345c1d70 r13=00007fff617c0d38 r14=00007fff617c0cc8 r15=00007fff617c0d70
--> Backtrace[6] 00007fff617c0e00 rip=000055db22e5dd8c rbx=000055db24152f89 rbp=00007fff617c0eb8 r12=00000000617c0e01 r13=00007fff617c0fa8 r14=00007fff617c0f70 r15=00007fff617c0fa8
--> Backtrace[7] 00007fff617c1050 rip=000055db22e61d6d rbx=00007fff617c1278 rbp=000055db25fe3a80 r12=00007fff617c1078 r13=000055db26b5c340 r14=000055db24123ef0 r15=0000000000000001
--> Backtrace[8] 00007fff617c10e0 rip=000055db22e5758d rbx=000055db2493b6f0 rbp=000055db25fe3a80 r12=00007fff617c1108 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000
--> Backtrace[9] 00007fff617c11a0 rip=00007f0e7b8a2b87 rbx=0000000000000000 rbp=000055db23f26d30 r12=000055db22e56b48 r13=00007fff617c1270 r14=0000000000000000 r15=0000000000000000
--> Backtrace[10] 00007fff617c1260 rip=000055db22e56b71 rbx=0000000000000000 rbp=0000000000000000 r12=000055db22e56b48 r13=00007fff617c1270 r14=0000000000000000 r15=0000000000000000
--> SymBacktrace[0] 00007fff617c09b0 rip=000055db23e6cfcc in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[1] 00007fff617c09f0 rip=000055db23e6d8db in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[2] 00007fff617c0a40 rip=000055db23e6d9d5 in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[3] 00007fff617c0a60 rip=000055db23e77316 in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[4] 00007fff617c0bc0 rip=000055db23e6b832 in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[5] 00007fff617c0c30 rip=000055db230747ef in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[6] 00007fff617c0e00 rip=000055db22e5dd8c in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[7] 00007fff617c1050 rip=000055db22e61d6d in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[8] 00007fff617c10e0 rip=000055db22e5758d in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
--> SymBacktrace[9] 00007fff617c11a0 rip=00007f0e7b8a2b87 in function __libc_start_main in object /lib/libc.so.6 loaded at 00007f0e7b880000
--> SymBacktrace[10] 00007fff617c1260 rip=000055db22e56b71 in function (null) in object /usr/lib/vmware-vpx/vpxd loaded at 000055db2275b000
-->
2022-02-12T15:59:58.839-05:00 warning vpxd[06141] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3864 ms
2022-02-12T15:59:58.839-05:00 error vpxd[06141] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down

This is the Postgres log:

2022-02-12 20:50:25.829 UTC 62081d91.82c 0 LOG: starting PostgreSQL 13.3 (VMware Postgres 13.3.0-18202630 release) on x86_64-pc-linux-gnu, compiled by x86_64-vmk-linux-gnu-gcc (GCC) 6.4.0, 64-bit
2022-02-12 20:50:25.829 UTC 62081d91.82c 0 LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-02-12 20:50:25.829 UTC 62081d91.82c 0 LOG: listening on IPv6 address "::", port 5432
2022-02-12 20:50:25.831 UTC 62081d91.82c 0 LOG: listening on Unix socket "/var/run/vpostgres/.s.PGSQL.5432"
2022-02-12 20:50:25.871 UTC 62081d91.860 0 LOG: database system was shut down at 2022-02-12 20:46:15 UTC
2022-02-12 20:50:25.891 UTC 62081d91.865 0 LOG: corrupted statistics file "pg_stat/global.stat"
2022-02-12 20:50:25.892 UTC 62081d91.82c 0 LOG: database system is ready to accept connections
2022-02-12 20:50:25.892 UTC 62081d91.866 0 LOG: Initialized health status worker.
2022-02-12 20:50:25.919 UTC 62081d91.866 0 LOG: vmware-vpostgres service is running in FIPS mode.
2022-02-12 20:50:25.920 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:50:25.920 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 4552 kB, mem_avail = 2277376 kB
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Updated instance status successfully.
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:50:25.974 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:50:25.975 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 4816 kB, mem_avail = 2277376 kB
2022-02-12 20:50:25.975 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:50:25.975 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:50:25.975 UTC 62081d91.866 0 LOG: Updated instance status successfully.
2022-02-12 20:50:26.920 UTC 62081d92.8ca 0 archiver archiver FATAL: password authentication failed for user "archiver"
2022-02-12 20:50:26.920 UTC 62081d92.8ca 0 archiver archiver DETAIL: User "archiver" has no password assigned.
Connection matched pg_hba.conf line 7: "local all all md5"
2022-02-12 20:50:26.920 UTC 62081d92.8ca 0 archiver archiver LOG: could not send data to client: Broken pipe
2022-02-12 20:50:26.949 UTC 62081d92.8d6 0 [unknown] archiver ERROR: replication slot "vpg_archiver" already exists
2022-02-12 20:50:31.213 UTC 62081d97.a48 0 root root FATAL: password authentication failed for user "root"
2022-02-12 20:50:31.213 UTC 62081d97.a48 0 root root DETAIL: Role "root" does not exist.
Connection matched pg_hba.conf line 7: "local all all md5"
2022-02-12 20:50:31.213 UTC 62081d97.a48 0 root root LOG: could not send data to client: Broken pipe
2022-02-12 20:50:31.216 UTC 62081d97.a4c 0 root root FATAL: password authentication failed for user "root"
2022-02-12 20:50:31.216 UTC 62081d97.a4c 0 root root DETAIL: Role "root" does not exist.
Connection matched pg_hba.conf line 7: "local all all md5"
2022-02-12 20:50:31.216 UTC 62081d97.a4c 0 root root LOG: could not send data to client: Broken pipe
2022-02-12 20:50:31.236 UTC 62081d97.a5b 0 root root FATAL: password authentication failed for user "root"
2022-02-12 20:50:31.236 UTC 62081d97.a5b 0 root root DETAIL: Role "root" does not exist.
Connection matched pg_hba.conf line 7: "local all all md5"
2022-02-12 20:50:31.236 UTC 62081d97.a5b 0 root root LOG: could not send data to client: Broken pipe
2022-02-12 20:50:32.030 UTC 62081d97.a68 0 VCDB postgres ERROR: index "pk_vpx_hist_stat1_62" contains unexpected zero page at block 3
2022-02-12 20:50:32.030 UTC 62081d97.a68 0 VCDB postgres HINT: Please REINDEX it.
2022-02-12 20:50:32.030 UTC 62081d97.a68 0 VCDB postgres STATEMENT: select stat_val from vpx_hist_stat1 where counter_id in (select counter_id from vpx_stat_counter where stat_id in (select id from VPX_STAT_DEF where name='sessionstats') and device_id in (select device_id from vpx_device where device_name='/SessionStats/SessionPool/NumSessions/Total')) order by time_id desc limit 1;
2022-02-12 20:50:32.192 UTC 62081d98.a9d 0 VCDB postgres ERROR: index "pk_vpx_hist_stat1_62" contains unexpected zero page at block 3
2022-02-12 20:50:32.192 UTC 62081d98.a9d 0 VCDB postgres HINT: Please REINDEX it.
2022-02-12 20:50:32.192 UTC 62081d98.a9d 0 VCDB postgres STATEMENT: select regexp_replace(regexp_replace (regexp_replace (vd.device_name, '(.*)(/Username='')(.*)(''/)(ClientIP='')(.*)(''.*)', '\3|-ClientIP-\6'), '(.*\\)(.*)', '\1_SLASH_\2'),'(.*)(@)(.*)', '\1_AT_\3') as username, vhs.stat_val from vpx_hist_stat1 vhs, vpx_device vd, vpx_stat_counter vsc where vsc.device_id = vd.device_id and vd.device_name LIKE '%Username%' and vhs.counter_id = vsc.counter_id and vsc.stat_id in (select id from VPX_STAT_DEF where name='sessionstats') order by time_id desc limit 1;
2022-02-12 20:50:32.532 UTC 62081d97.a5f 0 VCDB postgres LOG: duration: 1273.832 ms statement: select event_type,create_time from vpxv_event_all where event_type IN ('com.vmware.sso.STSCertExpiry', 'com.vmware.vc.vecs.CertExpirationEvent', 'com.vmware.vc.certmgr.HostCertExpiringEvent', 'com.vmware.vc.certmgr.HostCertExpiringShortlyEvent', 'com.vmware.vc.certmgr.HostCertExpirationImminentEvent') AND create_time > now() - interval '1 day';
2022-02-12 20:50:32.533 UTC 62081d97.a60 0 VCDB postgres LOG: duration: 1274.627 ms statement: select event_type,create_time from vpxv_event_all where event_type IN ('com.vmware.license.LicenseExpiryEvent', 'com.vmware.license.vsan.LicenseExpiryEvent', 'com.vmware.license.vsan.SubscriptionLicenseExpiredEvent', 'com.vmware.license.VcLicenseExpiredEvent', 'com.vmware.license.VcSubscriptionLicenseExpiredEvent', 'com.vmware.license.HostLicenseExpiredEvent', 'com.vmware.license.HostSubscriptionLicenseExpiredEvent') AND create_time > now() - interval '1 day';
2022-02-12 20:50:56.003 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:50:56.003 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:50:56.004 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 8496 kB, mem_avail = 2277376 kB
2022-02-12 20:50:56.005 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:50:56.005 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:50:56.005 UTC 62081d91.866 0 LOG: Updated instance status successfully.
2022-02-12 20:51:23.536 UTC 62081dcb.13be 45830229 VCDB vc ERROR: function replace(jsonb, unknown, unknown) does not exist at character 81
2022-02-12 20:51:23.536 UTC 62081dcb.13be 45830229 VCDB vc HINT: No function matches the given name and argument types. You might need to add explicit type casts.
2022-02-12 20:51:23.536 UTC 62081dcb.13be 45830229 VCDB vc STATEMENT: UPDATE CIS_KV_KEYVALUE SET KV_VALUE_TMP = CONCAT('{"value":"', REPLACE(REPLACE(REPLACE(KV_VALUE, '\' , '\\'),'''', '\'''),'"', '\"'), '"}')::jsonb
2022-02-12 20:51:23.556 UTC 62081dcb.13be 0 VCDB vc ERROR: relation "gin_kv_value" already exists
2022-02-12 20:51:23.556 UTC 62081dcb.13be 0 VCDB vc STATEMENT: CREATE INDEX GIN_KV_VALUE ON CIS_KV_KEYVALUE USING gin (KV_VALUE)
2022-02-12 20:51:26.033 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:51:26.033 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:51:26.035 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 11976 kB, mem_avail = 2277376 kB
2022-02-12 20:51:26.035 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:51:26.036 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:51:26.036 UTC 62081d91.866 0 LOG: Updated instance status successfully.
2022-02-12 20:51:40.785 UTC 62081dd8.1969 0 VCDB vc ERROR: MultiXactId 2877285 has not been created yet -- apparent wraparound
2022-02-12 20:51:40.785 UTC 62081dd8.1969 0 VCDB vc STATEMENT: BEGIN;declare "SQL_CUR0x7f1ac0018830" cursor with hold for SELECT ESX_AGENT_VM_STATE,LOCAL_FILE_NAME,AGENT_CNX_STATE,AGENT_ID,IS_TEMPLATE,DISABLED_SNAPSHOT,MEM_SIZE_MB,NUM_VCPU,POWER_STATE,RESOURCE_GROUP_ID,HOST_ID,DATACENTER_ID,ID,FILE_NAME FROM VPX_VM;fetch 1024 in "SQL_CUR0x7f1ac0018830"
2022-02-12 20:51:56.063 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:51:56.063 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:51:56.065 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 33056 kB, mem_avail = 2277376 kB
2022-02-12 20:51:56.065 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:51:56.065 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:51:56.065 UTC 62081d91.866 0 LOG: Updated instance status successfully.
2022-02-12 20:52:26.074 UTC 62081d91.866 0 LOG: Updating instance status...
2022-02-12 20:52:26.074 UTC 62081d91.866 0 LOG: Checking bloat of WAL caused by replication slots
2022-02-12 20:52:26.076 UTC 62081d91.866 0 LOG: Memory check: flag = rss, mem_used_bytes = 32620 kB, mem_avail = 2277376 kB
2022-02-12 20:52:26.076 UTC 62081d91.866 0 LOG: Writing instance status...
2022-02-12 20:52:26.076 UTC 62081d91.866 0 LOG: Wrote instance status successfully.
2022-02-12 20:52:26.076 UTC 62081d91.866 0 LOG: Updated instance status successfully.

[Ajay1988]

vpx_vm table has bad entry. Better to restore back to snapshot or backup. 
How many hosts and VM's approx you have?