Vcenter 6.0 - VCDB VC FATAL: the database system is starting up

[Vincentl2]

Dear All

I found that the inventory service is not starting up and look up the log in C:\ProgramData\VMware\vCenterServer\logs\vpostgres

The DB status is always in "starting up" with FATAL. Does anyone know how to fix it? Please find attached logs.

2019-10-08 14:28:33.079 HKT 5d9c2c91.a0c 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 14:29:48.425 HKT 5d9c0db9.2104 0   LOG:  received fast shutdown request

2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0   LOG:  database system was interrupted while in recovery at 2019-10-08 12:16:58 HKT

2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2019-10-08 15:10:02.234 HKT 5d9c361c.2790 0   LOG:  database system was not properly shut down; automatic recovery in progress

2019-10-08 15:10:02.241 HKT 5d9c361c.2790 0   LOG:  redo starts at 41/83FE29B8

2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0   LOG:  redo done at 41/85FFF6D8

2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0   LOG:  last completed transaction was at log time 2019-10-06 05:45:05.252+08

2019-10-08 15:20:42.924 HKT 5d9c361c.2370 0   LOG:  received fast shutdown request

2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0   LOG:  database system was interrupted while in recovery at 2019-10-08 15:10:02 HKT

2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2019-10-08 15:25:09.850 HKT 5d9c39d1.1870 0   LOG:  database system was not properly shut down; automatic recovery in progress

2019-10-08 15:25:09.874 HKT 5d9c39d1.1870 0   LOG:  redo starts at 41/83FE29B8

2019-10-08 15:25:14.612 HKT 5d9c39d1.1870 0   LOG:  redo done at 41/85FFF6D8

2019-10-08 15:25:14.618 HKT 5d9c39d1.1870 0   LOG:  last completed transaction was at log time 2019-10-06 05:45:05.252+08

2019-10-08 15:25:25.159 HKT 5d9c39e5.1ad4 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 15:36:04.705 HKT 5d9c3c64.8b8 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 15:36:59.679 HKT 5d9c3c9b.1d8c 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 15:38:25.192 HKT 5d9c3cf1.270 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 15:42:38.272 HKT 5d9c3dee.1648 0 VCDB vc FATAL:  the database system is starting up

2019-10-08 15:43:14.428 HKT 5d9c3e12.1f4c 0 VCDB vc FATAL:  the database system is starting up

[Vincentl2]

I am using Vcenter in windows server, i also got below log. is that because of no space?

2019-10-05 01:08:52.768 HKT 5d3827d0.1bf4 0   LOG:  using stale statistics instead of current ones because stats collector is not responding

2019-10-05 01:09:00.404 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

2019-10-05 01:09:07.786 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:07.895 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

2019-10-05 01:09:07.897 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:15.286 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:15.402 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

2019-10-05 01:09:15.404 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:22.792 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:22.900 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

2019-10-05 01:09:22.901 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:30.286 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:30.398 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

2019-10-05 01:09:30.400 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:37.794 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:37.907 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

2019-10-05 01:09:37.909 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:39.328 HKT 5d3827d0.1818 0   WARNING:  Failed to rename health status temp file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml.tmp to file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml, errno=13

2019-10-05 01:09:45.293 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:45.410 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

2019-10-05 01:09:45.412 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:52.792 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:52.901 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

2019-10-05 01:09:52.903 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

2019-10-05 01:09:59.757 HKT 5d975065.38d20 29430960 VCDB vc ERROR:  could not extend file "base/16384/16666": No space left on device

[RajeevVCP4]

vCenter server 6.0 service crashes due to postgres database filling up OS Partition. - vSphere Arena

[Vincentl2]

Thanks for your reply

Here i got the error after i ran the command

C:\Program Files\VMware\vCenter Server\vPostgres\bin>psql -U vc VCDB

psql: FATAL:  the database system is starting up

[Vijay2027]

Does the drive on which vCenter server is installed on has enough space??

[Vincentl2]

yes, it has more than 20GB space in C drive

[Derekius]

I have a vCenter (on Windows) server with the same issue.  The OS volume has 6GB free, but I had the same free space errors in the logs starting on 9/24, and when the server rebooted for automatic updates on 10/14 it effectively broke the DB.  I see the same "VCDB vc FATAL: the database system is starting up" error in the logs since then and I cannot get services to start.  Did you find a resolution for your server?