Dear All
I found that the inventory service is not starting up and look up the log in C:\ProgramData\VMware\vCenterServer\logs\vpostgres
The DB status is always in "starting up" with FATAL. Does anyone know how to fix it? Please find attached logs.
2019-10-08 14:28:33.079 HKT 5d9c2c91.a0c 0 VCDB vc FATAL: the database system is starting up
2019-10-08 14:29:48.425 HKT 5d9c0db9.2104 0 LOG: received fast shutdown request
2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0 LOG: database system was interrupted while in recovery at 2019-10-08 12:16:58 HKT
2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0 HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
2019-10-08 15:10:02.234 HKT 5d9c361c.2790 0 LOG: database system was not properly shut down; automatic recovery in progress
2019-10-08 15:10:02.241 HKT 5d9c361c.2790 0 LOG: redo starts at 41/83FE29B8
2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0 LOG: redo done at 41/85FFF6D8
2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0 LOG: last completed transaction was at log time 2019-10-06 05:45:05.252+08
2019-10-08 15:20:42.924 HKT 5d9c361c.2370 0 LOG: received fast shutdown request
2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0 LOG: database system was interrupted while in recovery at 2019-10-08 15:10:02 HKT
2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0 HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
2019-10-08 15:25:09.850 HKT 5d9c39d1.1870 0 LOG: database system was not properly shut down; automatic recovery in progress
2019-10-08 15:25:09.874 HKT 5d9c39d1.1870 0 LOG: redo starts at 41/83FE29B8
2019-10-08 15:25:14.612 HKT 5d9c39d1.1870 0 LOG: redo done at 41/85FFF6D8
2019-10-08 15:25:14.618 HKT 5d9c39d1.1870 0 LOG: last completed transaction was at log time 2019-10-06 05:45:05.252+08
2019-10-08 15:25:25.159 HKT 5d9c39e5.1ad4 0 VCDB vc FATAL: the database system is starting up
2019-10-08 15:36:04.705 HKT 5d9c3c64.8b8 0 VCDB vc FATAL: the database system is starting up
2019-10-08 15:36:59.679 HKT 5d9c3c9b.1d8c 0 VCDB vc FATAL: the database system is starting up
2019-10-08 15:38:25.192 HKT 5d9c3cf1.270 0 VCDB vc FATAL: the database system is starting up
2019-10-08 15:42:38.272 HKT 5d9c3dee.1648 0 VCDB vc FATAL: the database system is starting up
2019-10-08 15:43:14.428 HKT 5d9c3e12.1f4c 0 VCDB vc FATAL: the database system is starting up
I am using Vcenter in windows server, i also got below log. is that because of no space?
2019-10-05 01:08:52.768 HKT 5d3827d0.1bf4 0 LOG: using stale statistics instead of current ones because stats collector is not responding
2019-10-05 01:09:00.404 HKT 5d3827d0.1810 0 LOG: could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device
2019-10-05 01:09:07.786 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:07.895 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device
2019-10-05 01:09:07.897 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:15.286 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:15.402 HKT 5d3827d0.1810 0 LOG: could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device
2019-10-05 01:09:15.404 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:22.792 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:22.900 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device
2019-10-05 01:09:22.901 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:30.286 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:30.398 HKT 5d3827d0.1810 0 LOG: could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device
2019-10-05 01:09:30.400 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:37.794 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:37.907 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device
2019-10-05 01:09:37.909 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:39.328 HKT 5d3827d0.1818 0 WARNING: Failed to rename health status temp file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml.tmp to file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml, errno=13
2019-10-05 01:09:45.293 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:45.410 HKT 5d3827d0.1810 0 LOG: could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device
2019-10-05 01:09:45.412 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:52.792 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:52.901 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device
2019-10-05 01:09:52.903 HKT 5d3827d0.1810 0 LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2019-10-05 01:09:59.757 HKT 5d975065.38d20 29430960 VCDB vc ERROR: could not extend file "base/16384/16666": No space left on device
vCenter server 6.0 service crashes due to postgres database filling up OS Partition. - vSphere Arena
Thanks for your reply
Here i got the error after i ran the command
C:\Program Files\VMware\vCenter Server\vPostgres\bin>psql -U vc VCDB
psql: FATAL: the database system is starting up
Does the drive on which vCenter server is installed on has enough space??
yes, it has more than 20GB space in C drive
I have a vCenter (on Windows) server with the same issue. The OS volume has 6GB free, but I had the same free space errors in the logs starting on 9/24, and when the server rebooted for automatic updates on 10/14 it effectively broke the DB. I see the same "VCDB vc FATAL: the database system is starting up" error in the logs since then and I cannot get services to start. Did you find a resolution for your server?