5 Replies Latest reply on Oct 10, 2019 2:31 AM by Vincentl2

    Vcenter 6.0 - VCDB VC FATAL:  the database system is starting up

    Vincentl2 Lurker

      Dear All

       

      I found that the inventory service is not starting up and look up the log in C:\ProgramData\VMware\vCenterServer\logs\vpostgres

      The DB status is always in "starting up" with FATAL. Does anyone know how to fix it? Please find attached logs.

       

      2019-10-08 14:28:33.079 HKT 5d9c2c91.a0c 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 14:29:48.425 HKT 5d9c0db9.2104 0   LOG:  received fast shutdown request

      2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0   LOG:  database system was interrupted while in recovery at 2019-10-08 12:16:58 HKT

      2019-10-08 15:09:16.702 HKT 5d9c361c.2790 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

      2019-10-08 15:10:02.234 HKT 5d9c361c.2790 0   LOG:  database system was not properly shut down; automatic recovery in progress

      2019-10-08 15:10:02.241 HKT 5d9c361c.2790 0   LOG:  redo starts at 41/83FE29B8

      2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0   LOG:  redo done at 41/85FFF6D8

      2019-10-08 15:10:05.853 HKT 5d9c361c.2790 0   LOG:  last completed transaction was at log time 2019-10-06 05:45:05.252+08

      2019-10-08 15:20:42.924 HKT 5d9c361c.2370 0   LOG:  received fast shutdown request

      2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0   LOG:  database system was interrupted while in recovery at 2019-10-08 15:10:02 HKT

      2019-10-08 15:25:05.862 HKT 5d9c39d1.1870 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

      2019-10-08 15:25:09.850 HKT 5d9c39d1.1870 0   LOG:  database system was not properly shut down; automatic recovery in progress

      2019-10-08 15:25:09.874 HKT 5d9c39d1.1870 0   LOG:  redo starts at 41/83FE29B8

      2019-10-08 15:25:14.612 HKT 5d9c39d1.1870 0   LOG:  redo done at 41/85FFF6D8

      2019-10-08 15:25:14.618 HKT 5d9c39d1.1870 0   LOG:  last completed transaction was at log time 2019-10-06 05:45:05.252+08

      2019-10-08 15:25:25.159 HKT 5d9c39e5.1ad4 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 15:36:04.705 HKT 5d9c3c64.8b8 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 15:36:59.679 HKT 5d9c3c9b.1d8c 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 15:38:25.192 HKT 5d9c3cf1.270 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 15:42:38.272 HKT 5d9c3dee.1648 0 VCDB vc FATAL:  the database system is starting up

      2019-10-08 15:43:14.428 HKT 5d9c3e12.1f4c 0 VCDB vc FATAL:  the database system is starting up

        • 1. Re: Vcenter 6.0 - VCDB VC FATAL:  the database system is starting up
          Vincentl2 Lurker

          I am using Vcenter in windows server, i also got below log. is that because of no space?

           

          2019-10-05 01:08:52.768 HKT 5d3827d0.1bf4 0   LOG:  using stale statistics instead of current ones because stats collector is not responding

          2019-10-05 01:09:00.404 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

          2019-10-05 01:09:07.786 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:07.895 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

          2019-10-05 01:09:07.897 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:15.286 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:15.402 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

          2019-10-05 01:09:15.404 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:22.792 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:22.900 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

          2019-10-05 01:09:22.901 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:30.286 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:30.398 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

          2019-10-05 01:09:30.400 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:37.794 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:37.907 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

          2019-10-05 01:09:37.909 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:39.328 HKT 5d3827d0.1818 0   WARNING:  Failed to rename health status temp file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml.tmp to file C:/ProgramData/VMware/vCenterServer/cfg/sca/health/vmware-postgres-health-status.xml, errno=13

          2019-10-05 01:09:45.293 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:45.410 HKT 5d3827d0.1810 0   LOG:  could not write temporary statistics file "pg_stat_tmp/db_16384.tmp": No space left on device

          2019-10-05 01:09:45.412 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:52.792 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:52.901 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_12029.tmp": No space left on device

          2019-10-05 01:09:52.903 HKT 5d3827d0.1810 0   LOG:  could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device

          2019-10-05 01:09:59.757 HKT 5d975065.38d20 29430960 VCDB vc ERROR:  could not extend file "base/16384/16666": No space left on device

          • 3. Re: Vcenter 6.0 - VCDB VC FATAL:  the database system is starting up
            Vincentl2 Lurker

            Thanks for your reply

            Here i got the error after i ran the command

             

            C:\Program Files\VMware\vCenter Server\vPostgres\bin>psql -U vc VCDB

            psql: FATAL:  the database system is starting up

            • 4. Re: Vcenter 6.0 - VCDB VC FATAL:  the database system is starting up
              Vijay2027 Expert
              vExpert

              Does the drive on which vCenter server is installed on has enough space??

              • 5. Re: Vcenter 6.0 - VCDB VC FATAL:  the database system is starting up
                Vincentl2 Lurker

                yes, it has more than 20GB space in C drive