Restore of VCSA 7 Update 1 to 7 keep failing

[gbengaoshinubi]

My attempted restore of VCSA 7 Update 1 back to VCSA 7 is failing, I am restoring back to 7.0 because vCLS VMs are failing to start , a feature that came with update 1, this is because I have not enabled EVC on my cluster

RestoreManager encountered an exception. See logs for details. https://x.x.x.x:5480/vami/backend/vc-support.cgi

[scott28tt]

And what is in the logs mentioned? Perhaps add to this thread using the Attach function in the bottom-right of the post creator/editor.

[gbengaoshinubi]

HI Scott28tt, thanks for your response, please see attached logs

[scott28tt]

What errors did you see when you looked through the logs for the period when you were executing the restore?

[gbengaoshinubi]

[postgres_utils::StartPostgresServer:postgres_utils.py:309] INFO: Starting Postgres server with timeout 300 seconds.

2020-10-08T22:11:57.833 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [Proc::RunCmd:Proc.py:415] INFO: Executing command ['su', '-s', '/bin/bash', '-', 'vtsdbuser', '-c', "LD_LIBRARY_PATH=/opt/vmware/vpostgres/current/lib-private /opt/vmware/vpostgres/current/bin/pg_ctl start -w -D '/storage/vtsdb/vtsdbdata' --core-files -l /var/log/vmware/vtsdb/serverlog -t 300"].

2020-10-08T22:11:58.383 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [Log::run:Log.py:64] INFO: waiting for server to start.... stopped waiting

2020-10-08T22:11:58.383 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [Log::run:Log.py:64] ERROR: pg_ctl: could not start server

2020-10-08T22:11:58.383 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [Log::run:Log.py:64] ERROR: Examine the log output.

2020-10-08T22:11:58.384 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [Proc::RunCmd:Proc.py:437] INFO: Command "['su', '-s', '/bin/bash', '-', 'vtsdbuser', '-c', "LD_LIBRARY_PATH=/opt/vmware/vpostgres/current/lib-private /opt/vmware/vpostgres/current/bin/pg_ctl start -w -D '/storage/vtsdb/vtsdbdata' --core-files -l /var/log/vmware/vtsdb/serverlog -t 300"]" return non-zero exit status 1.

Failed to start postgres server.

2020-10-08T22:11:58.384 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [VTSDB::RestoreVTSDB:VTSDB.py:293] ERROR: Error with VTSDB restore commands execution.

Traceback (most recent call last):

  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VTSDB.py", line 261, in RestoreVTSDB

    component_name = "VTSDB"))

  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VTSDB.py", line 389, in _restore_vtsdb

    config['VTSDB_LOG'])

  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/postgres_utils.py", line 318, in StartPostgresServer

    RunCmd(logger, cmd, err_msg='Failed to start postgres server.')

  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 440, in RunCmd

    output=err_msg)

subprocess.CalledProcessError: Command '['su', '-s', '/bin/bash', '-', 'vtsdbuser', '-c', "LD_LIBRARY_PATH=/opt/vmware/vpostgres/current/lib-private /opt/vmware/vpostgres/current/bin/pg_ctl start -w -D '/storage/vtsdb/vtsdbdata' --core-files -l /var/log/vmware/vtsdb/serverlog -t 300"]' returned non-zero exit status 1.

2020-10-08T22:11:58.386 [20201008-221148-16749653] [VTSDBRestore:PID-29442] [VTSDB::RestoreVTSDBCleanup:VTSDB.py:314] INFO: Successfully cleaned up for VTSDB restore.

2020-10-08T22:11:59.35 [20201008-221148-16749653] [MainProcess:PID-29164] [Proc::VerifyProcStatusAndGetArchive:Proc.py:157] ERROR: Error at process VTSDBRestore; rc:1.

2020-10-08T22:11:59.36 [20201008-221148-16749653] [MainProcess:PID-29164] [Proc::VerifyProcStatusAndGetArchive:Proc.py:161] ERROR: stderr:Failed to start postgres server.

2020-10-08T22:11:59.36 [20201008-221148-16749653] [MainProcess:PID-29164] [Proc::VerifyProcStatusAndGetArchive:Proc.py:171] INFO: Following error message isn't localized:

  stderr:Failed to start postgres server.

2020-10-08T22:11:59.36 [20201008-221148-16749653] [MainProcess:PID-29164] [RestoreManager::main:RestoreManager.py:523] ERROR: RestoreManager encountered an exception: Hit exception inside process VTSDBRestore:

2020-10-08T22:11:59.66 [20201008-221148-16749653] [MainProcess:PID-29164] [RestoreManager::HandleRestoreCleanup:RestoreManager.py:373] INFO: Restore cleanup complete.

2020-10-08T22:11:59.70 [20201008-221148-16749653] [MainProcess:PID-29164] [Common::RestoreExitCleanup:Common.py:373] INFO: Wait 10 seconds before restore cleanup

2020-10-08T22:12:09.71 [20201008-221148-16749653] [MainProcess:PID-29164] [Proc::RunCmdForOutput:Proc.py:531] INFO: Executing command: ['systemctl', 'restart', 'vami-lighttp.service'].

2020-10-08T22:12:09.294 [20201008-221148-16749653] [MainProcess:PID-29164] [Common::RestoreExitCleanup:Common.py:387] INFO: No temp network to cleanup

2020-10-08T22:12:09.295 [20201008-221148-16749653] [MainProcess:PID-29164] [RestoreManager::main:RestoreManager.py:568] INFO: Restore job failed

[gbengaoshinubi]

I noticed something now, that the serverlog file is missing inside /var/log/vmware/vtsdb/, could that be the issue

[kgorai]

hi,

I would like to now more about why the vCLS VMs were not starting. Are you getting insufficient resource tasks?

What kind ESX are there in your Cluster? Are they nested ESX or physical? AMD or Intel?

Can you please login into the ESX and try to power on from there? Please attach the screenshot of the error when you try to power on.

[gbengaoshinubi]

Hi Kgorai,

Well, we are currently using ESXi 6.7 U3 on 7 hosts in the cluster. The error that keeps coming is insufficient resources and this has been traced to the CPU. I have narrowed this down to the option of Monitor/Mwait on the BIOS of the server. This has been enabled onthe test environment and the vCLS worked, about to apply the same on the LIVE server.

Thanks for your response

[Anil0210]

Do you have a UI screenshot when it failed. Can you look if it has failed due to memory or disk. When you restore it will deploy witha certain deployment size, that size maybe less than what is your backup is. If you can share UI screenshot of error, it will help.

[msripada]

Do you see any similarities from this thread vCenter 7.0.0d Recovery 2nd stage errors

thanks,

MS

[RickVerstegen]

I assume you using file based backup. I doubt if you can do a restore to vcsa7 from a backup made with a newer version (vcsa7 u1)

Check note section in below link.

Stage 1 - Deploy a New Appliance

[gbengaoshinubi · Accepted solution]

Thank you all for the assistance, I have been able to get the vCLS to work, so I do not need to restore vCenter 7.1 to 7.0 again