vCenter 6.7 - vpxd will not start due to postgresql database error

[carboncopy]

Hello,

I have an instance of vCenter 6.7 appliance running on ESXi 6.7 as a VM. I recently rebooted the vCenter appliance and when it came back up, I was presented with the infamous "Could not connect to one or more vCenter Server systems: https://vcenter@domain.local:443/sdk" error.

I researched for hours only to find solutions to other problems (change root password, duplicate key in DB, expired SSL cert, etc.), which do not apply in my case. The error I am seeing in the postgresql-xx.log file looks like this:

  2019-01-09 22:46:48.058 UTC 5c3679d4.2c5e 1543434 VCDB vc STATEMENT:  INSERT INTO VPX_TASK (TASK_ID, DESCRIPTIONID, ENTITY_ID, ENTITY_TYPE, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID, DESCRIPTION) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19)
  ​
  2019-01-09 22:46:48.060 UTC 5c3679d7.2c64 1543436 VCDB vc ERROR:  right sibling's left-link doesn't match: block 9 links to 11 instead of expected 4 in index "vpxi_task_username

The error I am seeing in the vpxd.log file looks like this:

  2019-01-09T22:59:29.363Z error vpxd[04281] [Originator@6876 sub=Default opID=780d04c9] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: right sibling's left-link doesn't match: block 9 links to 11 instead of expected 4 in index "vpxi_task_username";
  --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_TASK (TASK_ID, DESCRIPTIONID, ENTITY_ID, ENTITY_TYPE, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID, DESCRIPTION) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
  2019-01-09T22:59:29.364Z error vpxd[04281] [Originator@6876 sub=vpxCommon opID=780d04c9] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
  2019-01-09T22:59:29.364Z error vpxd[04281] [Originator@6876 sub=Default opID=780d04c9] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (XX000) - ERROR: right sibling's left-link doesn't match: block 9 links to 11 instead of expected 4 in index "vpxi_task_username";
  --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_TASK (TASK_ID, DESCRIPTIONID, ENTITY_ID, ENTITY_TYPE, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID, DESCRIPTION) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
  2019-01-09T22:59:29.379Z panic vpxd[04281] [Originator@6876 sub=Default opID=780d04c9]
  -->
  --> Panic: Unrecoverable VmRootError. Panic!
  --> Backtrace:
  --> [backtrace begin] product: VMware VirtualCenter, version: 6.7.0, build: build-9232942, tag: vpxd, cpu: x86_64, os: linux, buildType: release
  --> backtrace[00] libvmacore.so[0x002A9C48]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
  --> backtrace[01] libvmacore.so[0x001B2F1C]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
  --> backtrace[02] libvmacore.so[0x002A7E1E]
  --> backtrace[03] libvmacore.so[0x002A7EFE]: Vmacore::PanicExit(char const*)
  --> backtrace[04] vpxd[0x006E21CB]
  --> backtrace[05] vpxd[0x006F7535]
  --> backtrace[06] vpxd[0x006F8049]
  --> backtrace[07] vpxd[0x006FDAE6]
  --> backtrace[08] libvmacore.so[0x00230A3D]
  --> backtrace[09] libvmacore.so[0x00230D06]
  --> backtrace[10] libvmacore.so[0x002AF3E1]
  --> backtrace[11] libpthread.so.0[0x000073D4]
  --> backtrace[12] libc.so.6[0x000E8BBD]
  --> [backtrace end]

The obvious issue here appears the Postgres ERROR: right sibling's left-link doesn't match: block 9 links to 11 instead of expected 4 in index "vpxi_task_username"; error . I can't find a way to fix the issue online and the answers I do find involved rebuilding the database from a backup. I do have backups of vCenter and I actually tried to restore from backup today (on a new vCenter appliance) and immediately after a reboot, the new appliance had the same issue. So it seems like the database problem has been there for a while.

I have six ESXi hosts and almost 10 NSX networks on this vCenter, so I am really trying to avoid starting from scratch with a new server and reconfiguring everything.

Any suggestions will be greatly appreciated

[daphnissov]

If you have an active support agreement, I'd recommend you contact GSS for help with this issue.

[carboncopy]

Thanks for the feedback daphnissov. Unfortunately I do not as I am running this in my lab. My licenses are from VMUG Advantage which I believe does not come with support.

[carboncopy · Accepted solution]

I was able to solve the problem!

Seeing that the table with the problem was VPX_TASK, I performed another backup of the vCenter appliance that was not functioning, but this time I did the backup without including the Stats, Events, and Tasks. By excluding the Tasks, I would be excluding any SQL data associated with the VPX_TASK table, which meant the corrupted data would not be part of the backup. I then kicked off the vCenter installer and told it to restore from the new backup and after the restore was successful, everything started working again just fine. I am not seeing the error in the potgresql or vpxd logs.

Thanks for the help!

[Vijay2027]

Alternatively you can login to vpostgres database and truncate vpx_task table.

Login to DB:  /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

Query to truncate vpx_task: truncate table vpx_task;