nix01d
Contributor
Contributor

Failed to connect to endpoint (vCSA 6.5.0.22000)

Hello,

need help solving a problem.

On the start web page, the following error: "503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f7f000ab0e0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)"

Command> service-control --status --all

Running:

applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui

Stopped:

vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health

Vpxd service does not start.

Found errors in the following files:

vpxd.log

--> Error while executing the query

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] Connection diagnostic data from driver is XX000:0:110:ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] Bind parameters:

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] [0]datatype: 11, size: 4, arraySize: 0

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] value = "vm"

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] [1]datatype: 1, size: 4, arraySize: 0

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] value = 88

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [Vdb::IsRecoverableErrorCode] Unable to recover from XX000:7

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [Vdb::IsRecoverableErrorCode] Unable to recover from XX000:110

2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781;

--> Error while executing the query" is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"

2019-11-27T13:03:28.206Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] DoHostSync:host-23 [ProcessChanges] took 16139 ms

2019-11-27T13:03:28.221Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] DoHostSync:host-23 [DoHostSyncTime] took 17931 ms

2019-11-27T13:03:28.221Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] InvtHostSyncLRO::StartWork [HostSyncTime] took 17931 ms

2019-11-27T13:03:28.221Z error vpxd[7F82D3D7A700] [Originator@6876 sub=vpxCommon opID=HostSync-host-23-3f3aa0c5] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

2019-11-27T13:03:28.221Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (XX000) - ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781;

--> Error while executing the query" is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"

2019-11-27T13:03:28.224Z panic vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5]

postgresql.log

2019-11-27 13:01:45.298 UTC 5dde7384.125c 0 VCDB vc ERROR:  missing chunk number 0 for toast value 209473 in pg_toast_24781

2019-11-27 13:01:45.298 UTC 5dde7384.125c 0 VCDB vc STATEMENT:  declare "SQL_CUR0x7f3040180b30" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f3040180b30"

2019-11-27 13:01:48.172 UTC 5dde7307.a12 0   LOG:  Updating instance status...

2019-11-27 13:01:48.172 UTC 5dde7307.a12 0   LOG:  Memory check: flag = rss, mem_used_bytes = 26044 kB, mem_avail = 715776 kB

2019-11-27 13:01:48.172 UTC 5dde7307.a12 0   LOG:  Writing instance status...

2019-11-27 13:01:48.172 UTC 5dde7307.a12 0   LOG:  Wrote instance status successfully.

2019-11-27 13:01:48.172 UTC 5dde7307.a12 0   LOG:  Updated instance status successfully.

2019-11-27 13:01:53.282 UTC 5dde73bc.15ed 4594578   LOG:  automatic vacuum of table "VCDB.vc.vpx_ext_type_ids": index scans: 1

        pages: 0 removed, 46 remain

        tuples: 40 removed, 1329 remain, 0 are dead but not yet removable

        buffer usage: 190 hits, 18 misses, 6 dirtied

        avg read rate: 0.028 MB/s, avg write rate: 0.009 MB/s

        system usage: CPU 0.00s/0.00u sec elapsed 5.06 sec

2019-11-27 13:01:55.593 UTC 5dde7384.125b 4594241 VCDB vc LOG:  could not receive data from client: Connection reset by peer

2019-11-27 13:01:55.593 UTC 5dde7384.125b 4594241 VCDB vc LOG:  unexpected EOF on client connection with an open transaction

2019-11-27 13:02:40.508 UTC 5dde73c8.1798 0 VCDB vc ERROR:  missing chunk number 0 for toast value 209473 in pg_toast_24781

2019-11-27 13:02:40.508 UTC 5dde73c8.1798 0 VCDB vc STATEMENT:  declare "SQL_CUR0x7f453464d8b0" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f453464d8b0"

2019-11-27 13:02:48.196 UTC 5dde7307.a12 0   LOG:  Updating instance status...

2019-11-27 13:02:48.196 UTC 5dde7307.a12 0   LOG:  Memory check: flag = rss, mem_used_bytes = 16860 kB, mem_avail = 715776 kB

2019-11-27 13:02:48.196 UTC 5dde7307.a12 0   LOG:  Writing instance status...

2019-11-27 13:02:48.196 UTC 5dde7307.a12 0   LOG:  Wrote instance status successfully.

2019-11-27 13:02:48.196 UTC 5dde7307.a12 0   LOG:  Updated instance status successfully.

2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc ERROR:  missing chunk number 0 for toast value 209473 in pg_toast_24781

2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc STATEMENT:  declare "SQL_CUR0x7f82e44d45c0" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f82e44d45c0"

I don’t understand how to fix this problem.

7 Replies
daphnissov
Immortal
Immortal

It could be postgres corruption. Recommend you open a SR with VMware GSS for resolution.

0 Kudos
Omid_Heravi
Enthusiast
Enthusiast

Hi

Check vCenter DNS record is correct and your system and vcenter are connected to DNS server. Also check the NTP and Time is correct or not.

I hope you can resolve it!

0 Kudos
blazilla
Enthusiast
Enthusiast

Looks like a corrupt Postgres database. Have you checked free space for all mount points? Did something crash? Power loss of ESXi host or similar?

Best regards Patrick https://www.vcloudnine.de
0 Kudos
Vijay2027
Expert
Expert

vpxd crashed due to below error:

2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc ERROR:  missing chunk number 0 for toast value 209473 in pg_toast_24781

Connect to DB and run the below query to check which table is corrupt:

select pg_toast_24781::regclass;

nix01d
Contributor
Contributor

ERROR:  column "pg_toast_24781" does not exist

LINE 1: select pg_toast_24781::regclass;

               ^

0 Kudos
Vijay2027
Expert
Expert

Looks like there is an issue with copy paste. can you manually enter the query?

0 Kudos
msripada
Virtuoso
Virtuoso

try login to vpostgres

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

and run the command below

reindex table vpx_text_array;

start vpxd service using command service-control --start vpxd and share the latest vpxd.log output

if the above is still showing same error please get the output for

select pg_toast_24781::regclass;

you can try the below commands

reindex table pg_toast.pg_toast_24781; or reindex table pg_toast_24781;

to come out of vpostgres screen, press \q

thanks,

MS

0 Kudos