Hello,
need help solving a problem.
On the start web page, the following error: "503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f7f000ab0e0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)"
Command> service-control --status --all
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health
Vpxd service does not start.
Found errors in the following files:
vpxd.log
--> Error while executing the query
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] Connection diagnostic data from driver is XX000:0:110:ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] Bind parameters:
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] [0]datatype: 11, size: 4, arraySize: 0
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] value = "vm"
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] [1]datatype: 1, size: 4, arraySize: 0
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] value = 88
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [Vdb::IsRecoverableErrorCode] Unable to recover from XX000:7
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [Vdb::IsRecoverableErrorCode] Unable to recover from XX000:110
2019-11-27T13:03:28.206Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781;
--> Error while executing the query" is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"
2019-11-27T13:03:28.206Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] DoHostSync:host-23 [ProcessChanges] took 16139 ms
2019-11-27T13:03:28.221Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] DoHostSync:host-23 [DoHostSyncTime] took 17931 ms
2019-11-27T13:03:28.221Z warning vpxd[7F82D3D7A700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-23-3f3aa0c5] InvtHostSyncLRO::StartWork [HostSyncTime] took 17931 ms
2019-11-27T13:03:28.221Z error vpxd[7F82D3D7A700] [Originator@6876 sub=vpxCommon opID=HostSync-host-23-3f3aa0c5] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
2019-11-27T13:03:28.221Z error vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (XX000) - ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781;
--> Error while executing the query" is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"
2019-11-27T13:03:28.224Z panic vpxd[7F82D3D7A700] [Originator@6876 sub=Default opID=HostSync-host-23-3f3aa0c5]
postgresql.log
2019-11-27 13:01:45.298 UTC 5dde7384.125c 0 VCDB vc ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781
2019-11-27 13:01:45.298 UTC 5dde7384.125c 0 VCDB vc STATEMENT: declare "SQL_CUR0x7f3040180b30" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f3040180b30"
2019-11-27 13:01:48.172 UTC 5dde7307.a12 0 LOG: Updating instance status...
2019-11-27 13:01:48.172 UTC 5dde7307.a12 0 LOG: Memory check: flag = rss, mem_used_bytes = 26044 kB, mem_avail = 715776 kB
2019-11-27 13:01:48.172 UTC 5dde7307.a12 0 LOG: Writing instance status...
2019-11-27 13:01:48.172 UTC 5dde7307.a12 0 LOG: Wrote instance status successfully.
2019-11-27 13:01:48.172 UTC 5dde7307.a12 0 LOG: Updated instance status successfully.
2019-11-27 13:01:53.282 UTC 5dde73bc.15ed 4594578 LOG: automatic vacuum of table "VCDB.vc.vpx_ext_type_ids": index scans: 1
pages: 0 removed, 46 remain
tuples: 40 removed, 1329 remain, 0 are dead but not yet removable
buffer usage: 190 hits, 18 misses, 6 dirtied
avg read rate: 0.028 MB/s, avg write rate: 0.009 MB/s
system usage: CPU 0.00s/0.00u sec elapsed 5.06 sec
2019-11-27 13:01:55.593 UTC 5dde7384.125b 4594241 VCDB vc LOG: could not receive data from client: Connection reset by peer
2019-11-27 13:01:55.593 UTC 5dde7384.125b 4594241 VCDB vc LOG: unexpected EOF on client connection with an open transaction
2019-11-27 13:02:40.508 UTC 5dde73c8.1798 0 VCDB vc ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781
2019-11-27 13:02:40.508 UTC 5dde73c8.1798 0 VCDB vc STATEMENT: declare "SQL_CUR0x7f453464d8b0" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f453464d8b0"
2019-11-27 13:02:48.196 UTC 5dde7307.a12 0 LOG: Updating instance status...
2019-11-27 13:02:48.196 UTC 5dde7307.a12 0 LOG: Memory check: flag = rss, mem_used_bytes = 16860 kB, mem_avail = 715776 kB
2019-11-27 13:02:48.196 UTC 5dde7307.a12 0 LOG: Writing instance status...
2019-11-27 13:02:48.196 UTC 5dde7307.a12 0 LOG: Wrote instance status successfully.
2019-11-27 13:02:48.196 UTC 5dde7307.a12 0 LOG: Updated instance status successfully.
2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781
2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc STATEMENT: declare "SQL_CUR0x7f82e44d45c0" cursor with hold for SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = 'vm' AND MO_ID = 88;fetch 1024 in "SQL_CUR0x7f82e44d45c0"
I don’t understand how to fix this problem.
It could be postgres corruption. Recommend you open a SR with VMware GSS for resolution.
Hi
Check vCenter DNS record is correct and your system and vcenter are connected to DNS server. Also check the NTP and Time is correct or not.
I hope you can resolve it!
Looks like a corrupt Postgres database. Have you checked free space for all mount points? Did something crash? Power loss of ESXi host or similar?
vpxd crashed due to below error:
2019-11-27 13:03:28.205 UTC 5dde73f7.1b09 0 VCDB vc ERROR: missing chunk number 0 for toast value 209473 in pg_toast_24781
Connect to DB and run the below query to check which table is corrupt:
select pg_toast_24781::regclass;
ERROR: column "pg_toast_24781" does not exist
LINE 1: select pg_toast_24781::regclass;
^
Looks like there is an issue with copy paste. can you manually enter the query?
try login to vpostgres
and run the command below
start vpxd service using command service-control --start vpxd and share the latest vpxd.log output
if the above is still showing same error please get the output for
select pg_toast_24781::regclass;
you can try the below commands
reindex table pg_toast.pg_toast_24781; or reindex table pg_toast_24781;
to come out of vpostgres screen, press \q
thanks,
MS