VCSA reporting 'VMware vAPI Endpoint Service Healt...

HelloFelix · ‎09-22-2021

Hey VMUG user here.

Yesterday I upgraded my VCSA from 7.0u2c to 7.0u2d via "vCenter Server Management" web console.

The following yellow alert shows on the vCenter monitoring page after upgraded:

VMware vAPI Endpoint Service Health Alarm ... [Acknowledge] [Reset To Green]

Rebooting the vCenter server or rebooting vmware-vapi-endpoint doesn't help.

Here's the logs from /var/log/vmware/vapi/endpoint/endpoint.log :

2021-09-22T21:24:12.375Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild cis-api-connections-builder
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:7080/svcaccountmgmt/vapi
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.svcaccountmgmt : /usr/lib/vmidentity/install/svcaccountmgmt/svcaccountmgmt_authorization.json
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10090/hvc
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.hvc : /usr/lib/vmware-hvc/config/metadata/hvc_authorization.json
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10080/computepolicy
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsStateBuilder     | Cannot resolve protocol priorities between the following services. Will use the first one.
        First: 15b46e6f-70c0-4701-969e-666a080404f5\com.vmware.cis.cls.vapi at http://vcenter.cloudcdn.top:80/cls/
        Second: 15b46e6f-70c0-4701-969e-666a080404f5\com.vmware.cis.cls.vapi at http://localhost:16666/cls/
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10096/certificatemanagement
2021-09-22T21:24:12.546Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.certificatemanagement : /usr/lib/vmware-certificatemanagement/config/metadata/com.vmware.vcenter_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10096/certificatemanagement
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.certificatemanagement : /usr/lib/vmware-certificatemanagement/config/metadata/com.vmware.vcenter_authorization.json
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsStateBuilder     | Duplicate service URLs detected for URL http://localhost:10096/certificatemanagement and serviceIds cf25970e-5b20-42c4-8840-81c246447a1f\com.vmware.certificatemanagement.vapi and e8ae710c-f8b2-4516-bcb3-b1ebc02cdbdb\com.vmware.certificatemanagement.vapi. Service cf25970e-5b20-42c4-8840-81c246447a1f\com.vmware.certificatemanagement.vapi will not be registered.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH YELLOW Incorrect service registration found in Lookup Service. Multiple services registered on same URL - http://localhost:10096/certificatemanagement.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10098/topologysvc
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.topologysvc : /usr/lib/vmware-topologysvc/config/metadata/com.vmware.vcenter_metadata.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file : /usr/lib/vmware-updatemgr/etc/cis_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.health : /usr/lib/vmware-updatemgr/etc/health_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.hosts : /usr/lib/vmware-updatemgr/etc/hosts_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.settings : /usr/lib/vmware-updatemgr/etc/settings_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8083/esx-lcm
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.hcl : /usr/lib/vmware-updatemgr/etc/hcl_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8900/vmonapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:12080/ls/slms
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:13023/appliance/observability
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.observability : /etc/vmware-observability/config/metadata/com.vmware.appliance.observability_metadata.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8920/wcp
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:7080/svcaccountmgmt/vapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.svcaccountmgmt : /usr/lib/vmidentity/install/svcaccountmgmt/svcaccountmgmt_authorization.json
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsStateBuilder     | Duplicate service URLs detected for URL http://localhost:7080/svcaccountmgmt/vapi and serviceIds 8b87c833-6897-490d-80e9-d2d20e944f53\com.vmware.vcenter.svcaccountmgmt.vapi and 97962f90-eda1-46b9-a12f-a026bfaf4f8d\com.vmware.vcenter.svcaccountmgmt.vapi. Service 8b87c833-6897-490d-80e9-d2d20e944f53\com.vmware.vcenter.svcaccountmgmt.vapi will not be registered.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH YELLOW Incorrect service registration found in Lookup Service. Multiple services registered on same URL - http://localhost:7080/svcaccountmgmt/vapi.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:7080/tokenservice/vapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:5090/ds/vapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10095/trustmanagement
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.trustmanagement : /usr/lib/vmware-trustmanagement/config/metadata/trustmanagement_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:13080/statsreport/topn
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10080/invsvc/vapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:7080/system/vapi
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.activedirectory : /usr/lib/vmidentity/install/activedirectory/activedirectory_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8093/vcenter
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10098/topologysvc
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.topologysvc : /usr/lib/vmware-topologysvc/config/metadata/com.vmware.vcenter_metadata.json
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsStateBuilder     | Duplicate service URLs detected for URL http://localhost:10098/topologysvc and serviceIds f90cf0e4-1798-4d50-8980-9502f35bc752\com.vmware.topology.vapi and 107682e9-bc5a-4402-b283-d77d2ba49835\com.vmware.topology.vapi. Service f90cf0e4-1798-4d50-8980-9502f35bc752\com.vmware.topology.vapi will not be registered.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH YELLOW Incorrect service registration found in Lookup Service. Multiple services registered on same URL - http://localhost:10098/topologysvc.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:12020/lcm/api
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:11098/infraprofile
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.infraprofile : /usr/lib/vmware-infraprofile/config/metadata/com.vmware.appliance.infraprofile_metadata.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:11098/vcenter/settings/v1
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.vcenter.settings.v1 : /usr/lib/vmware-infraprofile/config/metadata/com.vmware.appliance.vcenter.settings.v1_metadata.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10102/certificateauthority
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.certificateauthority : /usr/lib/vmware-certificateauthority/config/metadata/com.vmware.vcenter_authorization.json
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8190/pbm/vapi
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.api.pbm with protocol vapi.json.http at http://localhost:8190/pbm/vapi
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.sms.ServiceDescription and id 23eb5d49-e75b-4f44-9d43-ba11918d288e.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10095/trustmanagement
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.trustmanagement : /usr/lib/vmware-trustmanagement/config/metadata/trustmanagement_authorization.json
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsStateBuilder     | Duplicate service URLs detected for URL http://localhost:10095/trustmanagement and serviceIds c14664ed-832c-43c4-920e-4b27d955fe98\com.vmware.trustmanagement.vapi and b8182845-a1ac-4f35-a3e8-d8a4dd0fd9b3\com.vmware.trustmanagement.vapi. Service c14664ed-832c-43c4-920e-4b27d955fe98\com.vmware.trustmanagement.vapi will not be registered.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH YELLOW Incorrect service registration found in Lookup Service. Multiple services registered on same URL - http://localhost:10095/trustmanagement.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:13180/stats
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10102/certificateauthority
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | ApiConnectionsLsUtil           | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.certificateauthority : /usr/lib/vmware-certificateauthority/config/metadata/com.vmware.vcenter_authorization.json
2021-09-22T21:24:12.547Z | WARN  | state-manager1            | ApiConnectionsStateBuilder     | Duplicate service URLs detected for URL http://localhost:10102/certificateauthority and serviceIds d2321153-eb6b-4f59-b8a3-30f019fd7ef4\com.vmware.certificateauthority.vapi and f2edc49e-9240-498d-b87f-984f4d4b7a84\com.vmware.certificateauthority.vapi. Service d2321153-eb6b-4f59-b8a3-30f019fd7ef4\com.vmware.certificateauthority.vapi will not be registered.
2021-09-22T21:24:12.547Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH YELLOW Incorrect service registration found in Lookup Service. Multiple services registered on same URL - http://localhost:10102/certificateauthority.
2021-09-22T21:24:12.548Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vim-adapter-settings-builder
2021-09-22T21:24:12.559Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vapi-vcenter-servlet-builder
2021-09-22T21:24:12.559Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild api-interfaces-builder
2021-09-22T21:24:12.622Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vcenter-shim-service-builder
2021-09-22T21:24:12.622Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild metadata-sync-builder
2021-09-22T21:24:12.622Z | INFO  | state-manager1            | MetadataSynchronizationBuilder | rebuild
2021-09-22T21:24:12.734Z | INFO  | state-manager1            | MetadataSynchronizationBuilder | rebuild finished.
2021-09-22T21:24:12.735Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild api-metadata-builder
2021-09-22T21:24:12.784Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild diagnostic-state-builder
2021-09-22T21:24:12.784Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild telemetry-state-builder
2021-09-22T21:24:12.784Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild security-dispatcher-builder
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild public-api-builder
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild request-filter-builder
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vapi-servlet-builder
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild rest-builder
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | RestBuilder                    | Publishing REST servlet rest-servlet.
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | RestBuilder                    | Publishing REST servlet rest-servlet-public.
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | RestBuilder                    | Publishing REST servlet verb-rest-servlet.
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild swagger-generator
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | StatusInfoFactory              | HEALTH GREEN Configuration health status is created between 2021-09-22T21:24:12UTC and 2021-09-22T21:24:12UTC.
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | CollectedHealthStatusProviderImpl | Computed health status is YELLOW.
2021-09-22T21:24:12.786Z | INFO  | state-manager1            | DefaultStateManager            | No changes. Skip rebuilding state

js17 · ‎04-13-2022

Have the exact same issue, don't see anyone with a solution :(. We are using terraform to deployment which leverages the API and we have intermittent issues.

Rebooting the server or restarting vmware-vapi-endpoint does not help

NAIKAKASH · ‎07-22-2022

I am also facing Windows VC failed to upgrade from 6.7U3O to 6.7U3Q and getting 1603 normal error while VCSServiceManager starting stage on Windows but during investigation I reached to external one of the PSC and there I found VAPI end Point service unable to find Metadata.

Please help

2022-07-20T17:06:07.931Z | WARN | state-manager1 | ApiConnectionsLsUtil | Cannot find metadata source files/URLs in VAPI endpoint Servi

ce Endpoint of type com.vmware.vapi.endpoint with protocol vapi.json.http at http://xyz//api

2022-07-20T17:06:07.931Z | WARN | state-manager1 | ApiConnectionsLsUtil | Unable to find metadata endpoint in service Service with loca

lization key cis.vapi.endpoint.serviceDescriptionResourceKey and id 2ac98138-f8a9-497c-8f8b-e330fd0bbb79.

2022-07-20T17:06:07.931Z | INFO | state-manager1 | ApiConnectionsLsUtil | Unsupported source (metadata) type in metadata source entry c

is.common.ep.localurl : http://localhost:12346/site/api

2022-07-20T17:06:07.931Z | WARN | state-manager1 | ApiConnectionsLsUtil | Cannot find metadata source definitions in VAPI endpoint Serv

NAIKAKASH · ‎07-25-2022

Hello,

Did you find any solution ?

Akash

All

VCSA reporting 'VMware vAPI Endpoint Service Health Alarm' after upgraded from 7.0u2c to 7.0u2d