VMware-vpxd-svcs and VMware-vpxd no longer start

[scugnin]

Good Morning everyone,

Sorry to disturb you. For over a week now, my service (and not the server) has no longer started following a restart.
I already checked the different statuses of the services and at first, the VMware-vpostgres and lookupsvc service did not start at all, which I managed to repair. However, I still have the VMware-vpxd-svcs and VMware-vpxd service which does not start at all. I read a lot of KB and tried a lot of things but nothing conclusive.

Here are the logs from the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file when I start the VMware-vpxd-svcs service:

root@vcenter [ ~ ]# tail -f /var/log/vmware/vpxd-svcs/vpxd-svcs.log
... 45 more
Caused by: java.lang.RuntimeException: Could not find any solution users from SSO.
at com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl.<init>(CheckPrivilegesRouterRiseImpl.java:82)
at com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterFactory.<init>(CheckPrivilegesRouterFactory.java:20)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204)
... 47 more
2023-10-09T09:35:33.004Z [pool-6-thread-1 ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : Administrator@vsphere.local
com.vmware.cis.server.ssoauthentication.exception.ServiceFaultException: com.vmware.cis.server.authentication.exception.TokenProviderException: com.vmware.cis.server.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/lookupservice/sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.setupAdminClientInternal(AdminClientWrapperImpl.java:93)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.setupAdminClient(AdminClientWrapperImpl.java:58)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.access$200(AdminClientWrapperImpl.java:37)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl$1.setupInitialState(AdminClientWrapperImpl.java:122)
at com.vmware.cis.server.ssoauthentication.impl.RetryAction.execute(RetryAction.java:48)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.getDomains(AdminClientWrapperImpl.java:135)
at com.vmware.cis.server.ssoauthentication.impl.DomainNameNormalizerImpl.populateDomainMap(DomainNameNormalizerImpl.java:77)
at com.vmware.cis.server.ssoauthentication.impl.DomainNameNormalizerImpl.toSsoDomain(DomainNameNormalizerImpl.java:57)
at com.vmware.cis.server.ssoauthentication.impl.SsoPrincipalFactoryImpl.principalIdFromName(SsoPrincipalFactoryImpl.java:123)
at com.vmware.cis.server.ssoauthentication.impl.SsoPrincipalFactoryImpl.createUserPrincipal(SsoPrincipalFactoryImpl.java:53)
at com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper.fetchSsoAdminGroup(AuthenticationHelper.java:844)
at com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper.access$000(AuthenticationHelper.java:54)
at com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper$1.run(AuthenticationHelper.java:881)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.cis.server.authentication.exception.TokenProviderException: com.vmware.cis.server.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/lookupservice/sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:66)
at com.vmware.cis.server.ssoauthentication.impl.AbstractTokenProvider.getSamlToken(AbstractTokenProvider.java:42)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.setupAdminClientInternal(AdminClientWrapperImpl.java:87)
... 19 more
Caused by: com.vmware.cis.server.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/lookupservice/sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
at com.vmware.cis.server.ssoauthentication.impl.CertificateProviderImpl.getTrustedRootCertificates(CertificateProviderImpl.java:48)
at com.vmware.cis.server.ssoauthentication.impl.StsClientFactoryWrapperImpl.createSts(StsClientFactoryWrapperImpl.java:65)
at com.vmware.cis.server.ssoauthentication.impl.StsClientFactoryWrapperImpl.getStsClient(StsClientFactoryWrapperImpl.java:102)
at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:52)
... 21 more
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/lookupservice/sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:321)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:55)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
at com.sun.proxy.$Proxy40.list(Unknown Source)
at com.vmware.cis.server.ssoauthentication.impl.ServiceLocatorImpl.populateUrlsUsingLS(ServiceLocatorImpl.java:78)
at com.vmware.cis.server.ssoauthentication.impl.ServiceLocatorImpl.getSsoAdminEndpoint(ServiceLocatorImpl.java:158)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientFactoryWrapperImpl.populateAdminClientFactory(AdminClientFactoryWrapperImpl.java:42)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientFactoryWrapperImpl.setupAdminClientFactory(AdminClientFactoryWrapperImpl.java:80)
at com.vmware.cis.server.ssoauthentication.impl.AdminClientFactoryWrapperImpl.createAdminClient(AdminClientFactoryWrapperImpl.java:72)
at com.vmware.cis.server.ssoauthentication.impl.CertificateProviderImpl.getTrustedRootCertificates(CertificateProviderImpl.java:37)
... 24 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:48)
... 40 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 50 more
2023-10-09T09:35:50.162Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 1
2023-10-09T09:36:10.175Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 2
2023-10-09T09:36:30.192Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 3
2023-10-09T09:36:50.202Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 4
2023-10-09T09:37:10.218Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 5
2023-10-09T09:37:30.226Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 6
2023-10-09T09:37:50.235Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 7
2023-10-09T09:38:10.243Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 8
2023-10-09T09:38:26.376Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:38:26.377Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:38:26.389Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:38:26.394Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:38:30.121Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:38:30.122Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:38:30.127Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:38:30.130Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:38:30.131Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:38:30.137Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:38:30.138Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:38:30.142Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:38:30.251Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 9
^[2023-10-09T09:38:50.255Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 10
2023-10-09T09:39:10.267Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 11
2023-10-09T09:39:30.281Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 12
2023-10-09T09:39:50.301Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 13
2023-10-09T09:40:10.309Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 14
2023-10-09T09:40:30.317Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 15
2023-10-09T09:40:50.336Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 16
2023-10-09T09:41:10.343Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 17
2023-10-09T09:41:26.401Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:41:26.401Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:41:26.404Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:41:26.412Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:41:30.148Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:41:30.149Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:41:30.152Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:41:30.154Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainControllerEx("") : vcenter.CENSORED
2023-10-09T09:41:30.154Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : vcenter.CENSORED
2023-10-09T09:41:30.157Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
2023-10-09T09:41:30.160Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:41:30.162Z [refresh-lotus-locator-task INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2023-10-09T09:41:30.354Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 18
2023-10-09T09:41:50.362Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 19
2023-10-09T09:42:10.368Z [Thread-15 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 20

I do not have any expired STS certificates or tokens:

root@vcenter [ /scripts ]# python checksts.py

2 VALID CERTS
================

LEAF CERTS:

[] Certificate CENSORED will expire in 727 days (2 years).

ROOT CERTS:

[] Certificate CENSORED will expire in 2831 days (8 years).

0 EXPIRED CERTS
================

LEAF CERTS:

None

ROOT CERTS:

None

And:

root@vcenter [ /scripts ]# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Jun 21 13:25:45 2024 GMT
[*] Store : TRUSTED_ROOTS
Alias : CENSORED
Not After : Jul 10 17:02:43 2031 GMT
Alias : CENSORED
Not After : Jul 25 12:52:28 2022 GMT
Alias : CENSORED
Not After : Jun 22 13:07:28 2042 GMT
[*] Store : machine
Alias : machine
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : vsphere-webclient
Alias : vsphere-webclient
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : vpxd
Alias : vpxd
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : vpxd-extension
Alias : vpxd-extension
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : hvc
Alias : hvc
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : data-encipherment
Alias : data-encipherment
Not After : Jul 10 17:02:43 2031 GMT
[*] Store : APPLMGMT_PASSWORD
Alias : location_password_default
[*] Store : SMS
Alias : sms_self_signed
Not After : Jul 15 17:11:25 2031 GMT
[*] Store : wcp
Alias : wcp
Not After : Jul 15 17:03:48 2023 GMT
[*] Store : BACKUP_STORE
Alias : __MACHINE_CERT
Not After : Jul 25 12:52:28 2022 GMT

Thanking you in advance for your help!
Good day to you.

[d3m1g0d]

Regarding the error: 

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

Seems that the Envoy proxy is not running/listening on port 1080. You can check it on the VCSA:

service-control --status envoy rhttpproxy

and specifically: netstat -anp | grep tcp.*LISTEN.*envoy

Have you tried to start/restart it the envoy service? service-control --restart envoy

Or restart all services? service-control --stop --all && service-control --start --all

Is there anything meaningful in the Envoy logs in the directory /var/log/vmware/envoy?

In the rhttpproxy logs in /var/log/vmware/rhttpproxy?

Also check /etc/hosts for any errors (see https://kb.vmware.com/s/article/59476).

[scugnin]

Good Morning d3m1g0d,

The configuration of /etc/hosts is correct, I checked by reading the KB.

Information about rhttpproxy and envoy.

root@vcenter [ ~ ]# service-control --status envoy rhttpproxy
Running:
rhttpproxy
Stopped:
envoy
root@vcenter [ ~ ]# service-control --start envoy
   Operation not cancellable. Please wait for it to finish...
   Performing start operation on service envoy...
   Error executing start on service envoy. Details {
   "detail": [
   {
   "id": "install.ciscommon.service.failstart",
   "translatable": "An error occurred while starting service '%(0)s'",
   "args": [
   "envoy"
   ],
   "localized": "An error occurred while starting service 'envoy'"
   }
   ],
   "componentKey": null,
   "problemIdWhen I try to restart all services, some crash": null,
   "resolution": null
    }
   Service-control failed. Error: {
   "detail": [
   {
   "id": "install.ciscommon.service.failstart",
   "translatable": "An error occurred while starting service '%(0)s'",
   "args": [
   "envoy"
],
"localized": "An error occurred while starting service 'envoy'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
root@vcenter [ ~ ]# netstat -anp | grep tcp.*LISTEN.*envoy
root@vcenter [ ~ ]#

 When I try to restart all services, some crash:

root@vcenter [ ~ ]# service-control --stop --all && service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service observability...
Successfully stopped service observability
Performing stop operation on service vmware-pod...
Successfully stopped service vmware-pod
Performing stop operation on service vmware-vdtc...
Successfully stopped service vmware-vdtc
Performing stop operation on profile: ALL...
Successfully stopped service vmware-vmon
Successfully stopped profile: ALL.
Performing stop operation on service vmcad...
Successfully stopped service vmcad
Performing stop operation on service vmdird...
Successfully stopped service vmdird
Performing stop operation on service vmafdd...
Successfully stopped service vmafdd
Performing stop operation on service lwsmd...
Successfully stopped service lwsmd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start eam, sca, vsphere-ui, envoy, vapi-endpoint, vtsdb, vpxd-svcs services. Error: Operation timed out

For the logs /var/log/vmware/envoy I find absolutely nothing relevant I only have logs older than 2022.

And for the /var/log/vmware/rhttpproxy, look below:

cat /var/log/vmware/rhttpproxy/rhttpproxy.log
2023-10-10T07:40:42.335Z - time the service was last started, Section for rhttpproxy, pid=642, version=7.0.2, build=17956808, option=Release
2023-10-10T07:40:42.333Z info -[00642] [Originator@6876 sub=Default] Glibc malloc guards disabled.
2023-10-10T07:40:42.333Z info -[00642] [Originator@6876 sub=Default] Initialized SystemFactory
2023-10-10T07:40:42.335Z warning rhttpproxy[00642] [Originator@6876 sub=Default] Unrecognized log/level '' using 'info'
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Logging uses fast path: true
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/etc/vmware/vsphereFeatures/vsphereFeatures.cfg": No such file or directory.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "/etc/vmware/vsphereFeatures/vsphereFeatures.cfg": No such file or directory.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Libs] FeatureStateLib: Error while loading feature config file: /etc/vmware/vsphereFeatures/vsphereFeatures.cfg, using default feature state values.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Initialized channel manager
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Current working directory: /storage/log/vmware/vmon
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=ThreadPool] Catch work item exceptions disabled.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=FairScheduler] Priority level 4 is now active.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOCompletionQueue] Created: WorkQueue.vmacoreDefaultIOCompletionQueue, type = fair, priority = 4, itemWeight = 1
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=FairScheduler] Priority level 8 is now active.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOQueue] Created: WorkQueue.vmacoreDefaultIOQueue, type = fair, priority = 8, itemWeight = 1
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=FairScheduler] Priority level 16 is now active.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.vmacoreDefaultLongTaskQueue] Created: WorkQueue.vmacoreDefaultLongTaskQueue, type = fair, priority = 16, itemWeight = 1
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 1, idle: 0
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=ThreadPool] Thread pool fair initial threads spawned. IO: 1, Min workers: 1, Max workers: 6, Reservation ratio: 9
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Syscommand enabled: false
2023-10-10T07:40:42.335Z info rhttpproxy[00646] [Originator@6876 sub=ThreadPool] Entering IO thread loop
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Successfully registered handler for signal 2.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Default] Successfully registered handler for signal 15.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy] Shutdown handler installed.
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.rhttpproxy.serial.queue] Created: WorkQueue.rhttpproxy.serial.queue, type = serial, priority = 16, itemWeight = 1
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF early PreferenceGet(tls.ciphers), using default
2023-10-10T07:40:42.335Z info rhttpproxy[00642] [Originator@6876 sub=Libs] lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF early PreferenceGet(tls.curves), using default
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] lib/ssl: curves list prime256v1:secp384r1:secp521r1
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF early PreferenceGet(tls.protocols), using default
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Default] Configured libraryPath =
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/etc/vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "/etc/vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF Optional preferences file not found at /etc/vmware/config. Using default values.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] Could not expand environment variable HOME.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] Could not expand environment variable HOME.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/usr/lib/vmware/settings": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "/usr/lib/vmware/settings": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF Optional preferences file not found at /usr/lib/vmware/settings. Using default values.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "/usr/lib/vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF Optional preferences file not found at /usr/lib/vmware/config. Using default values.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "~/.vmware/config": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF Optional preferences file not found at ~/.vmware/config. Using default values.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] [msg.dictionary.load.openFailed] Cannot open file "~/.vmware/preferences": No such file or directory.
2023-10-10T07:40:42.336Z info rhttpproxy[00642] [Originator@6876 sub=Libs] PREF Optional preferences file not found at ~/.vmware/preferences. Using default values.
2023-10-10T07:40:42.348Z info rhttpproxy[00647] [Originator@6876 sub=ThreadPool] Entering fair thread loop
2023-10-10T07:40:42.348Z info rhttpproxy[00642] [Originator@6876 sub=Libs] lib/ssl: OpenSSL using FIPS_drbg for RAND
2023-10-10T07:40:42.348Z info rhttpproxy[00642] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2023-10-10T07:40:42.361Z info rhttpproxy[00645] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.364Z info rhttpproxy[00642] [Originator@6876 sub=Default] Vmacore::InitSSL: handshakeTimeoutUs = 20000000
2023-10-10T07:40:42.364Z info rhttpproxy[00642] [Originator@6876 sub=Default] Service is running in FIPS mode.
2023-10-10T07:40:42.372Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: Auto-Expire-Cluster
2023-10-10T07:40:42.377Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy] PNID: "vcenter.ihu-strasbourg.eu"
2023-10-10T07:40:42.377Z info rhttpproxy[00645] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 2, idle: 0
2023-10-10T07:40:42.378Z info rhttpproxy[00645] [Originator@6876 sub=EnvoyAds] Server listening on unix:/var/run/vmware/envoy-ads-pipe
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=Default] [PacketCapture] Removing pcap files last modified before: 2023-10-07T07:40:42.378693Z
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.rhttpproxy.packetsniffer[000055ec007a0020].queue] Created: WorkQueue.rhttpproxy.packetsniffer[000055ec007a0020].queue, type = serial, priority = 16, itemWeight = 1
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.rhttpproxy.packetsniffer[000055ec007a0148].queue] Created: WorkQueue.rhttpproxy.packetsniffer[000055ec007a0148].queue, type = serial, priority = 16, itemWeight = 1
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=WorkQueue.rhttpproxy.packetsniffer[000055ec007a0270].queue] Created: WorkQueue.rhttpproxy.packetsniffer[000055ec007a0270].queue, type = serial, priority = 16, itemWeight = 1
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=Default] Successfully registered handler for signal 1.
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy] ReloadConfig handler installed.
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy.HTTPService] Max buffered response size is 104857600 bytes
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy.HTTPService] Http Service started: <server <acceptor p:0x000055ec00abbc60, h:11, <TCP '127.0.0.1 : 549'>>, >
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=RhttpProxy] Proxy REST endpoints config API service started.
2023-10-10T07:40:42.378Z info rhttpproxy[00642] [Originator@6876 sub=ThreadPool] Main thread queue created.
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 3, idle: 0
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/dataservice-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /ds
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vmonapi-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vmonapi
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/eam-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /eam
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/computepolicy-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /computepolicy
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vapi-endpoint.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /site/api
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /site/rest
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vapiendpoint/health
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vapiendpoint/resourcebundle
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /rest
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /api
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /apiexplorer
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/perfcharts-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /statsreport
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/analytics-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/cloudhealth/sdk
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/ceip/sdk
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/ceip/api/state
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/healthstatus
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/resourcebundle
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/telemetry/ph/api/hyper/send
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/telemetry/ph-stg/api/hyper/send
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/telemetry/ph/api/level
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics/telemetry/ph-stg/api/level
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /analytics
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/tokenservice.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /tokenservice/vapi
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/apigw-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /apigw
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/sca-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sca
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /supportbundle
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/activedirectory.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /system/vapi
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/applmgmt.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /applmgmt
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /appliance
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sdk
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sdkTunnel
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: sdkTunnel:8089
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: vcenter.ihu-strasbourg.eu:8089
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sdk-vpxd
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /mob
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vod
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /client/clients.xml
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /nfc
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /screen
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /qs
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vcenter
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /catalog
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/wcp-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /wcp
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/trustmanagement.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /trustmanagement
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vsanhealth-rhttpproxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/plugins
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/vum
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/hcl
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/vsanvp
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/health
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/ph
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth/fileService
2023-10-10T07:40:42.379Z warning rhttpproxy[00651] [Originator@6876 sub=Default] Malformed Mapping URL detected: /vsan/metrics/
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsan/metrics
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/cls.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /cls
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/certificateauthority.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /certificateauthority
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vdtc.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vdt
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/updatemgr-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /esx-lcm
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /hcl-reports
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vci/sdk
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vci/downloads/health.xml
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vci/downloads/vum-htmlclient.zip
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vum-fileupload
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vum-filedownload
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/imagebuilder-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vmw/imagebuilder/vmomi
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vmw/imagebuilder/httpd
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/infraprofile.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /infraprofile
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vcenter/settings/v1
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/netdump.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /netdump
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/hvc.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /hvc
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/invsvc-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /invsvc
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/license-proxy.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /ls
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /ls/ph
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/topologysvc.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /topologysvc
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vmidentity.conf
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sso-adminserver/idp
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sso-adminserver/sdk
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sts/STSService
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /websso
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /websso/SAML2/SSOCAC
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /openidconnect
2023-10-10T07:40:42.379Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /openidconnect/oidc/cac_rproxy
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /idm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /afd
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lookup
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vmdir
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sso-adminserver/system-sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sts/system-STSService
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/ui-plugins.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /plugins/com.vmware.lcm.client~1.0.0.0~1610180291/vcenter.ihu-strasbourg.eu-443
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/lookupservice.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lookupservice/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lookupservice/mob
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lookupservice
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/svcaccountmgmt.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /svcaccountmgmt/vapi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vstats-proxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /stats
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vsphere-ui-rhttpproxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /ui
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /ui/healthstatus
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/observability.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /observability
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /appliance/observability
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vcha-proxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vcha/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vsm-proxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/sps-rhttpproxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /sms
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /pbm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vslm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/snmpcfgd.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /VMwareSNMP
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/vlcm-proxy.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lcm/api
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lcm-ui
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lcm-reports
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /lcm/rest
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing file: /etc/vmware-rhttpproxy/endpoints.conf.d/certificatemanagement.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /certificatemanagement
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] Processing defaultMappingFile: /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /Query
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /pcnotification
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsan/sdkTunnel
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vsanHealth
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=Default] New/Updated endpoint added: /vws
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /ds
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vmonapi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /eam
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /computepolicy
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /site/api
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /site/rest
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vapiendpoint/health
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vapiendpoint/resourcebundle
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /rest
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /api
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /apiexplorer
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /statsreport
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/cloudhealth/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/ceip/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/ceip/api/state
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/healthstatus
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/resourcebundle
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/telemetry/ph/api/hyper/send
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/telemetry/ph-stg/api/hyper/send
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/telemetry/ph/api/level
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics/telemetry/ph-stg/api/level
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /analytics
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /tokenservice/vapi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /apigw
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sca
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /supportbundle
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /system/vapi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /applmgmt
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /appliance
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sdkTunnel
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: sdkTunnel:8089
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: vcenter.ihu-strasbourg.eu:8089
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sdk-vpxd
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /mob
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vod
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /client/clients.xml
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /nfc
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /screen
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /qs
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vcenter
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /catalog
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /wcp
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /trustmanagement
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/plugins
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/vum
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/hcl
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/vsanvp
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/health
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/ph
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth/fileService
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsan/metrics
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /cls
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /certificateauthority
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vdt
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /esx-lcm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /hcl-reports
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vci/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vci/downloads/health.xml
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vci/downloads/vum-htmlclient.zip
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vum-fileupload
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vum-filedownload
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vmw/imagebuilder/vmomi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vmw/imagebuilder/httpd
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /infraprofile
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vcenter/settings/v1
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /netdump
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /hvc
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /invsvc
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /ls
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /ls/ph
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /topologysvc
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sso-adminserver/idp
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sso-adminserver/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sts/STSService
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /websso
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /websso/SAML2/SSOCAC
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /openidconnect
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /openidconnect/oidc/cac_rproxy
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /idm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /afd
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lookup
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vmdir
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sso-adminserver/system-sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sts/system-STSService
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /plugins/com.vmware.lcm.client~1.0.0.0~1610180291/vcenter.ihu-strasbourg.eu-443
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lookupservice/sdk
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lookupservice/mob
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lookupservice
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /svcaccountmgmt/vapi
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /stats
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /ui
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /ui/healthstatus
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /observability
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /appliance/observability
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /sms
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /pbm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vslm
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /VMwareSNMP
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lcm/api
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lcm-ui
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lcm-reports
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /lcm/rest
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /certificatemanagement
2023-10-10T07:40:42.380Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /
2023-10-10T07:40:42.381Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /Query
2023-10-10T07:40:42.381Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /pcnotification
2023-10-10T07:40:42.381Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsan/sdkTunnel
2023-10-10T07:40:42.381Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vsanHealth
2023-10-10T07:40:42.381Z info rhttpproxy[00651] [Originator@6876 sub=RhttpProxy] New/Updated endpoint added: /vws
2023-10-10T07:40:42.391Z info rhttpproxy[00655] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.391Z info rhttpproxy[00655] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 4, idle: 0
2023-10-10T07:40:42.391Z info rhttpproxy[00655] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 5, idle: 1
2023-10-10T07:40:42.391Z info rhttpproxy[00655] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 6, idle: 2
2023-10-10T07:40:42.391Z info rhttpproxy[00658] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.391Z info rhttpproxy[00657] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.391Z info rhttpproxy[00656] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2023-10-10T07:40:42.392Z info rhttpproxy[00651] [Originator@6876 sub=EnvoyAds] Update started
2023-10-10T07:40:47.761Z error rhttpproxy[00657] [Originator@6876 sub=RhttpProxy] [Rhttpproxy routes REST DELETE Handler] Listener edge_https_v4, virtual host local_service or route /ui not found.
2023-10-10T07:40:47.840Z error rhttpproxy[00657] [Originator@6876 sub=RhttpProxy] [Rhttpproxy routes REST DELETE Handler] Listener edge_http_v4, virtual host local_service or route /ui not found.
2023-10-10T07:40:47.905Z error rhttpproxy[00655] [Originator@6876 sub=RhttpProxy] [Rhttpproxy routes REST DELETE Handler] Listener edge_https_v6, virtual host local_service or route /ui not found.
2023-10-10T07:40:48.051Z error rhttpproxy[00655] [Originator@6876 sub=RhttpProxy] [Rhttpproxy routes REST DELETE Handler] Listener edge_http_v6, virtual host local_service or route /ui not found.

 Thanks a lot for your help!

[scugnin]

Update:

I fixed every services who was failing when i was trying to start them.

I still have trouble with these services below:

[FAIL] vmware-vapi-endpoint IS STOPPED
[FAIL] vmware-perfcharts IS STOPPED
[FAIL] vmware-eam IS STOPPED
[FAIL] vmware-content-library IS STOPPED
[FAIL] vmware-updatemgr IS STOPPED
[FAIL] vsphere-ui IS STOPPED
[FAIL] vtsdb IS STOPPED
[FAIL] vstats IS STOPPED

For me it's a problem with the access rights on the folders.. but I don't know how to repair the rights on the different folders. I have the same error with these errors.

root@vcenter [ ~ ]# service-control --start vsphere-ui
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsphere-ui...
Error executing start on service vsphere-ui. Details {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vsphere-ui"
],
"localized": "An error occurred while starting service 'vsphere-ui'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
Service-control failed. Error: {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vsphere-ui"
],
"localized": "An error occurred while starting service 'vsphere-ui'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}

[d3m1g0d]

Have you managed to get services started by modifying permissions on files and folders in the filesystem? Is it related to the log folders?

If you really want to manually adjust files and folders in the VCSA: You could deploy another VCSA with the same version as a "blueprint", and modify permissions on the required files and folders in the broken VCSA as configured in the newly deployed appliance.

But I would rather recommend redeploying the VCSA and restore a previous working configuration from backup to have a clean system.

[scugnin]

It takes a long time to explain given the business context but I am not able to reassemble an instance of the same Build as my vCenter in production. My only solution is to repair the rights on the /storage/log/vmware* partition. But no way to check if the rights are good, so it's complicated. I need a tool or something that allows me to recover or restore rights.

Additional information:

vCenter Node Type: vCenter with Embedded PSC
vCenter Version: 7.0.2.00200 - 17958471

[scugnin]

Details of the startup command:

root@vcenter [ /scripts ]# service-control --verbose --start vsphere-ui
Operation not cancellable. Please wait for it to finish...
Perform start operation. vmon_profile=None, svc_names=['vsphere-ui'], include_coreossvcs=False, include_leafossvcs=False
Performing start operation on service vsphere-ui...
2023-10-11T13:18:10.600Z Running command: ['/sbin/service', 'vmware-vmon', 'status']
2023-10-11T13:18:10.655Z Done running command
2023-10-11T13:18:10.660Z Service vsphere-ui state STOPPED
2023-10-11T13:18:12.118Z ERROR starting vsphere-ui rc: 2, stdout: , stderr: Start service request failed. Error: Service crashed while starting

Error executing start on service vsphere-ui. Details {
   "detail": [
      {
          "id": "install.ciscommon.service.failstart",
          "translatable": "An error occurred while starting service '%(0)s'",
          "args": [
          "vsphere-ui"
          ],
          "localized": "An error occurred while starting service 'vsphere-ui'"
      }
   ],
   "componentKey": null,
   "problemId": null,
   "resolution": null
}
Service-control failed. Error: {
   "detail": [
      {
          "id": "install.ciscommon.service.failstart",
          "translatable": "An error occurred while starting service '%(0)s'",
          "args": [
          "vsphere-ui"
          ],
          "localized": "An error occurred while starting service 'vsphere-ui'"
      }
   ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

[d3m1g0d]

I'm not aware of a specific tool which recreates the log folders. Maybe VMware GSS has a script/program for this purpose.

If you want, I can send you a list of the files and directories of /storage/vmware/log along with their permissions as configured on one of my lab VCSAs (would be 7.0 U3 though, but this shouldn't matter).  You would have to create the missing objects in the filesystem manually based on the list or you could create a shell script which assembles the required commands out of this list.

Output would look as follows:

# find /storage/log/vmware/ -ls
   393217      4 drwxr-xr-x  54  root     root         4096 Oct 11 00:00 /storage/log/vmware/
   395749  20760 -rw-r-----   1  root     root     21254180 Oct  6 00:00 /storage/log/vmware/procstate-20231006.gz
   393258      4 drwxr-xr-x   3  root     root         4096 Oct 10 15:10 /storage/log/vmware/vmon
   393333    288 -rw-r--r--   1  root     root       287280 Sep 11 09:49 /storage/log/vmware/vmon/event-publisher.log
   393259      4 drwxr-xr-x   2  vmonapi  vmonapi      4096 Sep 11 09:48 /storage/log/vmware/vmon/vapi
   393412   3676 -rw-r--r--   1  vmonapi  vmonapi   3757576 Oct 11 13:22 /storage/log/vmware/vmon/vapi/vmon-vapi-provider-0.log
   394438      4 -rw-------   1  vmonapi  vmonapi        41 Aug  8 20:56 /storage/log/vmware/vmon/vapi/vapi-runtime.log-4.stdout
   393411      4 -rw-------   1  vmonapi  vmonapi       164 Aug 30 11:59 /storage/log/vmware/vmon/vapi/vapi-runtime.log-0.stderr
   394807      4 -rw-------   1  vmonapi  vmonapi        41 Aug 10 09:15 /storage/log/vmware/vmon/vapi/vapi-runtime.log-1.stdout
   394303      4 -rw-------   1  vmonapi  vmonapi       164 Sep 11 09:48 /storage/log/vmware/vmon/vapi/vapi-runtime.log.stderr

[scugnin · Accepted solution]

I wasted too much time, I remounted a vCenter it was simpler.