VMware Cloud Community
kontranavoj
Contributor
Contributor

vmware-vpxd service cannot start after importing Machine SSL certificate

Hi people,

I have implemented vCenter Server 6 WEB appliance and tried to import self signed SSL Machine certificate, in order to access on vCenter web interface using that certificate for HTTPS. Certificate was signed by Windows Server 2008 CA with template configured using these instrustions: Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.0 (211.... Also, I tried to import Comodo Trial Positive SSL certificate with same issue - couldn't import it.

I used the VMCA script for certificate management and also tried to import them manually using this procedure: vSphere 6.0 Documentation Center. In both cases, process crashed during vmware-vpxd service restarting process. VMCA script exited with rolling-back old certificates. After trying to manual replace certificates using commands certool and vecs-cli, I tried to start vmware-vpxd service using command service vmware-vpxd start. It produced following output:

virtual:~ # service vmware-vpxd start

vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Waiting for the embedded database to start up: success

Executing pre-startup scripts...

vmware-vpxd: Starting vpxd by administrative request.

success

vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

Waiting for vpxd to initialize: ..........................................................Fri Jun 17 14:19:51 CEST 2016 Captured live core: /var/core/live_core.vpxd.7892.06-17-2016-14-19-51

[INFO] writing vpxd process dump retry:2 Time(Y-M-D H:M:S):2016-06-17 12:19:48

.Fri Jun 17 14:20:13 CEST 2016 Captured live core: /var/core/live_core.vpxd.7892.06-17-2016-14-20-13

[INFO] writing vpxd process dump retry:1 Time(Y-M-D H:M:S):2016-06-17 12:20:01

.failed

failed

vmware-vpxd: vpxd failed to initialize in time.

End of the /var/log/messages log file contains following:

2016-06-17T14:10:04.149368+02:00 virtual vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

2016-06-17T14:10:04.158972+02:00 virtual root: RHTTPPROXY_HTTP_PORT = 80

2016-06-17T14:10:04.168334+02:00 virtual root: RHTTPPROXY_HTTPS_PORT = 443

2016-06-17T14:10:04.748884+02:00 virtual vmware-vpxd: Starting vpxd by administrative request.

2016-06-17T14:10:05.811146+02:00 virtual vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

2016-06-17T14:10:11.068787+02:00 virtual kernel: [ 8262.100377] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32356 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:10:41.045286+02:00 virtual kernel: [ 8292.062481] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32645 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:11:11.392806+02:00 virtual kernel: [ 8322.396547] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=280 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:11:41.380687+02:00 virtual kernel: [ 8352.368229] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=593 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:12:11.508805+02:00 virtual kernel: [ 8382.477595] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=905 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:12:41.124668+02:00 virtual kernel: [ 8412.078362] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1173 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:13:11.068788+02:00 virtual kernel: [ 8442.006593] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1480 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:13:41.052756+02:00 virtual kernel: [ 8471.975786] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1805 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:14:10.948713+02:00 virtual kernel: [ 8501.859385] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2068 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:14:41.048793+02:00 virtual kernel: [ 8531.942485] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2336 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:15:01.102632+02:00 virtual /usr/sbin/cron[6935]: (root) CMD ( /usr/sbin/iiad.sh >/dev/null 2>&1)

2016-06-17T14:15:10.976770+02:00 virtual kernel: [ 8561.853765] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2622 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:15:41.220689+02:00 virtual kernel: [ 8592.076813] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:fc:4d:d4:d2:e5:a9:08:00 SRC=192.168.0.23 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=5745 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:16:10.988809+02:00 virtual kernel: [ 8621.836572] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:3c:97:0e:32:f7:1c:08:00 SRC=192.168.0.80 DST=192.168.0.255 LEN=229 TOS=0x00 PREC=0x00 TTL=128 ID=16631 PROTO=UDP SPT=138 DPT=138 LEN=209

2016-06-17T14:16:41.200814+02:00 virtual kernel: [ 8652.031163] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3292 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:17:11.040715+02:00 virtual kernel: [ 8681.856944] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3595 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:17:41.012706+02:00 virtual kernel: [ 8711.812066] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3808 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:18:10.948718+02:00 virtual kernel: [ 8741.730820] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4046 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:18:40.936986+02:00 virtual kernel: [ 8771.705858] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4360 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:19:11.024790+02:00 virtual kernel: [ 8801.777535] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4647 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:19:41.144770+02:00 virtual kernel: [ 8831.881485] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=5909 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:20:01.147104+02:00 virtual /usr/sbin/cron[13172]: (root) CMD ( /usr/sbin/iiad.sh >/dev/null 2>&1)

2016-06-17T14:20:01.153980+02:00 virtual /usr/sbin/cron[13171]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

2016-06-17T14:20:01.157092+02:00 virtual /usr/sbin/cron[13175]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

2016-06-17T14:20:01.163979+02:00 virtual /usr/sbin/cron[13170]: (root) CMD ( test -x /usr/sbin/cloudvm_ram_size_periodic && /usr/sbin/cloudvm_ram_size_periodic >/dev/null 2>&1)

2016-06-17T14:20:11.664891+02:00 virtual kernel: [ 8862.387284] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6251 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:20:23.877678+02:00 virtual vmware-vpxd: vpxd failed to initialize in time.

2016-06-17T14:20:41.260666+02:00 virtual kernel: [ 8891.967607] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6551 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:21:10.940726+02:00 virtual kernel: [ 8921.632250] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6906 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:21:41.452588+02:00 virtual kernel: [ 8952.121422] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:8d:f4:26:08:00 SRC=192.168.0.51 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=22577 PROTO=UDP SPT=137 DPT=137 LEN=58

2016-06-17T14:22:11.080638+02:00 virtual kernel: [ 8981.739467] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=7505 PROTO=UDP SPT=137 DPT=137 LEN=58

Please help me, tell me where I made mistake. Thank you in advance.

Tags (1)
8 Replies
mccabejr
Contributor
Contributor

Has anyone else run into this issue, or more importantly how to work around it? I'm now in the same boat, and everything I've tried to do to resolve this issue has proven unsuccessful.

Similarly to the original poster, I've used the VMWare Knowledge Base articles for setting up the Certificate Templates, as well as confirmed the Certificate requirements are met - both with the Certificate Authority (CA) chain and the signed Machine SSL certificate.

Any help or guidance at all would be appreciated. Thanks!

Reply
0 Kudos
mccabejr
Contributor
Contributor

Here's some of what I'm seeing in addition to the original poster's snippet:

[ Certificate Manager Failure Notice ]

...

Updated 26 service(s)

Status : 85% Completed [starting services...]

Error while starting services, please see log for more details

Status : 0% Completed [Operation failed, performing automatic rollback]

Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...

Get site nameus : 0% Completed [Rollback Machine SSL Cert...]

Error while reverting certificate for store : MACHINE_SSL_CERT

Rollback Status : 0% Completed [Rollback operation failed]

Error while performing rollback operation, please try Reset operation...

please see /var/log/vmware/vmcad/certificate-manager.log for more information.

---

HOSTNAME:/var/tmp/vmware # less /var/log/vmware/vmcad/certificate-manager.log

INFO:root:Service: vmware-vpxd, Action: start

2016-11-16T21:16:31.950Z  Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']

2016-11-16T21:16:31.950Z  RC = 1

Stdout = vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Waiting for the embedded database to start up: success

Executing pre-startup scripts...

vmware-vpxd: Starting vpxd by administrative request.

success

vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

Waiting for vpxd to initialize: ..........................................................Wed Nov 16 16:16:09 EST 2016 Captured live core: /var/core/live_core.vpxd.26479.11-16-2016-16-16-09

[INFO] writing vpxd process dump retry:2 Time(Y-M-D H:M:S):2016-11-16 21:16:08

.Wed Nov 16 16:16:21 EST 2016 Captured live core: /var/core/live_core.vpxd.26479.11-16-2016-16-16-21

[INFO] writing vpxd process dump retry:1 Time(Y-M-D H:M:S):2016-11-16 21:16:19

.failed

failed

vmware-vpxd: vpxd failed to initialize in time.

vpxd is already starting up. Aborting the request.

Stderr =

2016-11-16T21:16:31.951Z  {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-vpxd, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-vpxd"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-vpxd'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

2016-11-16T21:16:31.958Z ERROR certificate-manager None

2016-11-16T21:16:31.958Z ERROR certificate-manager Error while starting services, please see log for more details

2016-11-16T21:16:31.958Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

2016-11-16T21:16:31.958Z ERROR certificate-manager {

    "resolution": null,

    "detail": [

        {

            "args": [

                "None"

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'None'",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        },

        "Error while starting services, please see log for more details"

    ],

    "componentKey": null,

    "problemId": null

}

2016-11-16T21:16:31.959Z INFO certificate-manager Performing rollback of Machine SSL Cert...

----

----

HOSTNAME:/certs # tail -f /var/log/vmware/vpxd/vpxd.log

2016-11-16T16:06:26.675-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Log path: /var/log/vmware/vpxd

2016-11-16T16:06:26.676-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Initializing SSL

2016-11-16T16:06:26.675-05:00 info vpxd[7FDD23D74700] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-11-16T16:06:26.679-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Vmacore::InitSSL: handshakeTimeoutUs = 120000000

2016-11-16T16:06:26.680-05:00 info vpxd[7FDD23C72700] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-11-16T16:06:26.680-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Daemon] Changed working directory to /var/log/vmware/vpxd

2016-11-16T16:06:26.685-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Starting VMware VirtualCenter 6.0.0 build-3634794

2016-11-16T16:06:26.685-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Log directory: /var/log/vmware/vpxd.

2016-11-16T16:06:26.686-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Main] Account name: (Account Removed)

2016-11-16T16:06:26.744-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Main] [HandleNetworkIdentityChanges] Machine SSL Cert changed

----

----

HOSTNAME:/certs # tail -f /var/log/vmware/vpxd/vmware-vpxd.log

vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Waiting for the embedded database to start up: success

Executing pre-startup scripts...

eth0: error fetching interface information: Device not found

eth0: error fetching interface information: Device not found

eth0: error fetching interface information: Device not found

vmware-vpxd: Starting vpxd by administrative request.

success

vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

Waiting for vpxd to initialize: ....................

----

----

HOSTNAME:/certs # tail -f /var/log/vmware/invsvc/inv-svc.log

2016-11-16T16:06:20.858-05:00 [WrapperListener_start_runner  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

Exception: Hit ExecutionException during SSO-Lookup

...

Caused by: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

...

Caused by: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

...

Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused

...

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused

...

Caused by: java.net.ConnectException: Connection refused

...

2016-11-16T16:06:23.039-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.common.util.impl.DiskSpaceCheckLog  opId=] [/storage/invsvc/xdb/xdb.bootstrap : 4.00]

2016-11-16T16:06:23.045-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.common.util.impl.DiskSpaceCheckLog  opId=] [/var/log/vmware/invsvc : 3.00]

2016-11-16T16:06:23.102-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TaggingAdminRole : 1001 already exists!

2016-11-16T16:06:23.102-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TagManager initialized

2016-11-16T16:06:23.103-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TagManager initialized

2016-11-16T16:06:23.127-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.authorization.impl.provider.AuthQueryHandlerRegistry  opId=] Registering provider query handler for : SRM

2016-11-16T16:06:23.146-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.dataservices.DataService  opId=] Inventory services server starting up...

2016-11-16T16:06:23.234-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.vmomi.server.http.impl.TcServer  opId=] Starting server on [HTTP:0.0.0.0:10080, maxIdleTime: 120000 ms, maxKeepAliveRequests: 100]

2016-11-16T16:06:23.637-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] starting provider pump for: urn:cis.cls:9fd51f44-f5b6-4f45-9461-aec93077bfb8

2016-11-16T16:06:23.638-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] starting provider pump for: c67d8f7e-6714-49fa-bc63-c11b29c70b2f

2016-11-16T16:06:23.639-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.dataservices.DataService  opId=] Inventory services server started.

2016-11-16T16:06:23.639-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.store.impl.QueryPerfLogger  opId=] Server startup time: 493 ms

2016-11-16T16:06:23.883-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

Exception: Hit ExecutionException during SSO-Lookup

...

2016-11-16T16:06:24.646-05:00 [provider-manager-task-68  INFO  com.vmware.vim.query.server.provider.impl.AtomPullProviderImpl  opId=] Attempting VAPI-based login for provider: urn:cis.cls:9fd51f44-f5b6-4f45-9461-aec93077bfb8 to

URL: http://localhost:16666/cls/ - using scheme Http :true

2016-11-16T16:06:24.660-05:00 [provider-manager-task-69  INFO  com.vmware.vim.query.server.provider.impl.AtomPullProviderImpl  opId=] Attempting SOAP-based login for provider: c67d8f7e-6714-49fa-bc63-c11b29c70b2f to URL: http:/

/localhost:8085/sdk

2016-11-16T16:06:24.702-05:00 [provider-manager-task-69  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] Cannot connect to provider: com.vmware.vim.query.server.store.exception.UnauthorizedExc

eption: not connected

2016-11-16T16:06:24.726-05:00 [provider-manager-task-68  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] Cannot connect to provider: com.vmware.vim.query.server.store.exception.UnauthorizedExc

eption: not connected

2016-11-16T16:06:26.944-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

Exception: Hit ExecutionException during SSO-Lookup

...

2016-11-16T16:06:32.082-05:00 [pool-30-thread-1  WARN  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Could not inject STS certificates into adapter servercom.vmware.vim.query.server

.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

2016-11-16T16:06:32.082-05:00 [pool-30-thread-1  INFO  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Failed to fetch trusted certs - Next trusted certs retrieval attempt to happen i

n 10s

2016-11-16T16:06:33.009-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

Exception: Hit ExecutionException during SSO-Lookup

----

Reply
0 Kudos
mmehl
Contributor
Contributor

Did you guys figure this out? I have the same issue.

Reply
0 Kudos
cypherx
Hot Shot
Hot Shot

I'm also having the same issue.  Working certs trusted from our Windows Domain CA expired on 6/13/2018.  I tried to update them, all seemed well until it certificate-manager hung at 85% for a very long time and then eventually rolled back.  Even tried doing option 8, to reset all certificates to factory default - just to rule out a CA configuration issue.  Same thing happens.  Upon further investigation it seems certificate-manager does its thing and replaces certificates, but then it hangs starting the VMware VirtualCenter Server service.  I'm seeing those similar entries in the logs.  Worked with support for a few hours today.  Still standing with a non-functioning vCenter server.  They took the core .dmp files that the service was generating for analysis.  I'm not sure how long this will take.  It took 2 days just to get support to remote on and take a look at the problem.

Reply
0 Kudos
sureshthirumala
Contributor
Contributor

any luck on this from VMware support? i am also having same issues.

Reply
0 Kudos
hermanc01
Enthusiast
Enthusiast

Was Support able to get this resolved for you?  I'm having the exact same issue.

Reply
0 Kudos
Nancorb
Contributor
Contributor

I ran into this same issue, using the certificate-manager to replace a self-signed SSL with a Digicert certificate.  It got to the end of the process, and then failed and rolled back because vmware-vpxd could not start. 

It turned out that I was missing a file.  The digicert email had 3 items I needed:

1. attachment, cert.cer

2. link to zip file 1555012429.zip, which contained the IntermediateCA.cer and ssl_certificate.cer

3. link to download the Root certificate:  DigiCert_Global_Root_CA.cer

I was missing the root certificate, so the certificate-manager was mistaking the IntermediateCA.cer for the root cert, and it kept failing.

I had a key file I had generated earlier.  vmca_issued_key.key

So, first, create a chain.pem:

cat DigiCert_Global_Root_CA.cer IntermediateCA.cer > chain.pem

Then run the certificate-manager again.

Select option 1

Select option 2

Please provide valid custom certificate for Machine SSL.

File : /certificate/cert.cer

Please provide valid custom key for Machine SSL.

File : /certificate/vmca_issued_key.key

Please provide the signing certificate of the Machine SSL certificate

File : /certificate/chain.pem

ApprehensiveEdg
Contributor
Contributor

hello,

i encountered the same issue after having attempted to import some certs to secure the web frontend on my vcsa 6.5.

when trying to pull up the page, i would get an error page with messages like:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f009c095810] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

running service-control --status would return:

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

running service-control --stop --all followed by service-control --start --all returned:

2019-07-02T16:39:01.784Z [main  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : user@xxx.tld

com.vmware.cis.server.ssoauthentication.exception.ServiceFaultException: com.vmware.cis.server.authentication.exception.TokenProviderException: com.vmware.cis.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

        at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.setupAdminClientInternal(AdminClientWrapperImpl.java:93)

i was able to resolve the issue by making the vcsa regenerate its default certs.

to do this, run shell to bring up bash, then run /usr/lib/vmware-vmca/bin/certificate-manager and choose option 8, for "Reset all Certificates".

it asks:

Do you wish to generate all certificates using configuration file : Option[Y/N] ?

i'm not sure what it means by "using configuration file", the wording is very unclear. i just input Y and followed the rest of the prompts, letting it fill in default values except for the fqdn and device name. i rebooted the device and after letting it sit for ~20 minutes while it started itself up, i was able to get back into the web client.

hope this helps some future googler, because i haven't seen any other solutions posted around.