8 Replies Latest reply on Jul 2, 2019 10:52 AM by ApprehensiveEdge6

    vmware-vpxd service cannot start after importing Machine SSL certificate

    kontranavoj Lurker

      Hi people,

       

      I have implemented vCenter Server 6 WEB appliance and tried to import self signed SSL Machine certificate, in order to access on vCenter web interface using that certificate for HTTPS. Certificate was signed by Windows Server 2008 CA with template configured using these instrustions: Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.0 (2112009) | VMware KB. Also, I tried to import Comodo Trial Positive SSL certificate with same issue - couldn't import it.

       

      I used the VMCA script for certificate management and also tried to import them manually using this procedure: vSphere 6.0 Documentation Center. In both cases, process crashed during vmware-vpxd service restarting process. VMCA script exited with rolling-back old certificates. After trying to manual replace certificates using commands certool and vecs-cli, I tried to start vmware-vpxd service using command service vmware-vpxd start. It produced following output:

       

      virtual:~ # service vmware-vpxd start

      vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

      Waiting for the embedded database to start up: success

      Executing pre-startup scripts...

      vmware-vpxd: Starting vpxd by administrative request.

      success

      vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

      Waiting for vpxd to initialize: ..........................................................Fri Jun 17 14:19:51 CEST 2016 Captured live core: /var/core/live_core.vpxd.7892.06-17-2016-14-19-51

      [INFO] writing vpxd process dump retry:2 Time(Y-M-D H:M:S):2016-06-17 12:19:48

      .Fri Jun 17 14:20:13 CEST 2016 Captured live core: /var/core/live_core.vpxd.7892.06-17-2016-14-20-13

      [INFO] writing vpxd process dump retry:1 Time(Y-M-D H:M:S):2016-06-17 12:20:01

      .failed

      failed

      vmware-vpxd: vpxd failed to initialize in time.

       

      End of the /var/log/messages log file contains following:

       

      2016-06-17T14:10:04.149368+02:00 virtual vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

      2016-06-17T14:10:04.158972+02:00 virtual root: RHTTPPROXY_HTTP_PORT = 80

      2016-06-17T14:10:04.168334+02:00 virtual root: RHTTPPROXY_HTTPS_PORT = 443

      2016-06-17T14:10:04.748884+02:00 virtual vmware-vpxd: Starting vpxd by administrative request.

      2016-06-17T14:10:05.811146+02:00 virtual vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

      2016-06-17T14:10:11.068787+02:00 virtual kernel: [ 8262.100377] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32356 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:10:41.045286+02:00 virtual kernel: [ 8292.062481] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32645 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:11:11.392806+02:00 virtual kernel: [ 8322.396547] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=280 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:11:41.380687+02:00 virtual kernel: [ 8352.368229] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=593 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:12:11.508805+02:00 virtual kernel: [ 8382.477595] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=905 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:12:41.124668+02:00 virtual kernel: [ 8412.078362] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1173 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:13:11.068788+02:00 virtual kernel: [ 8442.006593] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1480 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:13:41.052756+02:00 virtual kernel: [ 8471.975786] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=1805 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:14:10.948713+02:00 virtual kernel: [ 8501.859385] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2068 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:14:41.048793+02:00 virtual kernel: [ 8531.942485] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2336 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:15:01.102632+02:00 virtual /usr/sbin/cron[6935]: (root) CMD ( /usr/sbin/iiad.sh >/dev/null 2>&1)

      2016-06-17T14:15:10.976770+02:00 virtual kernel: [ 8561.853765] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=2622 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:15:41.220689+02:00 virtual kernel: [ 8592.076813] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:fc:4d:d4:d2:e5:a9:08:00 SRC=192.168.0.23 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=5745 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:16:10.988809+02:00 virtual kernel: [ 8621.836572] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:3c:97:0e:32:f7:1c:08:00 SRC=192.168.0.80 DST=192.168.0.255 LEN=229 TOS=0x00 PREC=0x00 TTL=128 ID=16631 PROTO=UDP SPT=138 DPT=138 LEN=209

      2016-06-17T14:16:41.200814+02:00 virtual kernel: [ 8652.031163] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3292 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:17:11.040715+02:00 virtual kernel: [ 8681.856944] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3595 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:17:41.012706+02:00 virtual kernel: [ 8711.812066] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=3808 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:18:10.948718+02:00 virtual kernel: [ 8741.730820] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4046 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:18:40.936986+02:00 virtual kernel: [ 8771.705858] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4360 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:19:11.024790+02:00 virtual kernel: [ 8801.777535] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=4647 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:19:41.144770+02:00 virtual kernel: [ 8831.881485] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=5909 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:20:01.147104+02:00 virtual /usr/sbin/cron[13172]: (root) CMD ( /usr/sbin/iiad.sh >/dev/null 2>&1)

      2016-06-17T14:20:01.153980+02:00 virtual /usr/sbin/cron[13171]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

      2016-06-17T14:20:01.157092+02:00 virtual /usr/sbin/cron[13175]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

      2016-06-17T14:20:01.163979+02:00 virtual /usr/sbin/cron[13170]: (root) CMD ( test -x /usr/sbin/cloudvm_ram_size_periodic && /usr/sbin/cloudvm_ram_size_periodic >/dev/null 2>&1)

      2016-06-17T14:20:11.664891+02:00 virtual kernel: [ 8862.387284] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6251 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:20:23.877678+02:00 virtual vmware-vpxd: vpxd failed to initialize in time.

      2016-06-17T14:20:41.260666+02:00 virtual kernel: [ 8891.967607] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6551 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:21:10.940726+02:00 virtual kernel: [ 8921.632250] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=6906 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:21:41.452588+02:00 virtual kernel: [ 8952.121422] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:8d:f4:26:08:00 SRC=192.168.0.51 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=22577 PROTO=UDP SPT=137 DPT=137 LEN=58

      2016-06-17T14:22:11.080638+02:00 virtual kernel: [ 8981.739467] IPfilter Dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:24:7c:8a:09:08:00 SRC=192.168.0.94 DST=192.168.0.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=7505 PROTO=UDP SPT=137 DPT=137 LEN=58

       

       

      Please help me, tell me where I made mistake. Thank you in advance.

        • 1. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
          mccabejr Lurker

          Has anyone else run into this issue, or more importantly how to work around it? I'm now in the same boat, and everything I've tried to do to resolve this issue has proven unsuccessful.

           

          Similarly to the original poster, I've used the VMWare Knowledge Base articles for setting up the Certificate Templates, as well as confirmed the Certificate requirements are met - both with the Certificate Authority (CA) chain and the signed Machine SSL certificate.

           

          Any help or guidance at all would be appreciated. Thanks!

          • 2. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
            mccabejr Lurker

            Here's some of what I'm seeing in addition to the original poster's snippet:

             

            [ Certificate Manager Failure Notice ]

            ...

            Updated 26 service(s)

            Status : 85% Completed [starting services...]

            Error while starting services, please see log for more details

            Status : 0% Completed [Operation failed, performing automatic rollback]

             

            Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

             

            Performing rollback of Machine SSL Cert...

            Get site nameus : 0% Completed [Rollback Machine SSL Cert...]

             

            Error while reverting certificate for store : MACHINE_SSL_CERT

            Rollback Status : 0% Completed [Rollback operation failed]

             

            Error while performing rollback operation, please try Reset operation...

             

            please see /var/log/vmware/vmcad/certificate-manager.log for more information.

             

            ---

            HOSTNAME:/var/tmp/vmware # less /var/log/vmware/vmcad/certificate-manager.log

            INFO:root:Service: vmware-vpxd, Action: start

            2016-11-16T21:16:31.950Z  Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']

            2016-11-16T21:16:31.950Z  RC = 1

            Stdout = vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

            Waiting for the embedded database to start up: success

            Executing pre-startup scripts...

            vmware-vpxd: Starting vpxd by administrative request.

            success

            vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

            Waiting for vpxd to initialize: ..........................................................Wed Nov 16 16:16:09 EST 2016 Captured live core: /var/core/live_core.vpxd.26479.11-16-2016-16-16-09

            [INFO] writing vpxd process dump retry:2 Time(Y-M-D H:M:S):2016-11-16 21:16:08

            .Wed Nov 16 16:16:21 EST 2016 Captured live core: /var/core/live_core.vpxd.26479.11-16-2016-16-16-21

            [INFO] writing vpxd process dump retry:1 Time(Y-M-D H:M:S):2016-11-16 21:16:19

            .failed

            failed

            vmware-vpxd: vpxd failed to initialize in time.

            vpxd is already starting up. Aborting the request.

             

            Stderr =

            2016-11-16T21:16:31.951Z  {

                "resolution": null,

                "detail": [

                    {

                        "args": [

                            "Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: "

                        ],

                        "id": "install.ciscommon.command.errinvoke",

                        "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: '",

                        "translatable": "An error occurred while invoking external command : '%(0)s'"

                    }

                ],

                "componentKey": null,

                "problemId": null

            }

            ERROR:root:Unable to start service vmware-vpxd, Exception: {

                "resolution": null,

                "detail": [

                    {

                        "args": [

                            "vmware-vpxd"

                        ],

                        "id": "install.ciscommon.service.failstart",

                        "localized": "An error occurred while starting service 'vmware-vpxd'",

                        "translatable": "An error occurred while starting service '%(0)s'"

                    }

                ],

                "componentKey": null,

                "problemId": null

            }

            2016-11-16T21:16:31.958Z ERROR certificate-manager None

            2016-11-16T21:16:31.958Z ERROR certificate-manager Error while starting services, please see log for more details

            2016-11-16T21:16:31.958Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

            2016-11-16T21:16:31.958Z ERROR certificate-manager {

                "resolution": null,

                "detail": [

                    {

                        "args": [

                            "None"

                        ],

                        "id": "install.ciscommon.command.errinvoke",

                        "localized": "An error occurred while invoking external command : 'None'",

                        "translatable": "An error occurred while invoking external command : '%(0)s'"

                    },

                    "Error while starting services, please see log for more details"

                ],

                "componentKey": null,

                "problemId": null

            }

            2016-11-16T21:16:31.959Z INFO certificate-manager Performing rollback of Machine SSL Cert...

            ----

             

             

            ----

            HOSTNAME:/certs # tail -f /var/log/vmware/vpxd/vpxd.log

            2016-11-16T16:06:26.675-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Log path: /var/log/vmware/vpxd

            2016-11-16T16:06:26.676-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Initializing SSL

            2016-11-16T16:06:26.675-05:00 info vpxd[7FDD23D74700] [Originator@6876 sub=ThreadPool] Thread enlisted

            2016-11-16T16:06:26.679-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Vmacore::InitSSL: handshakeTimeoutUs = 120000000

            2016-11-16T16:06:26.680-05:00 info vpxd[7FDD23C72700] [Originator@6876 sub=ThreadPool] Thread enlisted

            2016-11-16T16:06:26.680-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Daemon] Changed working directory to /var/log/vmware/vpxd

            2016-11-16T16:06:26.685-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Starting VMware VirtualCenter 6.0.0 build-3634794

            2016-11-16T16:06:26.685-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Default] Log directory: /var/log/vmware/vpxd.

            2016-11-16T16:06:26.686-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Main] Account name: (Account Removed)

            2016-11-16T16:06:26.744-05:00 info vpxd[7FDD34CDF7A0] [Originator@6876 sub=Main] [HandleNetworkIdentityChanges] Machine SSL Cert changed

            ----

             

             

            ----

            HOSTNAME:/certs # tail -f /var/log/vmware/vpxd/vmware-vpxd.log

            vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

            Waiting for the embedded database to start up: success

            Executing pre-startup scripts...

            eth0: error fetching interface information: Device not found

            eth0: error fetching interface information: Device not found

            eth0: error fetching interface information: Device not found

            vmware-vpxd: Starting vpxd by administrative request.

            success

            vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

            Waiting for vpxd to initialize: ....................

            ----

             

             

             

             

            ----

            HOSTNAME:/certs # tail -f /var/log/vmware/invsvc/inv-svc.log

            2016-11-16T16:06:20.858-05:00 [WrapperListener_start_runner  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

            com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

            Exception: Hit ExecutionException during SSO-Lookup

            ...

            Caused by: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

            ...

            Caused by: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

            ...

            Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused

            ...

            Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused

            ...

            Caused by: java.net.ConnectException: Connection refused

            ...

            2016-11-16T16:06:23.039-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.common.util.impl.DiskSpaceCheckLog  opId=] [/storage/invsvc/xdb/xdb.bootstrap : 4.00]

            2016-11-16T16:06:23.045-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.common.util.impl.DiskSpaceCheckLog  opId=] [/var/log/vmware/invsvc : 3.00]

            2016-11-16T16:06:23.102-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TaggingAdminRole : 1001 already exists!

            2016-11-16T16:06:23.102-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TagManager initialized

            2016-11-16T16:06:23.103-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.tagging2.vmodl.ManagerInitializer  opId=] TagManager initialized

            2016-11-16T16:06:23.127-05:00 [WrapperListener_start_runner  INFO  com.vmware.cis.authorization.impl.provider.AuthQueryHandlerRegistry  opId=] Registering provider query handler for : SRM

            2016-11-16T16:06:23.146-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.dataservices.DataService  opId=] Inventory services server starting up...

            2016-11-16T16:06:23.234-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.vmomi.server.http.impl.TcServer  opId=] Starting server on [HTTP:0.0.0.0:10080, maxIdleTime: 120000 ms, maxKeepAliveRequests: 100]

            2016-11-16T16:06:23.637-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] starting provider pump for: urn:cis.cls:9fd51f44-f5b6-4f45-9461-aec93077bfb8

            2016-11-16T16:06:23.638-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] starting provider pump for: c67d8f7e-6714-49fa-bc63-c11b29c70b2f

            2016-11-16T16:06:23.639-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.dataservices.DataService  opId=] Inventory services server started.

            2016-11-16T16:06:23.639-05:00 [WrapperListener_start_runner  INFO  com.vmware.vim.query.server.store.impl.QueryPerfLogger  opId=] Server startup time: 493 ms

            2016-11-16T16:06:23.883-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

            com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

            Exception: Hit ExecutionException during SSO-Lookup

            ...

            2016-11-16T16:06:24.646-05:00 [provider-manager-task-68  INFO  com.vmware.vim.query.server.provider.impl.AtomPullProviderImpl  opId=] Attempting VAPI-based login for provider: urn:cis.cls:9fd51f44-f5b6-4f45-9461-aec93077bfb8 to

            URL: http://localhost:16666/cls/ - using scheme Http :true

            2016-11-16T16:06:24.660-05:00 [provider-manager-task-69  INFO  com.vmware.vim.query.server.provider.impl.AtomPullProviderImpl  opId=] Attempting SOAP-based login for provider: c67d8f7e-6714-49fa-bc63-c11b29c70b2f to URL: http:/

            /localhost:8085/sdk

            2016-11-16T16:06:24.702-05:00 [provider-manager-task-69  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] Cannot connect to provider: com.vmware.vim.query.server.store.exception.UnauthorizedExc

            eption: not connected

            2016-11-16T16:06:24.726-05:00 [provider-manager-task-68  INFO  com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] Cannot connect to provider: com.vmware.vim.query.server.store.exception.UnauthorizedExc

            eption: not connected

            2016-11-16T16:06:26.944-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

            com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

            Exception: Hit ExecutionException during SSO-Lookup

            ...

            2016-11-16T16:06:32.082-05:00 [pool-30-thread-1  WARN  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Could not inject STS certificates into adapter servercom.vmware.vim.query.server

            .ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

            2016-11-16T16:06:32.082-05:00 [pool-30-thread-1  INFO  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Failed to fetch trusted certs - Next trusted certs retrieval attempt to happen i

            n 10s

            2016-11-16T16:06:33.009-05:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : (UPN Replaced)

            com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFound

            Exception: Hit ExecutionException during SSO-Lookup

            ----

            • 3. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
              mmehl Lurker

              Did you guys figure this out? I have the same issue.

              • 4. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
                cypherx Hot Shot

                I'm also having the same issue.  Working certs trusted from our Windows Domain CA expired on 6/13/2018.  I tried to update them, all seemed well until it certificate-manager hung at 85% for a very long time and then eventually rolled back.  Even tried doing option 8, to reset all certificates to factory default - just to rule out a CA configuration issue.  Same thing happens.  Upon further investigation it seems certificate-manager does its thing and replaces certificates, but then it hangs starting the VMware VirtualCenter Server service.  I'm seeing those similar entries in the logs.  Worked with support for a few hours today.  Still standing with a non-functioning vCenter server.  They took the core .dmp files that the service was generating for analysis.  I'm not sure how long this will take.  It took 2 days just to get support to remote on and take a look at the problem.

                • 5. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
                  sureshthirumalapudi Lurker

                  any luck on this from VMware support? i am also having same issues.

                  • 6. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
                    hermanc01 Enthusiast

                    Was Support able to get this resolved for you?  I'm having the exact same issue.

                    • 7. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
                      Nancorb Lurker

                      I ran into this same issue, using the certificate-manager to replace a self-signed SSL with a Digicert certificate.  It got to the end of the process, and then failed and rolled back because vmware-vpxd could not start. 

                       

                      It turned out that I was missing a file.  The digicert email had 3 items I needed:

                      1. attachment, cert.cer

                      2. link to zip file 1555012429.zip, which contained the IntermediateCA.cer and ssl_certificate.cer

                      3. link to download the Root certificate:  DigiCert_Global_Root_CA.cer

                       

                      I was missing the root certificate, so the certificate-manager was mistaking the IntermediateCA.cer for the root cert, and it kept failing.

                       

                      I had a key file I had generated earlier.  vmca_issued_key.key

                       

                      So, first, create a chain.pem:

                       

                      cat DigiCert_Global_Root_CA.cer IntermediateCA.cer > chain.pem

                       

                      Then run the certificate-manager again.

                       

                      Select option 1

                      Select option 2

                      Please provide valid custom certificate for Machine SSL.

                      File : /certificate/cert.cer

                       

                      Please provide valid custom key for Machine SSL.

                      File : /certificate/vmca_issued_key.key

                       

                      Please provide the signing certificate of the Machine SSL certificate

                      File : /certificate/chain.pem

                       

                       

                       

                       

                       

                       

                      • 8. Re: vmware-vpxd service cannot start after importing Machine SSL certificate
                        ApprehensiveEdge6 Lurker

                        hello,

                         

                        i encountered the same issue after having attempted to import some certs to secure the web frontend on my vcsa 6.5.

                         

                        when trying to pull up the page, i would get an error page with messages like:

                         

                        503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f009c095810] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

                         

                        running service-control --status would return:

                         

                        Running:

                        applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

                        Stopped:

                        pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

                         

                        running service-control --stop --all followed by service-control --start --all returned:

                         

                        2019-07-02T16:39:01.784Z [main  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : user@xxx.tld

                        com.vmware.cis.server.ssoauthentication.exception.ServiceFaultException: com.vmware.cis.server.authentication.exception.TokenProviderException: com.vmware.cis.server.ssoauthentication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

                                at com.vmware.cis.server.ssoauthentication.impl.AdminClientWrapperImpl.setupAdminClientInternal(AdminClientWrapperImpl.java:93)

                         

                        i was able to resolve the issue by making the vcsa regenerate its default certs.

                         

                        to do this, run shell to bring up bash, then run /usr/lib/vmware-vmca/bin/certificate-manager and choose option 8, for "Reset all Certificates".

                        it asks:

                         

                        Do you wish to generate all certificates using configuration file : Option[Y/N] ?

                         

                        i'm not sure what it means by "using configuration file", the wording is very unclear. i just input Y and followed the rest of the prompts, letting it fill in default values except for the fqdn and device name. i rebooted the device and after letting it sit for ~20 minutes while it started itself up, i was able to get back into the web client.

                         

                        hope this helps some future googler, because i haven't seen any other solutions posted around.