artem060593
Contributor
Contributor

VCenter Appliance VPXD service does not Start with Error

Hello. After rebooting, I could not connect to Vcentr every time I received a 503 error When I start vmware-vpxd manually, I get the following error. What is Stderr = Key not found: vpxd / secureSoapPort? How to fix  and what to do? Help me please! Thank!

service-control --start vmware-vpxd

INFO:root:Service: vmware-vpxd, Action: start

Service: vmware-vpxd, Action: start

2019-01-28T19:29:57.828Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']

2019-01-28T19:29:57.958Z   Done running command

2019-01-28T19:29:57.958Z   Running command: ['/sbin/service', u'vmware-vpxd', 'status']

2019-01-28T19:29:58.532Z   Done running command

2019-01-28T19:29:58.532Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-vpxd', 'on']

2019-01-28T19:29:58.595Z   Done running command

2019-01-28T19:29:58.595Z   Running command: ['/sbin/service', u'vmware-vpxd', 'start']

2019-01-28T19:30:01.535Z   Done running command

2019-01-28T19:30:01.536Z   Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']

2019-01-28T19:30:01.536Z   RC = 1

Stdout = Key not found: vpxd/secureSoapPort

vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Waiting for the embedded database to start up: success

Executing pre-startup scripts...

vmware-vpxd: Starting vpxd by administrative request.

success

vmware-vpxd: Waiting for vpxd to start listening for requests on

Waiting for vpxd to initialize: Usage: wait_for_vpxd.py [options]

wait_for_vpxd.py: error: -p option requires an argument

failed

failed

vmware-vpxd: vpxd failed to initialize in time.

vpxd is already starting up. Aborting the request.

Stderr = Key not found: vpxd/secureSoapPort

2019-01-28T19:30:01.536Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: Key not found: vpxd/secureSoapPort\n"

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Сommand: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: Key not found: vpxd/                                                                      secureSoapPort\n'",

            "translatable": "An error occurred while invoking external command :'%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-vpxd, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-vpxd"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-vpxd'                                                                      ",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-vpxd, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-vpxd"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-vpxd'                                                                      ",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

0 Kudos
16 Replies
Vijay2027
Expert
Expert

At the time of vpxd service crash what do you see in vpxd.log?

Reproduce the issue and attach vpxd.log to this thread.

Log location: /var/log/vmware/vpxd

0 Kudos
artem060593
Contributor
Contributor

Thanks for the response! writes nothing there. The last entry in the 25th, before the first reboot, after which Vcentr did not start

2019-01-25T22:03:35.855Z info vpxd[7F6CB2D51700] [Originator@6876 sub=vpxLro opID=opId-2d30bba9-80b5-46ab-9e8b-c97898f10914-07-25] [VpxLRO] -- FINISH task-internal-2177093

2019-01-25T22:03:36.125Z error vpxd[7F6CB33DE700] [Originator@6876 sub=SoapAdapter.HTTPService] Failed to read request; stream: <io_obj p:0x00007f6ccff0b880, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2019-01-25T22:03:36.126Z error vpxd[7F6CB33DE700] [Originator@6876 sub=SoapAdapter.HTTPService] Failed to read request; stream: <io_obj p:0x00007f6cd16416a0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2019-01-25T22:03:36.126Z error vpxd[7F6CB33DE700] [Originator@6876 sub=HttpSvc.HTTPService] Failed to read request; stream: <io_obj p:0x00007f6cac4cbfc8, h:-1, <UNIX ''>, <UNIX ''>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

0 Kudos
Vijay2027
Expert
Expert

# df -Th

# service-control --status --all

Please share output

0 Kudos
artem060593
Contributor
Contributor

A number of services except vmware-vpxd are not running, try to start everything one by one?

Clip2net_190128231846.png

Clip2net_190128231813.png

0 Kudos
Vijay2027
Expert
Expert

File-systems looks good...

- Start vpxd service.

- take a duplicate putty session to vcsa

- tail -f vpxd.log and see if the log file is updated with latest entries.

0 Kudos
artem060593
Contributor
Contributor

Nothing .. writes nothing at all to the log

0 Kudos
Vijay2027
Expert
Expert

I wanted to verify if the config file is fine.

Attach vpxd.cfg file from /etc/vmware-vpx location.

0 Kudos
artem060593
Contributor
Contributor

file contents  vpxd.cfg :

kp-srv-vc:/etc/vmware-vpx # cat vpxd.cfg

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><healthStatus schemaVersion="1.0" xmlns="http://www.vmware.com/cis/cm/common/jaxb/healthstatus"><status>GREEN</status><message messageKey="cis.vpg.health.healthy" defaultMessage="VMware Postgres service is healthy."></message><expiration>2019-01-25T22:05:15.000Z</expiration></healthStatus>  <support>

      <scriptDirectory>/usr/lib/vmware-vpx/</scriptDirectory>

    </support>

    <vdb>

      <space>

        <alertMB>1000</alertMB>

        <warningMB>5000</warningMB>

      </space>

    </vdb>

    <vecs>

      <storeName>vpxd</storeName>

    </vecs>

    <vsan>

      <dynmgr>

        <enable>true</enable>

      </dynmgr>

    </vsan>

    <workflow>

      <methods id="PowerOnVM_Task">true</methods>

    </workflow>

  </vpxd>

  <workflow>

    <port>8088</port>

  </workflow>

</config>

0 Kudos
Vijay2027
Expert
Expert

Okay..Usually vpxd.cfg file has approximately 130-150 lines.  We will have to figure out a way to fix the config file.

I am checking in my lab if it is possible to fix vpxd.cfg.

0 Kudos
artem060593
Contributor
Contributor

Thank!

I need a working vpxd.cfg? but it contains unique data for my system, isn't it? then how can I restore it? (

0 Kudos
Vijay2027
Expert
Expert

Yes it unique to your system. There are close to 10 lines where you will have to edit the file specific to your vsphere environment.

How many hosts are being managed by this vc?

Do you have a backup of this vc?

0 Kudos
artem060593
Contributor
Contributor

5 hosts. There is no backup. I found an article for Windows:

VMware Knowledge Base 

maybe there is something similar for Server Appliance?

0 Kudos
Vijay2027
Expert
Expert

I am not aware if there is any such config file in vcsa. There are 2 fields in vpxd.cfg file which I am not sure how to customize to your environment.

    <DB>

      <key_2>vc</key_2>

      <key_3>*p3i3Sdj/dxABy7IES3**************************pWm4HBX2c9BBCl</key_3>

    </DB>

    <key_EvaluationExpiryDate>AQD+ygg***********************************lYA/Rik29KAhMQ==</key_EvaluationExpiryDate>

  </registry>

  <task>

One option is to

- take a backup of DB

- Deploy a new vcsa with the same build, DNS

- Restore DB

VMware Knowledge Base

0 Kudos
Vijay2027
Expert
Expert

Use winscp to copy files off of vcsa

VMware Knowledge Base

0 Kudos
artem060593
Contributor
Contributor

It sounds interesting) Tomorrow I will try. Thank you very much for your help and for your time!

0 Kudos
Vijay2027
Expert
Expert

were you able to fix this?

0 Kudos