VCSA 6.7 U3: vmware-vpxd service fails to start

[aj800]

In trying to patch the VCSA, I normally create a "backup" by cloning the VCSA VM and then creating a snapshot.  In another thread here, I reported the issue I had with cloning (I could not clone it due to an error "vim.fault.NotFound" on the datastore which prevented starting that).  I was able to vMotion the VM to another host in spite of that error, and everything seemed to be functioning fine in vSphere.  So, I went ahead and created the snapshot of the VM and proceeded to perform the update via VAMI.

VAMI downloaded the newest version (53000/U3r), I staged it and then ran the installer.  It rebooted, but the VAMI UI did not comeb back immediately and I had to reload in the browser and log back in.  It showed that the update completed successfully.  So, then I went to load the vSphere UI, which gave me the following error after I believed all the services had started back up:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007faf68050a80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

The CLI shows the following when trying to start all:

root@ourhostname[ ~ ]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting
root@ourhostname[ ~ ]#

The CLI shows the following when trying to start the service directly:

root@ourhostname[ ~ ]# service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
"componentKey": null,
"detail": [
{
"id": "install.ciscommon.service.failstart",
"args": [
"vpxd"
],
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"problemId": null,
"resolution": null
}
Service-control failed. Error: {
"componentKey": null,
"detail": [
{
"id": "install.ciscommon.service.failstart",
"args": [
"vpxd"
],
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"problemId": null,
"resolution": null
}
root@ourhostname[ ~ ]#

I checked the services (watch service-control --status --all) and the vmware-vpxd service had not started, along with a few others I normally see as running (such as the vmware-updatemgr).  I tried rebooting and restarting all services but the same thing happened.

Storage space seems to be fine where only one partition is low or full (but this is normal, from what I've read):

root@ourhostname[ ~ ]# df -h |awk '0+$5 >= 78 {print}'
/dev/mapper/archive_vg-archive 50G 47G 13M 100% /storage/archive

Then I tried to revert back to the snapshot I took (directly from the host it was running on), and the snapshot version did the exact same thing, which leads me to believe it has something to do with the VM and the datastore it runs from.

What can I do to check on this and resolve the issue?  We don't have access to vSphere now and we still need to patch the hosts and the vCSA.