VCSA Failed to start after Restart of the VCenter VM

[Lokesh_gs]

After every restart i need to enter the management ip manually for my vcenter vm .

Status of the services

root@bnps-vcsa [ ~ ]# service-control --status

Stopped:

lwsmd pschealth vmafdd vmcad vmcam vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps vsphere-client vsphere-ui

Running:

applmgmt vmware-statsmonitor vmware-vcha vmware-vmon

Error starting the Vsphere UI

root@bnps-vcsa [ ~ ]# service-control --start vsphere-ui

Operation not cancellable. Please wait for it to finish...

Performing start operation on service vsphere-ui...

Error executing start on service vsphere-ui. Details {

    "resolution": null,

    "problemId": null,

    "detail": [

        {

            "translatable": "An error occurred while starting service '%(0)s'",

            "localized": "An error occurred while starting service 'vsphere-ui'",

            "args": [

                "vsphere-ui"

            ],

            "id": "install.ciscommon.service.failstart"

        }

    ],

    "componentKey": null

}

Service-control failed. Error: {

    "resolution": null,

    "problemId": null,

    "detail": [

        {

            "translatable": "An error occurred while starting service '%(0)s'",

            "localized": "An error occurred while starting service 'vsphere-ui'",

            "args": [

                "vsphere-ui"

            ],

            "id": "install.ciscommon.service.failstart"

        }

    ],

    "componentKey": null

}

I tried the vmon Cli utility also

root@bnps-vcsa [ /usr/lib/vmware-vmon ]# ./vmon-cli -h

Usage: ./vmon-cli [options]

Options:

  -c, --vmonserver <file>  vMon server connection path.

  -i, --start <serviceid>  Start service.

  -k, --stop <serviceid>   Stop Service.

  -r, --restart <serviceid> Restart Service.

  -s, --status <serviceid> Get Service status.

  -d, --dumpcore <serviceid> Dump service process core.

  -U, --update <serviceid> Update service state.

  -S  --starttype <MANUAL|AUTOMATIC|DISABLED> Requires -U option.

  -R  --runasuser <Username> Requires -U option.

  -l, --list               List registered services.

  --list-dependencies <serviceid> List service dependencies.

  --batchstart <profile> Start all services in profile.

  --batchstop  <profile> Stop all services in profile.

  --refreshhealth          Refreshes health state of all services.

  --ignorefail             Ignore batch operation failures.

  --dumpstat             Dump stat info without reboot.

  -h                       Print help information.

root@bnps-vcsa [ /usr/lib/vmware-vmon ]# ./vmon-cli -r vpxd

Restart service request failed. Error: A system error occurred. Check logs for details

root@bnps-vcsa [ /usr/lib/vmware-vmon ]# ./vmon-cli -s vpxd

Name: vpxd

Starttype: AUTOMATIC

RunState: STOPPED

RunAsUser: root

CurrentRunStateDuration(ms): 10111657

HealthState: UNHEALTHY

FailStop: TRUE

MainProcessId: N/A

root@bnps-vcsa [ /usr/lib/vmware-vmon ]# df -h

Filesystem                                                      Size  Used Avail Use% Mounted on

devtmpfs                                                         7.9G     0  7.9G   0% /dev

tmpfs                                                               7.9G     0  7.9G   0% /dev/shm

tmpfs                                                               7.9G  664K  7.9G   1% /run

tmpfs                                                                7.9G     0  7.9G   0% /sys/fs/cgroup

/dev/sda3                                                         11G  6.3G  3.9G  62% /

tmpfs                                                                7.9G  144K  7.9G   1% /tmp

/dev/mapper/netdump_vg-netdump                 985M  1.3M  916M   1% /storage/netdump

/dev/mapper/imagebuilder_vg-imagebuilder    9.8G   23M  9.2G   1% /storage/imagebuilder

/dev/mapper/autodeploy_vg-autodeploy          9.8G   23M  9.2G   1% /storage/autodeploy

/dev/mapper/core_vg-core                               50G  3.6G   44G   8% /storage/core

/dev/mapper/updatemgr_vg-updatemgr         99G  102M   94G   1% /storage/updatemgr

/dev/mapper/seat_vg-seat                              25G  374M   23G   2% /storage/seat

/dev/mapper/dblog_vg-dblog                           15G  262M   14G   2% /storage/dblog

/dev/mapper/archive_vg-archive                      50G  1.4G   46G   3% /storage/archive

/dev/mapper/log_vg-log                                    9.8G  1.7G  7.6G  18% /storage/log

/dev/sda1                                                          120M   31M   81M  28% /boot

/dev/mapper/db_vg-db                                      9.8G  195M  9.1G   3% /storage/db

VPXD Log

root@bnps-vcsa [ /var/log/vmware/vpxd ]# vi vpxd.log

2020-05-24T15:58:50.634+08:00 Section for VMware VirtualCenter, pid=14083, version=6.7.0, build=10244857, option=Release

2020-05-24T15:58:50.634+08:00 info vpxd[51812] [Originator@6876 sub=vpxLro opID=vb-64555:auto-name-37131:01-0] [VpxLRO] -- BEGIN lro-1333550 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:50.635+08:00 info vpxd[51812] [Originator@6876 sub=vpxLro opID=vb-64555:auto-name-37131:01-0] [VpxLRO] -- FINISH lro-1333550

2020-05-24T15:58:50.639+08:00 info vpxd[14168] [Originator@6876 sub=vpxLro opID=vb-64556:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-78] [VpxLRO] -- BEGIN lro-1333551 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:50.639+08:00 info vpxd[14168] [Originator@6876 sub=vpxLro opID=vb-64556:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-78] [VpxLRO] -- FINISH lro-1333551

2020-05-24T15:58:50.849+08:00 info vpxd[14221] [Originator@6876 sub=vpxLro opID=vb-64559:auto-name-37133:01-b] [VpxLRO] -- BEGIN lro-1333554 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:50.849+08:00 info vpxd[14221] [Originator@6876 sub=vpxLro opID=vb-64559:auto-name-37133:01-b] [VpxLRO] -- FINISH lro-1333554

2020-05-24T15:58:50.853+08:00 info vpxd[14122] [Originator@6876 sub=vpxLro opID=vb-64560:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-0] [VpxLRO] -- BEGIN lro-1333555 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:50.853+08:00 info vpxd[14122] [Originator@6876 sub=vpxLro opID=vb-64560:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-0] [VpxLRO] -- FINISH lro-1333555

2020-05-24T15:58:51.064+08:00 info vpxd[14214] [Originator@6876 sub=vpxLro opID=vb-64563:auto-name-37135:01-8b] [VpxLRO] -- BEGIN lro-1333557 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.064+08:00 info vpxd[14214] [Originator@6876 sub=vpxLro opID=vb-64563:auto-name-37135:01-8b] [VpxLRO] -- FINISH lro-1333557

2020-05-24T15:58:51.068+08:00 info vpxd[14216] [Originator@6876 sub=vpxLro opID=vb-64564:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-0] [VpxLRO] -- BEGIN lro-1333558 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.068+08:00 info vpxd[14216] [Originator@6876 sub=vpxLro opID=vb-64564:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-0] [VpxLRO] -- FINISH lro-1333558

2020-05-24T15:58:51.277+08:00 info vpxd[51826] [Originator@6876 sub=vpxLro opID=vb-64567:auto-name-37137:01-f6] [VpxLRO] -- BEGIN lro-1333560 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.277+08:00 info vpxd[51826] [Originator@6876 sub=vpxLro opID=vb-64567:auto-name-37137:01-f6] [VpxLRO] -- FINISH lro-1333560

2020-05-24T15:58:51.280+08:00 info vpxd[14310] [Originator@6876 sub=vpxLro opID=vb-64568:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-7b] [VpxLRO] -- BEGIN lro-1333561 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.281+08:00 info vpxd[14310] [Originator@6876 sub=vpxLro opID=vb-64568:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-7b] [VpxLRO] -- FINISH lro-1333561

2020-05-24T15:58:51.492+08:00 info vpxd[14148] [Originator@6876 sub=vpxLro opID=vb-64571:auto-name-37139:01-a2] [VpxLRO] -- BEGIN lro-1333563 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.493+08:00 info vpxd[14148] [Originator@6876 sub=vpxLro opID=vb-64571:auto-name-37139:01-a2] [VpxLRO] -- FINISH lro-1333563

2020-05-24T15:58:51.497+08:00 info vpxd[14204] [Originator@6876 sub=vpxLro opID=vb-64572:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-21] [VpxLRO] -- BEGIN lro-1333564 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.498+08:00 info vpxd[14204] [Originator@6876 sub=vpxLro opID=vb-64572:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-21] [VpxLRO] -- FINISH lro-1333564

2020-05-24T15:58:51.704+08:00 info vpxd[14478] [Originator@6876 sub=vpxLro opID=vb-64575:auto-name-37141:01-99] [VpxLRO] -- BEGIN lro-1333566 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

2020-05-24T15:58:51.704+08:00 info vpxd[14478] [Originator@6876 sub=vpxLro opID=vb-64575:auto-name-37141:01-99] [VpxLRO] -- FINISH lro-1333566

2020-05-24T15:58:51.707+08:00 info vpxd[14166] [Originator@6876 sub=vpxLro opID=vb-64576:urn:vmomi:VirtualMachine:vm-2536:630367e1-a9b8-4f9b-972a-859f5ac994f6.properties:01-1b] [VpxLRO] -- BEGIN lro-1333567 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 529b2558-c996-48ab-e86e-ec94f894efd5(52503d8a-bf79-5bde-4f8f-2780c3483e3e)

@

"vpxd.log" [noeol] 15152L, 3413762C    

Help me guys how to fix my VCSA.

[daphnissov]

If you have to manually change things every time you reboot, something wasn't done right. I'd say you should probably redeploy the appliance and ensure you do it the right way (with FQDN and local DNS).

[a_p_]

Please take a look at https://kb.vmware.com/s/article/68149​ which describes the issue with the stopped applmgmt service.

I ran into this issue on several customer environments too, after upgrading them.

André

[Lokesh_gs]

Thanks for the Reply daphnissov

i am also considering the point of re-installation. can you clarify on the license part? How to retain the license when destroying the existing one.

Before going to this I want to investigate what actually went wrong. what is the root cause so that i maybe not doing that again and face the same situation. My Bad i haven't created a backup from VCSA. i used Third-party tools but to restore that you need Vcenter Running.

[Lokesh_gs]

Thanks for the reply

I am not only having the appmgmt service but also all other services. If i increase will it fix for all or but only Appmgmt service? How about the management IP is not getting saved, why was that? any idea?

[a_p_]

Sorry, my bad. I somehow missed the Stopped and Running state in the services output, because it's usually the other way around. So that's definitely another issue.

André