Recovering vCenter 7

[Dr_Virt]

Had a lab vCenter crash and am trying to figure out why.

Current symptoms:

* If starting vCenter from shell with "service-control --start --all" the process will fail with vPostgres couldn't start.

* If starting vPostgres manually ("service-control --start --vmware-vpostgres") and then starting vCenter ("service-control --start --all") the proccess will fail with vpxd-svcs failed to start. 

* I logged into vCenter VCDB and verified administrator account

* I reset vCenter certificates and validated with lsdoctor

* vxpd.log shows "Failed to connect to Authz service" and "Failed to initialize authorizeManager"

Anyone seen something like this?

[Dr_Virt]

@hirschinho 

It is a standard cluster linked to another (management & compute vCenters).

I have done a single node vSAN deployment before. The premise is a single host is taken out of the existing cluster (4 hosts), cleaned, and then a single node vSAN deplyment and vCenter are used to start the new rebuild cluster. 

NSX is 3.2.2.

[Dr_Virt · Accepted solution]

Well, was able to recover. VMware sent a certificate tool (vCert) which identified some trust issues and registrations which the standard tools didn't address. 

Then I found an issue with setting up logging within the tomcat instance. I commented out the "isAccessLogCreated" and "accessLogCleaner" beans from the Tomcat config.

I also had to manually rebuild the vPostgresql certificate store.

I restarted the services and got the core up and running. I got a good snapshot of the VCSA. I attempted to do a VCSA back and it failed continuously. I decided to attempt an upgrade to repair the VCSA. It took about 2 hours, but the upgrade completed from 7.03f to g. I continue to walk the update path all the way to the latest 7.03 release. 

I tested the VCSA backup and it ran successfully. 

I tested the Tomcat by uncommenting the previously commented out beans. It ran successfully. 

In summary, there was corruption at multiple points within the VCSA. The help here and from VMware was able to recover it. Thank you all.